nix copy --no-recursive: cannot add to the binary cache because the reference is not valid

[andrewhamon]

Describe the bug

I am trying to copy incomplete closures to a binary cache.

In my specific case, the reason is because I want to copy to a temporary directory, then use a different tool (gcloud) to actually copy the files into the bucket. In order to save on unnecessary disk IO and nar serialization/compression overhead, I don't want to copy the recursive closure. Instead, i want to copy only the specific paths which I know are missing from the remote destination.

(getting off topic, but you might ask: why not just use S3 compatibility mode? the answer is, we do, but the way GCS HMAC tokens work really sucks if you care about security, so I'm trying to cut our dependency on writable HMAC tokens).

So the flow I was hoping for:

• expand closure recursively
• check which paths exist in the remote store
• copy only the missing ones to a tmp dir nix copy --no-recursive --to file:///tmp/binary-cache/ some/path
• sync tmp dir to gcs bucket with gcloud CLI

Another possible use case is a implementing a cachix-like mechanism to filter out paths already in public cache (tho I'm not a huge fan of that, feels like freeloading to me).

All of this is pretty easy to script with a little nix path-info and jq, except that copying incomplete closures fails.

Steps To Reproduce

nix copy --no-recursive --to file:///tmp/binary-cache/ /nix/store/qplid9wf2bfbl3i7w6xyh0769gz7v3a2-nix-2.18.2
error: cannot add '/nix/store/qplid9wf2bfbl3i7w6xyh0769gz7v3a2-nix-2.18.2' to the binary cache because the reference '/nix/store/1xqzb5zzj7ig47bpyybhv2l0hnib9ggk-nlohmann_json-3.11.3' is not valid

Expected behavior

If --no-recursive is specified and the target store is a binary cache, do not check that references are valid.

For other types of stores, allowing this might not make sense (e.g. a local store maintains a DB of valid paths, and I can't think of a circumstance where it is okay to violate the store integrity.)

Metadata

nix-env (Nix) 2.26.3

Additional context

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.