Allow replaced git sources to not include the git history

[nmattia]

Problem

The Source Replacement mechanism allows for sandboxed build systems to download crates ahead of times and replace remote crates with local downloads at build time.

There is however an issue with git sources that include workspaces (virtual manifests). Such a git source can only be replaced with another (local) git source as (as far as I can tell) only git sources are searched recursively (for workspace members). Since some build systems only support downloading archives (tar.gz, zip, etc) without the actual .git history, the git sources cannot be replaced:

[source]

[source.naersk-git-rand]
git = "file:///nix/store/downloaded-ahead-of-time/rand"

[source."foo"]
git = "git+https://github.com/rust-random/rand"
rev="703452450770d4b2bb0b117dfc83aea5ba61ec60"

replace-with = "naersk-git-rand"

cargo check      
   Updating git repository `file:///nix/store/downloaded-ahead-of-time/rand`
error: failed to get `rand` as a dependency of package `git-dep v0.1.0 (/Users/nicolas/naersk/test/fast/git-dep/fixtures)`

Caused by:
 failed to load source for dependency `rand`

Caused by:
 Unable to update https://github.com/rust-random/rand?rev=703452450770d4b2bb0b117dfc83aea5ba61ec60#70345245

Caused by:
 failed to update replaced source https://github.com/rust-random/rand?rev=703452450770d4b2bb0b117dfc83aea5ba61ec60#70345245

Caused by:
 revspec '703452450770d4b2bb0b117dfc83aea5ba61ec60' not found; class=Reference (4); code=NotFound (-3)

Here are some Nix projects that run into the issue (one with a workaround):

• Error when using a git dependency that includes a workspace, when the flake is also in a workspace nix-community/naersk#310
• Properly handle workspace inheritance in git dependencies ipetkov/crane#224 with workaround
• error inheriting version from workspace root manifest's workspace.package.version nix-community/crate2nix#311

I haven't looked at the cargo internals in a while; hope I'm not missing something obvious!

Proposed Solution

If other source types could be searched recursively, or if the rev could also be replaced, replacing git sources would be much easier.

Something like this would allow the build system to recreate a git history and use this:

[source.naersk-git]
git = "file:///nix/store/..."
rev = "<the-new-rev>"

[source."foo"]

git = "https://github.com/rust-random/rand"
rev="703452450770d4b2bb0b117dfc83aea5ba61ec60"

replace-with = "naersk-git"

Alternatively, existing source types like directory could support recursively looking for packages.

Thanks!

Notes

No response

[epage]

From the title, this sounds like this was going to be about shallow clones (#13285) but instead you are wanting to replace a git source with.

For the linked issues, is it that you are trying to vendor the source manually, taking the git repo and re-arranging it to look like what a directory source needs? Have you considered running cargo vendor which will do that in a way that works?

[epage]

As for having path sources recursive like git, we have #13360.

[nmattia]

Hi and thanks for the quick reply!

As for having path sources recursive like git, we have #13360.

This seems to be related to dependencies in the Cargo.toml, is this correct? I would like to have source replacements of type directory support this, like source replacements of type git do. That being said, maybe I'm missing the connection!

is it that you are trying to vendor the source manually

I am trying to instruct cargo to avoid downloading sources or crates at build time. In order to achieve this for crates-io dependencies, eg https://github.com/nix-community/naersk uses the built-in Nix fetcher which has network access in a controlled way.

For git sources, we've (until now) simply replaced them with a directory source which was populated with the built-in git fetcher (which unfortunately does not retain the .git/).

Regarding cargo vendor I gave this a try, but was under the impression that it would also try to download transitive dependencies (at build time); when running it inside the rand repo, it tried to download wasi and others. Maybe there's a way to prevent this?

[epage]

This seems to be related to dependencies in the Cargo.toml, is this correct? I would like to have source replacements of type directory support this, like source replacements of type git do. That being said, maybe I'm missing the connection!

For Cargo's internals, all sources are interchangeable. The only question is what we expose in each location. path isn't exposed today for source replacement, likely because it doesn't make sense. directory isn't expose today for dependencies. If we made path recursive, then it could make sense to expose that for source replacement. As this would be work on top of #13360, I didn't close this as a duplicate but this Issue could inform the conversation on that one.

Regarding cargo vendor I gave this a try, but was under the impression that it would also try to download transitive dependencies (at build time); when running it inside the rand repo, it tried to download wasi and others. Maybe there's a way to prevent this?

It will. If its ok to download but not keep, then you can filter out the results.

A potential hacky route would be to run cargo package (especially with the workspace support added in 1.90) and then lay out the .crate files in the directory source. That is effectively what cargo vendor is doing.

[nmattia]

If its ok to download but not keep, then you can filter out the results.

Unfortunately not, in Nix network access is generally restricted.

A potential hacky route would be to run cargo package (especially with the workspace support added in 1.90) and then lay out the .crate files in the directory source. That is effectively what cargo vendor is doing.

I've tried this; it looks like I then need local-registry instead of directory. Then it appears I might have to build a proper registry index myself or use cargo-local-registry.

Before I sink too much time in this, can you confirm that's what you had in mind? or is there an alternative way to simply add the cargo package-generated crates in a directory and call it a day?

Also, in order to avoid network access, I had to sprinkle a couple of arguments:

cargo package --no-verify --no-metadata --offline -p "rand" -p ... --exclude-lockfile

Either way, thanks a ton for the tips so far!

[epage]

Before I sink too much time in this, can you confirm that's what you had in mind? or is there an alternative way to simply add the cargo package-generated crates in a directory and call it a day?

My expectation with the suggestion was to compress the .crate files into a directory, creating a directory source by hand.

[nmattia]

the suggestion was to compress the .crate files into a directory

I'm assuming you meant decompress, and it seemed to work! Cargo does find my project's direct dependencies.

Unfortunately transitive dependencies from within the workspace can't be found; it looks like through cargo package the rand crate is now looking for eg rand_core on crates.io instead of the one from its own workspace.

You mentioned this above:

(especially with the workspace support added in 1.90)

I can't trivially update to rustc 1.90.0 so I'll shelve this for now and pick it up later. Thanks for the support thus far!

[nmattia]

I've had another go at the hacky solution. I landed on this combination of arguments:

            args=(
                --offline # assume no network
                --exclude-lockfile # don't create a lockfile (would involve resolving dependencies from crates.io)
                --no-metadata # don't warn if homepage, etc are not set
                --no-verify # don't build the dependency
                --manifest-path "$manifest" --target-dir ./target
            )

            for name in $packages; do
                cargo package "''${args[@]}" -p "$name" || true
            done

This works well for git dependencies that are specified with rev=.... Unfortunately for git deps with branch=, cargo fails to find the dependency and wants to update the lockfile:

...
git-dep-deps> cargo --locked build $cargo_release -j "$NIX_BUILD_CORES" --message-format=$cargo_message_format
git-dep-deps> error: the lock file /private/tmp/nix-build-git-dep-deps-0.1.0.drv-1/dummy-src/Cargo.lock needs to be updated but --locked was passed to prevent this
git-dep-deps> If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.

...
$ cat /private/tmp/nix-build-git-dep-deps-0.1.0.drv-1/dummy-src/.cargo-home/config
[source.crates-io]
directory = "/nix/store/r5fcd4x1jji5ps39nbsrhihsx35frk1x-crates-io-dependencies"

[source."https://github.com/rust-random/rand?branch=0.7"]
branch = "0.7"
git = "https://github.com/rust-random/rand"
replace-with = "crates-io"

$ ls /nix/store/r5fcd4x1jji5ps39nbsrhihsx35frk1x-crates-io-dependencies | grep rand-
rand-0.7.3

$ grep </private/tmp/nix-build-git-dep-deps-0.1.0.drv-1/dummy-src/Cargo.lock 'checksum.*rand'
"checksum getrandom 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)" = "e7db7ca94ed4cd01190ceee0d8a8052f08a247aa1b469a7f68c6a3b71afcf407"
"checksum rand 0.7.3 (git+https://github.com/rust-random/rand?branch=0.7)" = "<none>"
"checksum rand_chacha 0.2.2 (git+https://github.com/rust-random/rand?branch=0.7)" = "<none>"
"checksum rand_core 0.5.1 (git+https://github.com/rust-random/rand?branch=0.7)" = "<none>"
"checksum rand_hc 0.2.0 (git+https://github.com/rust-random/rand?branch=0.7)" = "<none>"

is there a reason why in the branch= case the source is not replaced with source.crates-io, but with rev= it is?