feat(build): git source

[remimimimimi]

Add support for specifying a git source for the pixi build backend. As suggested, we now use pin_and_checkout for source fetching.

When source package is added to some environment its source will be pinned. Correspondingly pixi build won't update lock file.

Tests are added to pixi-build-testsuite here.

Still draft as it depends on rattler release.

[lucascolley]

this looks great! Want to make a companion PR to pixi-build-testsuite?

[baszalmstra]

I think we also need to add this to the lock file!

[remimimimimi]

I think we also need to add this to the lock file!

So, I should change the lock file format to include sources as well, in a manner similar to how we currently store conda packages? Do you have any suggestions for implementing this? @baszalmstra

[baszalmstra]

Yes essentially. But we can also do that in a follow up PR, there is more we should add to the lockfile, like the variants.

And/or we can only allow revs and not allow branch, tag or none.

Without the pinned source in the lockfile the changes that a package will break is pretty large.

[remimimimimi]

Yes essentially. But we can also do that in a follow up PR, there is more we should add to the lockfile, like the variants.

And/or we can only allow revs and not allow branch, tag or none.

Without the pinned source in the lockfile the changes that a package will break is pretty large.

I'll allow only rev then.

[ruben-arts]

Touching the rev gives these explosive errors:

examples/pixi-build/git-source on  feat/pixi-build-git-source [$?] 🧚  took 3s 
➜ pixi i
 WARN overriding build backend with system prefixed tools
ERROR Error fetching Git repository: Object ID can be at most 40 hex characters
Error:   × failed to build 'sdl_example' from '.'
  ╰─▶   × Object ID can be at most 40 hex characters
      

This should mention the rev, hopefully doing this error on parsing so we can point to the span with it.

examples/pixi-build/git-source on  feat/pixi-build-git-source [$?] 🧚  took 2s 
❯ pixi i
 WARN overriding build backend with system prefixed tools
ERROR Error fetching Git repository: failed to fetch https://github.com/prefix-dev/pixi-build-testsuite.git: fatal: couldn't find remote ref refs/tags/ee87916a49d5e96d4f32268c3s650e8ff6b8866b

Error:   × failed to build 'sdl_example' from '.'
  ╰─▶   × failed to fetch https://github.com/prefix-dev/pixi-build-testsuite.git: fatal: couldn't find remote ref refs/tags/ee87916a49d5e96d4f32268c3s650e8ff6b8866b
        │

This prints the error first in a tracing and after in the miette error. I would only do the miette error.

Toml used for testing:

[package.build.source]
git = "https://github.com/prefix-dev/pixi-build-testsuite.git"
rev = "ee87916a49d5e96d4f32268c3s650e8ff6b8866b"
subdirectory = "tests/data/pixi_build/cpp-with-path-to-source/project"

[package.build.backend]
channels = [
    "https://prefix.dev/pixi-build-backends",
    "https://prefix.dev/conda-forge",
]
name = "pixi-build-cmake"
version = "*"

[package]
name = "sdl_example"
version = "0.1.0"

[package.host-dependencies]
# This ensures that SDL2 is available at build time.
sdl2 = ">=2.26.5,<3.0"

[workspace]
channels = ["https://prefix.dev/conda-forge"]
platforms = ["osx-arm64"]
preview = ["pixi-build"]

[dependencies]
sdl_example = { path = "." }

[ruben-arts]

Changing the rev doesn't trigger a new checkout of the project.

[package.build.source]
git = "https://github.com/prefix-dev/pixi.git"
rev = "1eec91ba3be59e93d487f61801882d05d6f76a6c"
#rev = "2e05115c55fd119b7296062d38081c57aa64fdeb"
subdirectory = "examples/pixi-build/cpp-sdl"

Switching these two revs should give you two different main.cpp files which should trigger a rebuild and reinstall.

It only triggered a rebuild but no refetch and reinstall.

[ruben-arts]

I don't agree on merging this without the tag and branch. That is where this becomes really cool IMO.

I would expect the cargo workflow where the lockfile stores therev but the manifest allows for having only git or git with tag, branch or rev.

I would also like to have an example of this in theexamples/pixi-build as it's something I want to refer to more often.

[ruben-arts]

Adding this to avoid accidental merging.

[lucascolley]

@remimimimimi could you resolve the merge conflict?

[ruben-arts]

The UX that I would like to have tested in integration test:

Assuming this manifest:

[package.build.source]
git = "url"
branch = "branch"

Generate the lockfile with:

pixi lock

Test that the correct rev is put in the lockfile

Run:

pixi install --locked

This should work and not error

Now update the branch in the manifest to a different one that points to a different commit.

Run:

pixi install --locked

This should error as the branch doesn't satisfy the lockfile

Run:

pixi lock

This should update the rev in the lockfile
Test that the correct rev is put in the lockfile

[ruben-arts]

This looks good to me!

[remimimimimi]

@Hofer-Julian There here's are the parts of the PR that would help you to review it more easily:

1. Passing value through: I may have overdone this because I rewrote things several times and might have forgot to remove something.
2. Conversion between types here.
3. Overriding required for lock file handling to work correctly with pixi build.
4. Lock file invalidation.
5. Tests from here and testsuite PR.

outdated