Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

During realization, consider realizations whose build inputs are not in the store #39

Open
zombiezen opened this issue Sep 13, 2024 · 0 comments
Labels
backend Build server enhancement New feature or request
Milestone

Comments

@zombiezen
Copy link
Collaborator

Consider a hypothetical (simplified) build of a Go program where P (the final program) depends on G (the Go compiler) depends on C (the C compiler). To build derivation P, we only need a realization for G: C's output does not need to be present unless G's output contains references to C's output. This is especially useful once substituters are implemented, because it means that trusted realization metadata can be downloaded (very cheaply) and then dependencies of a complex toolchain can be skipped if only the toolchain itself is needed.

I started mocking this up while writing 064a339. Basically:

  1. Walk the derivation graph only performing realizations and hashing derivations. During this stage, you permit realizations that don't exist in the store. (Downloading new realization data from substituters is fine, but we want to avoid downloading new store objects.)
  2. Keep walking until you run out of realizations.
  3. Now you should have a map of new realizations and a map of new hashes. If there is a path in the build graph from the desired outputs to a realization that doesn't exist in the store without crossing a realization that does exist in the store, remove any realizations from the map that depend on the realization that doesn't exist in the store. Also remove any realizations that don't exist in the store and the associated hashes for those derivations.
  4. Add any remaining realizations and hashes to the builder.
  5. Proceed with the realization as normal.

This effectively prunes portions of the dependency graph that have local dependencies available.

@zombiezen zombiezen added enhancement New feature or request backend Build server labels Sep 13, 2024
@zombiezen zombiezen added this to the MVP milestone Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Build server enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant