Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support multiple constraints file #539

Open
tiran opened this issue Jan 24, 2025 · 0 comments · May be fixed by #540
Open

Support multiple constraints file #539

tiran opened this issue Jan 24, 2025 · 0 comments · May be fixed by #540

Comments

@tiran
Copy link
Collaborator

tiran commented Jan 24, 2025

In version 0.36 and earlier, Fromager supports only a single constraints file. The option --constraints-file is single value. It would be useful to support multiple constraints files. In downstream, we have variant-specific constraints and product-specific constraints for all product variants. For example, we have to constrain some packages for all variants, because latest version of the package does not build for us. Support for multiple constraints files would allow us to move common constraints for all variants into a shared file.

Implementation

Let's start with a simple implementation:

  • all constraints file must exist and be readable
  • all constraints are combined into a single constraints set
  • a package can only be constraint once across all constraints files. The second occurrence of a package raises an error.

The implementation should cover majority of use cases. We can look into more complex ways to deal with multiple constraints if the need arises.

Env vars and URLs

Fromager uses click to parse command line arguments. Click also supports env vars, e.g. --constraints-file can also be set with FROMAGER_CONSTRAINTS_FILE env var. Env vars are a common way to pass settings into containers. Click uses different rules to split multiple values from env vars. Path-like types are split by os.pathsep (: like PATH vars) and other values are split by white space.

A while ago --constraints-file was switched from path-like to str type to support https:// URLs for remote constraints files. We cannot switch --constraints-file to a type=str, multiple=True option. This would break cases like FROMAGER_CONSTRAINTS_FILE="/home/user name/constraints.txt". Click would split the env var into two items /home/user and name/constraints.txt.

I have considered and experimented several approaches how to reconcile local paths, remote URLs, and env vars. My preferred approach is:

  • change --constraints-file to multi-value option that accepts only local paths. FROMAGER_CONSTRAINTS_FILE splits on colon ':'.
  • introduce second option --constraints-url that accepts multiple remote URLs. FROMAGER_CONSTRAINTS_URL splits on white space ' '.
tiran added a commit to tiran/fromager that referenced this issue Jan 24, 2025
Fromager now supports multiple constraints files. The option
`fromager --constraints-file` can be supplied multiple times. The option
no longer accepts URLs. Remote constraints files are now specified with
`--constraints-url`. The split was necessary to correctly parse
environment variables with multiple entries and white spaces in local
file names. Paths are split at `:`, URLs are split at ` ` (white space).

There can only be one constraint per package.

Internally, remote constraints from `https://` URLs are no longer
retrieved twice. Instead all constraints are merged and the final set is
dumped into a single constraints file.

Fixes: python-wheel-build#539
Signed-off-by: Christian Heimes <[email protected]>
@tiran tiran linked a pull request Jan 24, 2025 that will close this issue
tiran added a commit to tiran/fromager that referenced this issue Jan 24, 2025
Fromager now supports multiple constraints files. The option
`fromager --constraints-file` can be supplied multiple times. The option
no longer accepts URLs. Remote constraints files are now specified with
`--constraints-url`. The split was necessary to correctly parse
environment variables with multiple entries and white spaces in local
file names. Paths are split at `:`, URLs are split at ` ` (white space).

There can only be one constraint per package.

Internally, remote constraints from `https://` URLs are no longer
retrieved twice. Instead all constraints are merged and the final set is
dumped into a single constraints file.

Fixes: python-wheel-build#539
Signed-off-by: Christian Heimes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant