Enable `pip.dep` to be used for concrete dependency management

[csmulhern]

🚀 feature request

Relevant Rules

This is an extension of the existing pip.dep feature: https://rules-python.readthedocs.io/en/latest/api/rules_python/python/extensions/pip.html#pip.dep.

Description

pip.dep today is only used for abstract dependency management. The concrete dependencies still must come through pip.parse and a requirements.txt file.

Describe the solution you'd like

It would be nice to be able to specify dependencies / version requirements entirely in MODULE.bazel using pip.dep. This is similar to how the crate_universe feature works in rules_rust.

Describe alternatives you've considered

N/A

[aignas]

This means that the users would also have to define the transitive deps or you would expect bazel to handle that? What target platforms and python versions should be targeted? Could you please add API examples of what you would expect?

[rickeylev]

I don't see how transitive deps could be avoided. Well, maybe if bzlmod traversed the simple API? Ah, but then that gets into issues of platform-specific dependencies, constraint solving, ...

The more I think about this, the more i keep coming back to: it's the same as taking a requirements.in or pyproject.toml as input. i.e. those are just a listing of packages that are needed, not the extra transitive dependencies of those that are needed. And that on its own isn't very useful -- hence why so much code has been spilt on requirements.txt, pylock.toml, and uv.lock processing

If we look at our docs build as an example -- which is a small project -- it's 600-1000 lines of configuration. I suppose it would be shorter if you just had to express a dependency graph without any of the hashes, indexes etc. Though, I don't know how one could avoid that part of the config.

[aignas]

The only way I can see it working is:

• In the extension evaluation phase we create a pyproject.toml file.
• Run uv lock to generate the lock file.
• Parse it and write the contents into the MODULE.bazel.lock file.
• The next time we run the module extension evaluation phase, we read the contents from MODULE.bazel.lock file.

This means if the pip.dep usage becomes pervasive, the root module user will have to pay the price of potentially multiple uv lock invocations.

There are 2 options:

1. Instead of using MODULE.bazel.lock file write it to something more persistent that can be used by non-root modules published through the registry.
2. Ask users to input all of the deps via MODULE.bazel

I think 2 is better, because it is more explicit. However, that essentially becomes using whl_library via use_repo_rule helper with some extra glue. The minimum requirements for this is to specify the name, version, index_url, which extras are enabled and the target platforms (since some users are concerned about downloading unnecessary deps during bazel query). I have opened #3902 for other reasons, but that could be the first test of the API if it is possible to do it.

So my thinking about attributes to pip.dep in that case (where it is only added through the unified_pip_hub):

• name - mandatory
• version - if specified, we'll actually look up index_url for the particular wheels. Sorry, no sdists for now.
• sha256 - list for the lookup, can be empty.
• index_url - defaults to public PyPI, can be also controlled through pip.default
• extras - extras enabled for the package to generate the deps inside the whl_library. Defaults to none.
• venv - default, can be overriden for something else, what, I am not sure.
• BUILD.bazel - optional to override any BUILD.bazel that would be created.
• patches - optional, to patch the wheel if needed.
• extra_targets - the list of targets that should be symlinked to the hub.

If multiple pip.dep are specified we can either:

• Select the root module specification.
• Use maximum version out of all - we have Starlark code to do comparison.
• Use minimum version out of all.

If someone is interested in driving this and do research I can mentor. That said, whether this will result in something mergable is subject to having a better tech design than what I have written in 15mins above.