Emit RunfilesGroupInfo in py_binary targets

[malt3]

🚀 feature request

Relevant Rules

I want py_binary to return an additional provider, RunfilesGroupInfo.
py_library could contribute by also returning this provider, but that's more of an implementation detail.

I'd be open to work on this if you agree that this is a good idea.

Description

I'd like to make packaging of py_binary targets in container images more efficient. To that end, I have designed a set of providers and described their semantics in a new Bazel module.

Describe the solution you'd like

py_library targets should forward RunfilesGroupInfo and RunfilesGroupMetadataInfo from their deps and add their own runfiles (srcs, data) to a group.
py_binary targets should aggregate runfiles groups from their deps and emit RunfilesGroupInfo and RunfilesGroupMetadataInfo that contain at least:

• A group for the interpreter + standard library
• One group per third party dependency

Additionally, it could also make sense to produce groups for each py_library target in the transitive dependency graph.

Describe alternatives you've considered

Doing nothing would mean we continue creating (very large) container image layers that contain the full runfiles tree of a py_binary target.

Writing custom packaging rules just for rules_python is unsustainable in my opinion.

[aignas]

I am not against having a generic runfiles grouping, however, I don't fully understand yet how this may clash with our current work that we are doing with the VenvSymlink provider. I also agree that custom packaging rules is not the way forward either. A few observations about packaging of rules_python targets.

• If we just take the binary output it will be <package-prefix>/target and <package-prefix>/target.runfiles.
• Inside there may be symlinks that we have to preserve within the tar files. At the moment those symlinks may come from either the interpreter or the py_library targets.
• Usually those files don't change much and the main variables are: interpreter, 3rd party deps and the 1st party code. If needed, the symlinks could be in a separate layer? They usually don't change.
• The dependency whl_library rules have <prefix>_<whl_name>_<sha> type of naming. The prefix in here is an implementation detail and is not stable and may change at any time.

A few questions that I don't understand about your proposal is how would the user use it and what is the maintenance burden put on the ruleset maintainers here. Do you plan to be the maintainer of this particular integration? How much coupling do we get when taking on this dependency?

[rickeylev]

malt3 posted something with some more details bazelbuild/proposals#417

The basic idea is to provide multiple runfiles objects instead of a single one in DefaultInfo. The sum of these differnet runfiles objects equals what's in DefaultInfo

Insofar as we're concerned, the main technical blocker i see is the proposal only supports plan depsets, but we make use of runfiles.root_symlinks, too.

what about preserving symlinks

This can be automatic for Bazel 8+, which has the File.is_symlink attribute. In the py_zipapp code, it mostly just takes a runfiles object and put it in the zip. Windows and Bazel 7 require special hacks, unfortunately.

what about VenvSymlinks

I don't think it would clash, but it could make for a weird interaction. The VenvSymlinks info is metadata that eventually turns into runfiles at the binary level. It can't turn into runfiles earlier because only a binary knows where its venv is created. A library could give back a RunfilesGroupInfo, but it would just be the "raw" extraction of the library, not the "logical" way it looks in a venv. Not necessarily conflicting but more like incomplete.

If py_binary collects, aggregates, and forwards RGI, that would probably be workable. But if an aspect is traversing to find RGI separately, that will probably be strange. Maybe if, when the binary computes the venv, it groups it with the RGI from the 3rd party dep? IDK.

[malt3]

Insofar as we're concerned, the main technical blocker i see is the proposal only supports plan depsets, but we make use of runfiles.root_symlinks, too.

I think you are right. The current proposal doesn't split root_symlinks or symlinks into groups. We should handle that in RGI as well. Let me revise it.

This can be automatic for Bazel 8+, which has the File.is_symlink attribute. In the py_zipapp code, it mostly just takes a runfiles object and put it in the zip. Windows and Bazel 7 require special hacks, unfortunately.

We should definitely add guidance on what to do as an archiver if you encounter files with File.is_symlink. I don't see this as a hard blocker.

If py_binary collects, aggregates, and forwards RGI, that would probably be workable. But if an aspect is traversing to find RGI separately, that will probably be strange. Maybe if, when the binary computes the venv, it groups it with the RGI from the 3rd party dep? IDK.

The way that I found works best is if each dep is able to contribute its own groups (where this makes sense) and the binary rule (py_binary) collects, aggregates and forwards RGI.
So in this case, third party deps could be in their own groups that are aggregated by the binary rule.

[malt3]

I have now overhauled the provider. Each group now stores a full runfiles object with files, symlinks, root_symlinks, and empty files: bazel-contrib/rules_runfiles_group#16

[aignas]

Nice! As long as the rules_runfiles_group remains minimal (right now it is bazel_features and bazel_skylib) and does not require us adding extra WORKSPACE/bzlmod setup steps, I see no reason why we should hold off from accepting contributions implementing the groups in our py_library, py_binary and py_runtime rules.