Description
What's the problem this feature will solve?
Hello, pip
maintainers!
This is (another) proposal for a plugin/extension system for pip
. My goals with it are twofold:
- Define a minimal plugin API that allows plugins, but keeps them at an arm's length from
pip
's API internals - Define a subcommand API that allows commands under the
pip ext CMD
hierarchy, allowing existingpip-
tooling (including tools that can't easily be integrated intopip
itself or shouldn't be) to provide a better and more consistent UX.
TL;DR: a minimal plugin architecture for pip
would allow for better integrations with external tooling, including codebases (e.g. cryptographic codebases with native components) that cannot be easily or desirably vendored into pip
itself.
Describe the solution you'd like
I have two things in mind:
- A limited entry points-based plugin architecture for
pip
, allowing third-party packages to register plugins. - A
pip ext ...
subcommand hierarchy, populated by plugins that register the appropriate entry point, allowing third-party packages to register wholly independent subcommands.
I think both of these would be nice to have, but I think either also makes a good proposal. So I'm curious to hear what others think!
Plugin architecture
My high level idea: pip
gains awareness of the pip.plugins
entry point group.
For example, a plugin might register as:
[project.entry-points."pip.plugins"]
quux = "sampleproject.plugin"
...where plugin
is a module object with the following minimal interface:
def plugin_type() -> PluginType:
...
and PluginType
is:
PluginType = Literal["some"] | Literal["another"]
from here, the remaining attributes of the plugin
module are determined by PluginType
; the intended contract between pip
and the plugin is that pip
will ignore (and warn on?) any plugin of a type it does not recognize.
(I have ideas for an initial trial-run PluginType
, but I want to make sure this basic approach/architecture is amenable before I get into the details there!)
pip ext
commands
pip ext
subcommands would be a specialization of the above architecture. For example, to register pip ext frobulate
, a third-party package might register the following:
[project.entry-points."pip.plugins.ext"]
frobulate = "sampleproject.cli"
From here, the cli
attribute is expected to be a module with the following attributes:
def description() -> str:
return "a brief oneline description of the command"
def main(args: list[str]) -> None:
...
...where args
is the list of arguments passed after pip ext frobulate
.
Under this model, subcommands under pip ext
are entirely responsible for their own lifecycle: pip
provides no public APIs, no additional context besides args
(and os.environ
), and the subcommand is expected to handle its own errors.
The description
callable is used solely to populate pip ext --list
, e.g. to an effect like this (probably more nicely rendered):
$ pip ext --list
plugin description
frobulate a brief oneline description of the command
wangjangle randomly install a python package
compile run pip-compile
Timeline
Either (or both) of these would be a significant feature addition to pip
. As such, my thinking is that they should go through pip --use-feature
like other experimental features, e.g.:
pip --use-feature=plugins
pip --use-feature=extensions
# or combined, no distinction?
pip --use-feature=plugins
From there, plugin/pip ext
developers could experiment with either feature before they're fully stabilized, without pip
committing to an exact API/interface until stabilization.
Alternative Solutions
The minimal alternative here is "do nothing." 🙂
However, for each of the above:
-
pip
plugins: expect people to wrappip
instead via its public CLI (or a wrapper likepip-api
. Where this isn't sufficiently introspective, users/communities could build their own one-off tools. This is more or less the status quo, and results in a lot of duplication/tools that buggily wrappip
(like some of my tools). -
pip ext
subcommands: Continue the status quo of people (informally) signaling the adjacency of their tool topip
viapip-
, e.g.pip-compile
,pip-tools
,pip-audit
, etc. This is workable, although it's not the nicest UX compared to a unified subcommand CLI. Moreover, it can result in weird mismatches (e.g. wherepip
uses one Python/environment andpip-compile
uses another).
Additional context
A lot of ink has been spilled over plugin architectures before: #3999 and #3121 are probably the oldest and most immediately relevant, but there are references to user requests for various plugin/API architectures scattered throughout the issues. I can try to collate all of them, if desired 🙂
After discussion, if some variant of this proposal is amenable, I (and my colleagues) will happily implement it and provide ongoing maintenance for it (like we do for PyPI, twine, gh-action-pypi-publish, etc.) -- our objective is not to drop a pile of new code on pip
and run away, but to work closely with you all and make sure that anything we propose strikes the right balance between value provided to end users, potential new error modes, and your limited maintenance time.
Code of Conduct
- I agree to follow the PSF Code of Conduct.