Skip to content

[RFC] A plugin/extension architecture for pip #12766

Open
@woodruffw

Description

@woodruffw

What's the problem this feature will solve?

Hello, pip maintainers!

This is (another) proposal for a plugin/extension system for pip. My goals with it are twofold:

  1. Define a minimal plugin API that allows plugins, but keeps them at an arm's length from pip's API internals
  2. Define a subcommand API that allows commands under the pip ext CMD hierarchy, allowing existing pip- tooling (including tools that can't easily be integrated into pip itself or shouldn't be) to provide a better and more consistent UX.

TL;DR: a minimal plugin architecture for pip would allow for better integrations with external tooling, including codebases (e.g. cryptographic codebases with native components) that cannot be easily or desirably vendored into pip itself.

Describe the solution you'd like

I have two things in mind:

  1. A limited entry points-based plugin architecture for pip, allowing third-party packages to register plugins.
  2. A pip ext ... subcommand hierarchy, populated by plugins that register the appropriate entry point, allowing third-party packages to register wholly independent subcommands.

I think both of these would be nice to have, but I think either also makes a good proposal. So I'm curious to hear what others think!

Plugin architecture

My high level idea: pip gains awareness of the pip.plugins entry point group.

For example, a plugin might register as:

[project.entry-points."pip.plugins"]
quux = "sampleproject.plugin"

...where plugin is a module object with the following minimal interface:

def plugin_type() -> PluginType:
    ...

and PluginType is:

PluginType = Literal["some"] | Literal["another"]

from here, the remaining attributes of the plugin module are determined by PluginType; the intended contract between pip and the plugin is that pip will ignore (and warn on?) any plugin of a type it does not recognize.

(I have ideas for an initial trial-run PluginType, but I want to make sure this basic approach/architecture is amenable before I get into the details there!)

pip ext commands

pip ext subcommands would be a specialization of the above architecture. For example, to register pip ext frobulate, a third-party package might register the following:

[project.entry-points."pip.plugins.ext"]
frobulate = "sampleproject.cli"

From here, the cli attribute is expected to be a module with the following attributes:

def description() -> str:
    return "a brief oneline description of the command"

def main(args: list[str]) -> None:
    ...

...where args is the list of arguments passed after pip ext frobulate.

Under this model, subcommands under pip ext are entirely responsible for their own lifecycle: pip provides no public APIs, no additional context besides args (and os.environ), and the subcommand is expected to handle its own errors.

The description callable is used solely to populate pip ext --list, e.g. to an effect like this (probably more nicely rendered):

$ pip ext --list
plugin           description
frobulate        a brief oneline description of the command
wangjangle       randomly install a python package
compile          run pip-compile

Timeline

Either (or both) of these would be a significant feature addition to pip. As such, my thinking is that they should go through pip --use-feature like other experimental features, e.g.:

pip --use-feature=plugins
pip --use-feature=extensions

# or combined, no distinction?
pip --use-feature=plugins

From there, plugin/pip ext developers could experiment with either feature before they're fully stabilized, without pip committing to an exact API/interface until stabilization.

Alternative Solutions

The minimal alternative here is "do nothing." 🙂

However, for each of the above:

  1. pip plugins: expect people to wrap pip instead via its public CLI (or a wrapper like pip-api. Where this isn't sufficiently introspective, users/communities could build their own one-off tools. This is more or less the status quo, and results in a lot of duplication/tools that buggily wrap pip (like some of my tools).

  2. pip ext subcommands: Continue the status quo of people (informally) signaling the adjacency of their tool to pip via pip-, e.g. pip-compile, pip-tools, pip-audit, etc. This is workable, although it's not the nicest UX compared to a unified subcommand CLI. Moreover, it can result in weird mismatches (e.g. where pip uses one Python/environment and pip-compile uses another).

Additional context

A lot of ink has been spilled over plugin architectures before: #3999 and #3121 are probably the oldest and most immediately relevant, but there are references to user requests for various plugin/API architectures scattered throughout the issues. I can try to collate all of them, if desired 🙂

After discussion, if some variant of this proposal is amenable, I (and my colleagues) will happily implement it and provide ongoing maintenance for it (like we do for PyPI, twine, gh-action-pypi-publish, etc.) -- our objective is not to drop a pile of new code on pip and run away, but to work closely with you all and make sure that anything we propose strikes the right balance between value provided to end users, potential new error modes, and your limited maintenance time.

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions