Help keep a project's venv automatically in sync with its declared dependencies

[jherland]

In https://mastodon.social/@[email protected]/110666956985225219, Xavier describes a need to automatically keep a project's venv in sync with its declared deps.

Is your feature request related to a problem? Please describe.

Python provides many ways to declare dependencies (requirements.txt, pyproject.toml, etc.), as well as several ways to satisfy these (system-wide install, pip --user installed, virtualenvs in various forms ,__pypackages__, etc.). However, a very common pattern is to use a project-specific virtualenv that should always stay in sync with the declared dependencies.

The real problem here is the "should always" part... Typically, a command can be run to manually sync these (e.g. poetry install --sync), but this is not done automatically.

Describe the solution you'd like

The full solution to this problem is not yet known, and might vary quite a bit from one project to the next (e.g. it depends on the exact tool used to manage venvs). However, an automatic solution should be able to:

1. Know there the project's venv is located, and where the deps are declared
2. Quickly determine if the project's venv is currently in line with the declared deps
3. If the answer is not, then run a command (customizable) to fix the discrepancy.

It should be fairly straightforward to improve FawltyDeps (or build a tool that uses FawltyDeps as a library) to solve the first two steps above:

• FawltyDeps is able to find declared deps in a project, as well as embedded venvs. If more alternatives are present for either, the --deps and --pyenv options can be used to specify precisely which is to be used.
• FalwtyDeps can enumerate (via --list-deps) the declared dependencies of the project.
• FawltyDeps should be able to (although it currently does not expose this) enumerate the packages that are installed in a given venv.
• Currently FawltyDeps does not concern itself with the version number of either declared deps, or installed packages, but this could potentially be improved upon. Doing so has been considered for our longer-term roadmap in any case.
• Given a set of declared deps (w/versions) and a set of installed packages (w/versions) it should be straightforward to match them against each other and report any discrepancies.

For the third step above, FawltyDeps currently has no concept of which tool is used to manage venvs, and hence, it has no idea what command would be appropriate for bringing the project venv in sync with the declared deps. We could leave this step out of FawltyDeps altogether (leaving it to the user to assemble a complete triggering mechanism), or we could consider adding more support/configurability to FawltyDeps to bake this part into FawltyDeps as well.

What remains, however, is a mechanism for triggering this logic at the appropriate time, and here I suspect that FawltyDeps has nothing to offer on its own, and would have to rely on some external mechanism like direnv, cron, or similar.

Describe alternatives you've considered

The ensuing discussion on https://mastodon.social/@[email protected]/110666956985225219 suggests using Nix + direnv, but also presents why that solution might not be popular.

There are potentially several other ways to approach the same problem. Unconditionally running the sync command (e.g. poetry install --sync) at certain intervals (either triggered via entering the dev environment or with a cron job) is a low-effort way to achieve the same.