Deprecate pip-compile-multi

[mistercrunch]

To manage our very complex tree of dependencies, we use pip-compile-multi, which describes itself as

Pip-compile-multi is a command line utility, that compiles multiple requirements files to lock dependency versions. Underneath it uses pip-tools for actual compilation. Pip-compile-multi targets complex projects and provides highest level of automation and flexibility in class.

Now while this fits our needs, there's one major drawback is that it does not support dependabot, the GitHub-acquired utility that watches the repo and submits PRs individually to bump each library. More on this dependabot/dependabot-core#536, but it's been years and seems like GH don't care much about this.

Now without dependabot, it's required that periodically we go and run pip-compile-multi which goes and bumps all the libs it can as one big PR.

Now the issue we have is that one of our dependency bump - we don't know which one - is breaking the build in an intricate way, this means that bumping all the other library in an automated fashion is held back by this one issue. Sure we could fix the issue, or we could bissect the upgrades, but that's a lot of manual labor, manual labor we don't really have time for.

In some ways, we're forced to choose between two tools that help us manage the burden of a deep-and-wide dep tree:

• pip-compile-multi allows for composable management of different intricate sets of deps
• dependabot allows for good day-to-day good hygiene, atomic upgrades of libraries, making it easy to pinpoint which update breaks what feature/test

what's in our pip-compile-multi setup?

Now do we really need pip-compile-multi? Can't we just use pip-compile? Let's look at what's in our multi-setup

• base.in: pointer to setup.py, Superset can't run without these
• development.in: utility tools for development (like ipython) + some extra_requires packages used for tests (hive, mysql, ...)
• local.in: useless, currently a pointer to development.in
• integration.in: pip-compile-multi, pre-commit and tox
• docker.in: greenlet(?)
• testing.in: other dev tools, docker is here not above for unknown reason, I guess docker is used for testing(?)

alternative

Could we get away with just two dependency packs, and have plain pip-compile manage those 2?

• requirements.txt, from pip-compile of our setup.py
• requirements-dev.in: what's needed for all of the testing/integration/docker - the kitchen sync, including some .[hive] and such
• requirements-dev.txt: generated by pip-compile of the file above, note that this would not overlap with requirements.txt, meaning for a dev setup you need to pip install both files.

backwards compatibility

People may have used requirements/* as a public interface for their installation setup. If we wanted to offer some backward compatibility, we could use symlinks or scripts that would stub the old files.

[mistercrunch]

@john-bodley curious what you think

[john-bodley]

@mistercrunch a couple of thoughts:

Firstly, how viable (if any) would it be to have dependabot support pip-compile-multi? There is peterdemin/pip-compile-multi#188 which might be relevant?

Secondly, you mentioned,

requirements-dev.txt: generated by pip-compile of the file above, note that this would not overlap with requirements.txt, meaning for a dev setup you need to pip install both files.

what mechanism ensures that these pinned requirements mutually exclusive? My sense is there would be overlap of some shared packages would be problematic from an installation/consistency perspective. For example in #27465 you have flask being pinned to 2.2.5 and 2.3.3 and in base.txt and development.txt respectively.

[mistercrunch]

Yes I read through the issue(s) around pip-compile-multi and dependabot support and there seem to be little traction. It might be a viable path to push and contribute a solution there, but little action over many years points to some uncertainty on getting this through.

About my POC PR, it's very much a first try and trying to disprove as much as proving whether things could work there. Next step I have to try there seem to point towards moving from setup.py to pyproject.toml (new recommended way to set these things up) and doing some pre-work ahead of pip-compile. Here I would assemble the equivalent of ".in" files from combining extra_requires groups of dependency rnages. It could be almost elegant to do this straight on top of that toml file. If I get something to work we can evaluate tradeoffs and whether that's more manageable than the current setup.

Note that we have similar complexity downstream in our shell repo that wraps Superset to productionize it at Preset. We're looking to replicate/build upon the best pattern.

I'll keep pushing to see if it's a dead end or not, but I'll be looking for input/feedback along the way.

Oh, one more thing, regardless of the mechanics we chose for python dependency management, I'm wondering how many environments we want to segment here. My impression is that 5-6 as we have now is too many, and 2-3 seems more manageable.

[john-bodley]

@mistercrunch for reference at Airbnb our dependencies look something like this:

# requirements/base.in
-e file:apache-superset[cors,druid,hive,mysql,prophet,trino]
airbnb-auth
...

# requirements/production.in
-r base.in
gevent

i.e., our dependencies are built in a similar fashion using pip-compile-multi.