Split test/development dependencies

[whatnick]

Expected behaviour

Production docker image does not ship with development dependencies

Actual behaviour

A single requirements.txt packs both test/pycqa and production dependencies.

Imporvements

• Split the requirements.txt into multiple files
• Created dev,test and production dependencies sections in setup.py

[benjimin]

Looking generally at the docker image..

The WPS image is currently 1.3GB (which intuitively seems kind of excessive, I'm unsure how much of this is actually used by the WPS application):

• According to dive it is 99% efficient (i.e. only 7mb updated between layers, such as the debian package manager log and template database).
• 420MB is associated with node. (As was noted in #122 we could excise the node dependency if processes allow charts to be rendered in the client rather than the server.)
• 470MB is python site-packages
• 150MB is various libraries installed by ODC Geobase (e.g. libgdal and dependencies)
• 90MB is the python base
• 70MB is ubuntu 20.04

The image seems fairly slow to build, e.g. ~9 minutes. This includes things like compiling rasterio from source, and downloading 120 different versions of boto3 (while pip searches for a version conflict resolution).

I think it might be worth comparing e.g. a micromamba based image (which uses a different dependency solver and draws from an externally managed compile chain) to see if the image can be made a bit lighter and more nippy before focussing on whether pytest etc are included with everything else.

Excluding testing dependencies from the production image is certainly best practice, but may complicate the testing a little. (Do we also separate unit tests and system integration tests, so that the former can run at the end of the first stage of a multi-stage docker build, and the latter interact from the outside with the final image?)

[uchchwhash]

micromamba does look good! Hope we find a path forward.