Reorganize Python deps and use pip-compile

This change is to simplify our Python dependency specifications and to use [pip-tools](https://pypi.org/project/pip-tools/#description)'s `pip-compile` to generate hashed requirements files with the full dependency tree from input files. This also performs dependency resolution and will highlight any conflicting versions. The model for this is that of [Mozilla's Bedrock project, the Django project that serves mozilla.org](https://github.com/mozilla/bedrock).

The new file structure is as follows:

- `prod.in`: requirements for deployment to production and other servers
- `test.in`: requirements for executing Python tests locally or in CI
- `dev.in`: requirements for development work, running, and testing
- `docs.txt`: requirements to build the consumerfinance.gov docs.
- `scripts.txt`: Requirements for running certain jobs on Jenkins, so scripts can run in Jenkins without having to install all the other requirements.

The contents of `base.txt`, `django.txt`, `wagtail.txt`, and `libraries.txt` move into `prod.in`, which contains the minimum necessary to run consumerfinance.gov. `test.in` includes tools like `coverage`, `diff_cover`, and `tox` that are required to run our Python tests. `docs.in` and `scripts.in` are relatively unchanged, and `dev.in` combines all of the above to construct a local environment.

These relationships between files now look like this:

```mermaid
flowchart TD
    prod.in
    test.in
    dev.in
    docs.in
    scripts.in

    prod.in --> dev.in
    test.in --> dev.in
    scripts.in --> dev.in
```

Where previously they looked like this:

```mermaid
flowchart TD
    ci.txt
    docs.txt

    base.txt
    deployment.txt
    django.txt
    libraries.txt
    local.txt
    wagtail.txt

    scripts.txt
    test.txt

    django.txt --> base.txt
    wagtail.txt --> base.txt
    libraries.txt --> base.txt

    base.txt --> deployment.txt

    base.txt --> local.txt
```

.github/workflows/backend.yml

@@ -71,7 +71,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements/ci.txt
+          pip install -r requirements/test.txt
 
       - name: Run tox -e ${{ matrix.toxenv }}
         run: |
@@ -110,7 +110,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements/ci.txt
+          pip install -r requirements/test.txt
 
       - name: Retrieve backend coverage
         uses: actions/download-artifact@v3

.github/workflows/frontend.yml

@@ -61,7 +61,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements/ci.txt
+          pip install -r requirements/test.txt
 
       - name: Retrieve frontend coverage
         uses: actions/download-artifact@v3

.github/workflows/functional-tests.yml

@@ -56,7 +56,7 @@ jobs:
     - name: Install dependencies
       run: |
         python3 -m pip install --upgrade pip
-        python3 -m pip install -r requirements/local.txt
+        python3 -m pip install -r requirements/dev.txt
 
     - name: Set up initial data
       run: ./refresh-data.sh test.sql.gz

Dockerfile

@@ -29,7 +29,7 @@ RUN apk add --no-cache --virtual .build-deps gcc gettext git libffi-dev musl-dev
 
 # Install python requirements
 COPY requirements requirements
-RUN mkdir /build && pip install --prefix=/build -r requirements/deployment.txt
+RUN mkdir /build && pip install --prefix=/build -r requirements/prod.txt
 
 #######################################################################
 # cfgov-dev is used for local development, as well as a base for frontend.
@@ -48,7 +48,7 @@ RUN apk add --no-cache --virtual .backend-deps bash curl postgresql
 
 # Install python requirements
 COPY requirements requirements
-RUN cp -Rfp /build/* /usr/local && rm -Rf /build && pip install -r requirements/local.txt
+RUN cp -Rfp /build/* /usr/local && rm -Rf /build && pip install -r requirements/prod.txt
 
 EXPOSE 8000
 

backend.sh

@@ -22,7 +22,7 @@ install() {
   echo 'Installing back-end dependencies...'
 
   # Install requirements for Django Server or tox.
-  pip install -r ./requirements/local.txt
+  pip install -r ./requirements/dev.txt
 }
 
 init "$1"

compile-requirements.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# ==========================================================================
+# Import data from a gzipped dump. Provide the filename as the first arg.
+# NOTE: Run this script while in the project root directory.
+#       It will not run correctly when run from another directory.
+# ==========================================================================
+
+set -e
+
+# Always use the latest pip
+pip install -U pip
+
+# Use a pinned version of pip-tools
+pip install pip-tools==7.1.0
+
+# Generate our requirements files in the order in which they reference each other:
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/prod.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/test.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/scripts.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/dev.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/docs.in

docker/deployable-zipfile/_build.sh

@@ -32,7 +32,7 @@ popd
 # Prepare arguments for the deployable zipfile build.
 build_args=(
     "$cfgov_refresh_volume/cfgov"
-    "$cfgov_refresh_volume/requirements/deployment.txt"
+    "$cfgov_refresh_volume/requirements/prod.txt"
     "$build_artifact_name"
     "--extra-static" "$webfonts_path"
 )

docs/index.md

@@ -35,13 +35,8 @@ Versions for most front-end packages are kept updated in the project's [package.
 Versions for back-end software including Django, Wagtail, Jinja, etc. are kept in the project's requirements files:
 https://github.com/cfpb/consumerfinance.gov/tree/main/requirements
 
-- `base.txt`: shortcut for `django.txt` + `wagtail.txt` + `libraries.txt`
-- `ci.txt`: specific requirements for the continuous integration environment. Should/could be moved to CI configuration files?
-- `deployment.txt`: requirements for deployment, includes `base.txt` and a New Relic library that we don't install anywhere else.
-- `django.txt`: specifies the Django version. The file is used when running the site, but by having it separate we can test against other versions of Django by excluding this file.
-- `docs.txt`: requirements to build the consumerfinance.gov docs.
-- `libraries.txt`: Python libraries.
-- `local.txt`: includes `base.txt` and some useful libraries when developing locally.
-- `scripts.txt`: Requirements for running certain jobs on Jenkins, so scripts can run in Jenkins without having to install all the other requirements.
-- `test.txt`: requirements for running Python tests.
-- `wagtail.txt`: specifies Wagtail version. In its own file to make it easier to test multiple versions, same as with `django.txt`.
+- `prod.in`: requirements for deployment to production and other servers
+- `test.in`: requirements for executing Python tests locally or in CI
+- `dev.in`: requirements for development work, running, and testing
+- `docs.in`: requirements to build the consumerfinance.gov docs.
+- `scripts.in`: Requirements for running our smoke test and alert polling scripts without having to install all the other requirements.

docs/installation.md

@@ -103,10 +103,10 @@ Activating the virtualenv is necessary before using it in the future as well:
 pyenv activate consumerfinance.gov
 ```
 
-Once activated, our Python CI requirements can be installed in the virtualenv:
+Once activated, our Python dev requirements can be installed in the virtualenv:
 
 ```sh
-pip install -r requirements/ci.txt
+pip install -r requirements/dev.txt
 ```
 
 !!! note "Use local for application-specific python version"
@@ -351,7 +351,7 @@ all the Python dependencies for running locally can be installed:
 
 ```sh
 pyenv activate consumerfinance.gov
-pip install -r requirements/local.txt
+pip install -r requirements/dev.txt
 ```
 
 Once complete, our `runserver.sh` script will bring up the site at

docs/related-projects.md

@@ -16,7 +16,7 @@ We have six satellite apps that are maintained outside of the consumerfinance.go
 - [teachers-digital-platform](https://github.com/cfpb/teachers-digital-platform)
 
 These satellite apps are imported into consumerfinance.gov as part of the project
-[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/libraries.txt).
+[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/prod.txt).
 
 !!! note "Thinking about making a new satellite app?"
 Satellite apps were originally built to be imported into the
@@ -49,7 +49,7 @@ want or need to test their work as part of the larger consumerfinance.gov projec
 
 The standard [installation](../installation/) process for consumerfinance.gov
 includes whatever versions of these packages are specified in project
-[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/libraries.txt).
+[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/prod.txt).
 Developers may want to temporarily or permanently replace those with a local
 copy of package source code.
 

requirements/dev.in

@@ -0,0 +1,14 @@
+-r prod.txt
+-r test.txt
+-r scripts.txt
+
+django-cprofile-middleware==1.0.5
+django-debug-toolbar==3.4.0
+django-slowtests==1.1.1
+django-sslserver==0.22
+freezegun==1.2.1
+github3.py==3.2.0
+model-bakery==1.5.0
+moto==3.1.7
+pre-commit==2.18.1
+responses==0.20.0