Reorganize Python deps and use pip-compile

.github/workflows/backend.yml

@@ -5,20 +5,23 @@ on:
     paths:
       - '**.py'
       - 'requirements/**.txt'
+      - 'requirements/**.in'
       - '**.html'
       - '**.mo'
       - '**.po'
   pull_request:
     paths:
       - '**.py'
       - 'requirements/**.txt'
+      - 'requirements/**.in'
       - '**.html'
       - '**.mo'
       - '**.po'
   merge_group:
     paths:
       - '**.py'
       - 'requirements/**.txt'
+      - 'requirements/**.in'
       - '**.html'
       - '**.mo'
       - '**.po'
@@ -34,6 +37,7 @@ jobs:
           - validate-migrations
           - validate-assets
           - validate-translations
+          - validate-requirements
           - unittest
 
     services:
@@ -71,7 +75,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements/ci.txt
+          pip install -r requirements/test.txt
 
       - name: Run tox -e ${{ matrix.toxenv }}
         run: |
@@ -110,7 +114,7 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements/ci.txt
+          pip install -r requirements/test.txt
 
       - name: Retrieve backend coverage
         uses: actions/download-artifact@v3

.github/workflows/functional-tests.yml

@@ -56,7 +56,7 @@ jobs:
     - name: Install dependencies
       run: |
         python3 -m pip install --upgrade pip
-        python3 -m pip install -r requirements/local.txt
+        python3 -m pip install -r requirements/dev.txt
 
     - name: Set up initial data
       run: ./refresh-data.sh test.sql.gz

Dockerfile

@@ -46,10 +46,6 @@ ENV ALLOWED_HOSTS '["*"]'
 # .backend-deps and .frontend-deps are required to run the application
 RUN apk add --no-cache --virtual .backend-deps bash curl postgresql
 
-# Install python requirements
-COPY requirements requirements
-RUN cp -Rfp /build/* /usr/local && rm -Rf /build && pip install -r requirements/local.txt
-
 EXPOSE 8000
 
 ENTRYPOINT ["./docker-entrypoint.sh"]

backend.sh

@@ -22,7 +22,7 @@ install() {
   echo 'Installing back-end dependencies...'
 
   # Install requirements for Django Server or tox.
-  pip install -r ./requirements/local.txt
+  pip install -r ./requirements/dev.txt
 }
 
 init "$1"

compile-requirements.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# ==========================================================================
+# Compile full and hashed set of dependencies from requirements/*.in files
+# NOTE: Run this script while in the project root directory.
+#       It will not run correctly when run from another directory.
+# ==========================================================================
+
+set -e
+
+# Always use the latest pip
+pip install -U pip
+
+# Use a pinned version of pip-tools
+pip install pip-tools==7.1.0
+
+# Generate our requirements files in the order in which they reference each other:
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/deployment.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/test.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/scripts.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/dev.in
+pip-compile --generate-hashes --resolver=backtracking --rebuild -r requirements/docs.in

docs/debugging-monitoring.md

@@ -57,19 +57,6 @@ This project can be configured for real-time monitoring with
 [New Relic](https://newrelic.com/). This requires an active New Relic account
 and a valid license key.
 
-First, you'll need to install
-[the New Relic Python Agent](https://docs.newrelic.com/docs/agents/python-agent)
-from
-[from PyPI](https://pypi.org/project/newrelic/).
-This package is not part of the standard development requirements but is
-included in
-the [`requirements/deployment.txt`](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/deployment.txt) file.
-It can be installed into your Python environment using the following command:
-
-```shell
-$ pip install newrelic
-```
-
 New Relic monitoring is implemented by wrapping every request to the Django
 web server with a call to the Python agent. This agent records information
 about web requests and reports it back to New Relic's servers. Additionally,
@@ -103,12 +90,7 @@ file contains placeholder entries for these two variables.
 Once you've set these two variables, start or restart your local web server.
 
 When you make your first web request, you'll see messages in the console
-indicating that the New Relic Python agent has been activated. You should also
-see a message containing a link to the New Relic console:
-
-```txt
-Reporting to: https://rpm.newrelic.com/accounts/XXXXXXXX/applications/XXXXXXXX
-```
+indicating that the New Relic Python agent has been activated.
 
 You should now be able to use that link to navigate to the New Relic console
 and, after a few seconds, see your web server traffic in New Relic APM and

docs/index.md

@@ -35,13 +35,8 @@ Versions for most front-end packages are kept updated in the project's [package.
 Versions for back-end software including Django, Wagtail, Jinja, etc. are kept in the project's requirements files:
 https://github.com/cfpb/consumerfinance.gov/tree/main/requirements
 
-- `base.txt`: shortcut for `django.txt` + `wagtail.txt` + `libraries.txt`
-- `ci.txt`: specific requirements for the continuous integration environment. Should/could be moved to CI configuration files?
-- `deployment.txt`: requirements for deployment, includes `base.txt` and a New Relic library that we don't install anywhere else.
-- `django.txt`: specifies the Django version. The file is used when running the site, but by having it separate we can test against other versions of Django by excluding this file.
-- `docs.txt`: requirements to build the consumerfinance.gov docs.
-- `libraries.txt`: Python libraries.
-- `local.txt`: includes `base.txt` and some useful libraries when developing locally.
-- `scripts.txt`: Requirements for running certain jobs on Jenkins, so scripts can run in Jenkins without having to install all the other requirements.
-- `test.txt`: requirements for running Python tests.
-- `wagtail.txt`: specifies Wagtail version. In its own file to make it easier to test multiple versions, same as with `django.txt`.
+- `deployment.in`: requirements to run consumerfinance.gov in any environment
+- `test.in`: requirements for executing Python tests locally or in CI
+- `dev.in`: requirements for development work, running, and testing
+- `docs.in`: requirements to build the consumerfinance.gov docs
+- `scripts.in`: Requirements for running our smoke test and alert polling scripts without having to install all the other requirements

docs/installation.md

@@ -102,10 +102,10 @@ Activating the virtualenv is necessary before using it in the future as well:
 pyenv activate consumerfinance.gov
 ```
 
-Once activated, our Python CI requirements can be installed in the virtualenv:
+Once activated, our Python dev requirements can be installed in the virtualenv:
 
 ```sh
-pip install -r requirements/ci.txt
+pip install -r requirements/dev.txt
 ```
 
 !!! note "Use local for application-specific python version"
@@ -353,7 +353,7 @@ all the Python dependencies for running locally can be installed:
 
 ```sh
 pyenv activate consumerfinance.gov
-pip install -r requirements/local.txt
+pip install -r requirements/dev.txt
 ```
 
 Once complete, our `runserver.sh` script will bring up the site at

docs/related-projects.md

@@ -11,12 +11,10 @@ set up the full website.
 We have six satellite apps that are maintained outside of the consumerfinance.gov codebase:
 
 - [ccdb5-api](https://github.com/cfpb/ccdb5-api)
-- [ccdb5-ui](https://github.com/cfpb/ccdb5-ui)
 - [owning-a-home-api](https://github.com/cfpb/owning-a-home-api)
-- [teachers-digital-platform](https://github.com/cfpb/teachers-digital-platform)
 
 These satellite apps are imported into consumerfinance.gov as part of the project
-[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/libraries.txt).
+[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/deployment.in).
 
 !!! note "Thinking about making a new satellite app?"
 
@@ -50,7 +48,7 @@ want or need to test their work as part of the larger consumerfinance.gov projec
 
 The standard [installation](../installation/) process for consumerfinance.gov
 includes whatever versions of these packages are specified in project
-[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/libraries.txt).
+[requirements files](https://github.com/cfpb/consumerfinance.gov/blob/main/requirements/deployment.in).
 Developers may want to temporarily or permanently replace those with a local
 copy of package source code.
 

requirements/libraries.txt → requirements/deployment.in

@@ -1,7 +1,7 @@
-# These packages are installed from PyPI.
 base36==0.1.1
 beautifulsoup4==4.8.2
 boto3==1.22.7
+django==3.2.23
 django-autocomplete-light==3.9.4
 django-axes==5.41.0
 django-cors-headers==3.11.0
@@ -22,13 +22,15 @@ elasticsearch<7.11  # Keep pinned to the deployed ES version
 govdelivery==1.4.0
 Jinja2==3.1.2
 lxml==4.9.1
+newrelic==8.5.0
 opensearch-py==2.2.0
 psycopg2-binary==2.8.6
 python-dateutil==2.8.2
 regdown==1.0.7
 requests-aws4auth==1.1.2
 s3transfer==0.5.2
 setuptools>=65.5.1
+wagtail==4.2.4
 wagtail-autocomplete==0.9.0
 wagtail-flags==5.3.0
 wagtail-inventory==2.3