Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate db migrations #144

Closed
wants to merge 49 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
c0621f7
Remove code not currently in use on feature branch
steventux Jan 6, 2025
b12ece2
Add Flask dependencies
steventux Jan 6, 2025
df390a2
Add basic Flask app
steventux Jan 6, 2025
51b3f02
Add function app which delegates requests to Flask
steventux Jan 6, 2025
99bc84d
Test Flask app in CI
steventux Jan 6, 2025
62883dc
Revise Dockerfile/compose.yml for one function app with db
steventux Jan 7, 2025
52f7f9e
alembic init
dnimmo Jan 9, 2025
6403908
Add Flask dependencies
steventux Jan 6, 2025
4757e84
Add basic Flask app
steventux Jan 6, 2025
b40ad35
Add function app which delegates requests to Flask
steventux Jan 6, 2025
557764d
Test Flask app in CI
steventux Jan 6, 2025
6f47c02
Revise Dockerfile/compose.yml for one function app with db
steventux Jan 7, 2025
8e618ff
added alembic for migrations
dnimmo Jan 14, 2025
7148b32
alembic init
dnimmo Jan 9, 2025
3af3ccf
added alembic for migrations
dnimmo Jan 14, 2025
b87b43e
updated lockfile
dnimmo Jan 14, 2025
c53cc53
Add pytest-sugar
steventux Jan 14, 2025
b8d65b1
Add request verifier
steventux Jan 14, 2025
f02ddb9
Access headers with lowercase keys
steventux Jan 14, 2025
5e96744
Add POST status create endpoint
steventux Jan 14, 2025
fe893c3
Add jsonschema validation library
steventux Jan 14, 2025
8364991
Validate request body data against NHS Notify schema
steventux Jan 15, 2025
803059a
Move status endpoints to route handlers
steventux Jan 15, 2025
d6209cd
Update name of healthcheck route function to match path
steventux Jan 15, 2025
ea4fbdd
Merge pull request #140 from NHSDigital/add-status-endpoint-to-flask
steventux Jan 15, 2025
2bef605
Ensure local test runs use test db
steventux Jan 16, 2025
5a8f46f
Add status recorder service
steventux Jan 16, 2025
e1126d1
Use psycopg2 sql.Identifier to safely interpolate the table name
steventux Jan 16, 2025
9722944
Add FLASK_DEBUG env vars
steventux Jan 16, 2025
acf0110
Extract hmac signature generation
steventux Jan 16, 2025
24ffac4
Read the post body as json and ensure key sorting is consistent
steventux Jan 16, 2025
372eb62
Truncate all tables after each test
steventux Jan 16, 2025
4263a47
Return useful error descriptions in the status/create response
steventux Jan 16, 2025
ec5d825
Add postgres to CI test jobs
steventux Jan 16, 2025
a427e07
PR comments
dnimmo Jan 17, 2025
8a8f8e3
Merge branch 'feature-flask-and-function-apps' of https://github.com/…
dnimmo Jan 20, 2025
385c076
file formatting fixes for auto-generated alembic init
dnimmo Jan 20, 2025
9ef879a
removed schema.hcl
dnimmo Jan 20, 2025
8d415f5
re-add schema.hcl
dnimmo Jan 20, 2025
df1c138
DTOSS-6505 update Pipfile
dnimmo Jan 20, 2025
e0ef3d4
replace print with logger
dnimmo Jan 20, 2025
97472a4
Refactor persistence tests
steventux Jan 20, 2025
afa461f
Merge pull request #142 from NHSDigital/persist-status-callback-requests
steventux Jan 21, 2025
46476d6
Merge pull request #139 from NHSDigital/DTOSS-6505
dnimmo Jan 21, 2025
a2a5763
Add sqlalchemy_utils dependency
steventux Jan 21, 2025
c2ae0cb
Add alembic_postgresql_enum dependency
steventux Jan 21, 2025
a6f87ff
Refactor schema for batch message endpoint
steventux Jan 21, 2025
4ae3227
Remove schema initialisation from database module
steventux Jan 22, 2025
86dff02
Run alembic upgrade head on test CI dbs
steventux Jan 22, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revise Dockerfile/compose.yml for one function app with db
steventux committed Jan 9, 2025

Verified

This commit was signed with the committer’s verified signature.
steventux Steve Laing
commit 6f47c02a9b25c69a11fa200570cd41679de4dd6f
142 changes: 5 additions & 137 deletions compose.yml
Original file line number Diff line number Diff line change
@@ -1,38 +1,6 @@
name: communication-management

services:
# External Dependencies
azurite:
container_name: azurite
image: mcr.microsoft.com/azure-storage/azurite
command: "azurite --loose --blobHost 0.0.0.0 --blobPort 10000 --queueHost 0.0.0.0 --queuePort 10001"
ports:
- 10000:10000
- 10001:10001
- 10002:10002
profiles:
- test-end-to-end
- test-integration
- dev
networks:
- frontend

azurite-setup:
container_name: azurite-setup
build:
context: ./
dockerfile: ./dependencies/azurite/Dockerfile
network_mode: host
depends_on:
- azurite
environment:
- AZURITE_CONNECTION_STRING=${AZURITE_CONNECTION_STRING}
- BLOB_CONTAINER_NAME=${BLOB_CONTAINER_NAME}
profiles:
- test-end-to-end
- test-integration
- dev

db:
image: postgres
healthcheck:
@@ -52,130 +20,30 @@ services:
volumes:
- ./pgdata:/var/lib/postgresql/pgdata
profiles:
- test-end-to-end
- test-integration
- dev

end-to-end-tests:
container_name: end-to-end-tests
network_mode: host
build:
context: ./tests/end_to_end/
additional_contexts:
root_dir: .
shared_dir: ./src/shared/
dockerfile: Dockerfile
environment:
- AZURITE_CONNECTION_STRING=${AZURITE_CONNECTION_STRING}
- BLOB_CONTAINER_NAME=${BLOB_CONTAINER_NAME}
- DATABASE_HOST=${DATABASE_HOST}
- DATABASE_NAME=${DATABASE_NAME}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_SSLMODE=${DATABASE_SSLMODE}
- DATABASE_USER=${DATABASE_USER}
profiles:
- test-end-to-end
depends_on:
azurite-setup:
condition: service_completed_successfully
azurite:
condition: service_started
db:
condition: service_healthy
notify:
condition: service_started
process-pilot-data:
condition: service_started
volumes:
- ./log/functions/notify:/tests/end_to_end/log/functions/notify
- ./log/functions/process-pilot-data:/tests/end_to_end/log/functions/process-pilot-data

# Functions
message-status:
container_name: message-status
network_mode: host
build:
context: ./src/functions/message_status/
additional_contexts:
root_dir: .
dockerfile: Dockerfile
environment:
- AzureWebJobsStorage=UseDevelopmentStorage=true
- ASPNETCORE_URLS=http://*:7073
- APPLICATION_ID=${APPLICATION_ID}
- NOTIFY_API_KEY=${NOTIFY_API_KEY}
- DATABASE_HOST=${DATABASE_HOST}
- DATABASE_NAME=${DATABASE_NAME}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_SSLMODE=${DATABASE_SSLMODE}
- DATABASE_USER=${DATABASE_USER}

notify:
container_name: notify
network_mode: host
build:
context: ./src/functions/notify/
context: ./src/notify
dockerfile: Dockerfile
additional_contexts:
root_dir: .
dockerfile: Dockerfile
environment:
- AzureWebJobsStorage=UseDevelopmentStorage=true
- APPLICATION_ID=${APPLICATION_ID}
- ASPNETCORE_URLS=http://*:7072
- DATABASE_HOST=${DATABASE_HOST}
- DATABASE_NAME=${DATABASE_NAME}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_SSLMODE=${DATABASE_SSLMODE}
- DATABASE_USER=${DATABASE_USER}
- NOTIFY_API_KEY=${NOTIFY_API_KEY}
- NOTIFY_API_URL=${NOTIFY_API_URL}
- OAUTH2_API_KEY=${OAUTH2_API_KEY}
- OAUTH2_API_KID=${OAUTH2_API_KID}
- OAUTH2_TOKEN_URL=${OAUTH2_TOKEN_URL}
- PRIVATE_KEY=${PRIVATE_KEY}
volumes:
- ./log/functions/notify:/tmp/Functions/Host

process-pilot-data:
container_name: process-pilot-data
network_mode: host
build:
context: ./src/functions/process_pilot_data/
additional_contexts:
root_dir: .
dockerfile: Dockerfile
environment:
- AzureWebJobsStorage=UseDevelopmentStorage=true
- ASPNETCORE_URLS=http://*:7071
- FUNCTIONS_WORKER_RUNTIME=python
- BLOB_CONTAINER_NAME=${BLOB_CONTAINER_NAME}
- NOTIFY_FUNCTION_URL=${NOTIFY_FUNCTION_URL}
volumes:
- ./log/functions/process-pilot-data:/tmp/Functions/Host

integration-tests:
container_name: integration-tests
network_mode: host
build:
context: ./tests/integration/
additional_contexts:
root_dir: .
shared_dir: ./src/shared/
dockerfile: Dockerfile
environment:
- DATABASE_HOST=${DATABASE_HOST}
- DATABASE_NAME=${DATABASE_NAME}
- DATABASE_PASSWORD=${DATABASE_PASSWORD}
- DATABASE_SSLMODE=${DATABASE_SSLMODE}
- DATABASE_USER=${DATABASE_USER}
- NOTIFY_API_URL=${NOTIFY_API_URL}
profiles:
- test-integration
depends_on:
db:
condition: service_healthy

networks:
frontend:
# Specify driver options
driver: bridge
driver_opts:
com.docker.network.bridge.host_binding_ipv4: "127.0.0.1"
- ./log/function/communication-management:/tmp/Functions/Host
23 changes: 23 additions & 0 deletions src/notify/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# To enable ssh & remote debugging on app service change the base image to the one below
# FROM mcr.microsoft.com/azure-functions/python:4-python3.11-appservice
FROM mcr.microsoft.com/azure-functions/python:4-python3.11

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
AzureWebJobsFeatureFlags=EnableWorkerIndexing \
APPLICATION_ID=${APPLICATION_ID} \
DATABASE_HOST=${DATABASE_HOST} \
DATABASE_USER=${DATABASE_USER} \
DATABASE_PASSWORD=${DATABASE_PASSWORD} \
DATABASE_NAME=${DATABASE_NAME} \
FUNCTIONS_WORKER_RUNTIME=python \
NOTIFY_API_KEY=${NOTIFY_API_KEY} \
NOTIFY_API_URL=${NOTIFY_API_URL}

COPY --from=root_dir Pipfile /
COPY --from=root_dir Pipfile.lock /
RUN pip install pipenv
RUN pipenv install --system

COPY . /home/site/wwwroot
COPY --from=root_dir ./database/schema.sql /home/site/wwwroot/database/schema.sql