feat: add database performance evaluation

.github/workflows/ci.yml

@@ -46,6 +46,7 @@ jobs:
 
     env:
       DATABASE_URL: postgresql+psycopg://fastapi_user:fastapi_password@localhost:5432/fastapi_test
+      BENCHMARK_DATABASE_URL: postgresql+psycopg://fastapi_user:fastapi_password@localhost:5432/fastapi_test
       SECRET_KEY: ci-only-secret-key-with-at-least-32-bytes
       JWT_AUDIENCE: fastapi-client
       JWT_ISSUER: fastapi-production-api
@@ -81,6 +82,32 @@ jobs:
       - name: Run tests
         run: uv run pytest --cov-report=xml
 
+      - name: Run database benchmark smoke test
+        run: |
+          uv run python -m benchmarks.db prepare --records 25
+          uv run python -m benchmarks.db run \
+            --mode both \
+            --scenarios authenticated_read refresh_rotation \
+            --iterations 8 \
+            --warmup 2 \
+            --concurrency 2 \
+            --pool-size 2 \
+            --output benchmark-smoke.json
+          uv run python -m benchmarks.db verify-async-rollback
+
+      - name: Clean up database benchmark fixtures
+        if: always()
+        run: uv run python -m benchmarks.db cleanup
+
+      - name: Upload database benchmark smoke result
+        if: always()
+        uses: actions/upload-artifact@v7
+        with:
+          name: database-benchmark-smoke
+          path: benchmark-smoke.json
+          if-no-files-found: warn
+          retention-days: 14
+
       - name: Upload coverage report
         uses: actions/upload-artifact@v7
         with:

ARCHITECTURE.md

@@ -42,6 +42,7 @@ when distributed rate limiting is enabled.
 | Error boundary | `src/app/exceptions/` | Stable client errors and safe unexpected-error responses |
 | Schema evolution | `alembic/` | Ordered PostgreSQL migrations |
 | Verification | `tests/` | Endpoint, security, failure-path, and operational tests |
+| Database evaluation | `benchmarks/` | Opt-in sync baseline and isolated async prototype; never imported by production startup |
 
 Routers should remain thin: validate input, compose dependencies and services,
 and translate results into HTTP responses. Reusable authentication or database
@@ -146,6 +147,13 @@ for schema changes.
 Production releases should run migrations once before starting new application
 workers. Do not let every worker race to apply schema changes.
 
+The production persistence path remains synchronous for v1.3.0. The asyncpg
+engine under `benchmarks/` exists only for controlled comparison and preserves
+equivalent SQL and transaction boundaries. It is not an alternate application
+dependency. See [DATABASE_BENCHMARKS.md](DATABASE_BENCHMARKS.md) and
+[ADR 0001](docs/decisions/0001-keep-sync-sqlalchemy.md) for the evidence gate
+and decision rationale.
+
 In transactional delivery mode, the lifecycle token and encrypted outbox row
 commit together. Workers claim due rows in short `FOR UPDATE SKIP LOCKED`
 transactions, commit leases before SMTP I/O, and condition finalization on the

CHANGELOG.md

@@ -24,6 +24,11 @@ project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - W3C Trace Context propagation through bounded outbox metadata, JSON log
   `trace_id`/`span_id` correlation, OTLP/HTTP export, sampling controls, and
   secret-redaction safeguards
+- Reproducible PostgreSQL benchmark harness with deterministic isolated fixtures,
+  sync/async query parity, latency percentiles, pool telemetry, and JSON output
+- CI correctness smoke coverage for the asyncpg prototype and rollback semantics
+- Architecture decision retaining synchronous SQLAlchemy for v1.3.0 until
+  representative measurements justify a separately scoped migration
 
 ## [1.2.0] - 2026-08-09
 

DATABASE_BENCHMARKS.md

@@ -0,0 +1,144 @@
+# Database benchmark and async evaluation
+
+This guide defines the reproducible database evaluation used for the v1.3
+architecture decision. The production application remains synchronous. The
+async implementation under `benchmarks/` is an isolated experiment and is not
+imported by application startup or included as a console entry point.
+
+## Decision
+
+Keep synchronous SQLAlchemy for v1.3.0. The repository does not yet have
+representative production measurements showing that an async migration would
+justify its transaction, testing, debugging, and operational cost. The new
+comparison harness makes that decision repeatable; a future dedicated migration
+issue requires sustained evidence against a real workload before changing the
+production path.
+
+CI smoke timings are deliberately not treated as performance evidence. Shared
+runners are noisy and the smoke workload is too short. The decision can be
+revisited when five extended runs show a material, repeatable improvement at
+the expected concurrency without increasing error rate or weakening existing
+transaction semantics.
+
+## What is measured
+
+Both modes execute the same SQL and transaction boundaries against the same
+fixture data and PostgreSQL instance.
+
+| Scenario | Database behavior represented |
+| --- | --- |
+| `authenticated_read` | Indexed current-user lookup after token validation |
+| `session_list` | Bounded device-session list through a compound index |
+| `refresh_rotation` | Locked read and update in one rotation transaction |
+| `write` | Representative bounded update transaction |
+| `mixed` | Equal deterministic mix of the four database-bound scenarios |
+
+The harness reports requests per second, median/p95/p99/min/max latency, error
+rate and bounded error classes, connection acquisition time, checkout/checkin
+counts, and peak checked-out connections. It does not record credentials,
+queries containing user input, application tokens, or row contents.
+
+The database-focused scenarios intentionally exclude bcrypt, JWT signing,
+HTTP parsing, Redis latency, SMTP, and tracing export. Those costs should be
+measured separately by the planned end-to-end load-testing work. This separation
+prevents external dependency latency from being mistaken for a database-driver
+effect.
+
+## Safety model
+
+The tool never implicitly uses `DATABASE_URL`. Set `BENCHMARK_DATABASE_URL` or
+pass `--database-url`. It accepts only PostgreSQL on `localhost`, `127.0.0.1`,
+`::1`, or the local Compose service name `postgres`, and the database name must
+contain `test`, `bench`, `local`, or `dev`.
+
+Fixtures live only in the `fastapi_benchmark` schema. `prepare` resets that
+schema's two tables; `cleanup` drops only that schema. Never point the tool at a
+shared or production database.
+
+## Short local smoke run
+
+Start and migrate the local services, then use the test database or create a
+dedicated database whose name contains `benchmark`:
+
+```bash
+export BENCHMARK_DATABASE_URL='postgresql+psycopg://fastapi_user:fastapi_password@localhost:5432/fastapi_test'
+uv run python -m benchmarks.db prepare --records 100
+uv run python -m benchmarks.db run --mode both --scenarios authenticated_read refresh_rotation --iterations 20 --warmup 5 --concurrency 2 --pool-size 2 --output benchmark-smoke.json
+uv run python -m benchmarks.db verify-async-rollback
+uv run python -m benchmarks.db cleanup
+```
+
+PowerShell uses the same commands after setting:
+
+```powershell
+$env:BENCHMARK_DATABASE_URL='postgresql+psycopg://fastapi_user:fastapi_password@localhost:5432/fastapi_test'
+```
+
+CI runs this bounded smoke path and uploads `benchmark-smoke.json`. It verifies
+imports, fixtures, both drivers, stable result structure, and async rollback;
+it enforces no timing threshold.
+
+## Extended comparison methodology
+
+Record the following alongside every retained result artifact:
+
+- commit SHA and unchanged `uv.lock`;
+- Python 3.13 patch version, operating system, CPU model/count, and memory;
+- PostgreSQL 17 patch version, configuration, and whether it is local;
+- one application/benchmark process unless a worker comparison is explicit;
+- identical `--pool-size` and `--max-overflow` for both modes;
+- dataset size, scenario, concurrency, warm-up, iterations, and repetition;
+- tracing disabled, log level, metrics mode, and Redis excluded from DB-only runs;
+- CPU and resident-memory observations from the OS or container runtime;
+- cold-start observations kept separate from steady-state results.
+
+Recommended controlled run:
+
+1. Use a dedicated local PostgreSQL 17 database on loopback with no competing
+   workload and prepare at least 10,000 principals.
+2. Keep pool size at 5 and overflow at 0 for both drivers.
+3. For each concurrency in 1, 5, 10, 25, and 50, run all scenarios with 500
+   warm-up operations and at least 10,000 measured operations.
+4. Alternate sync-first and async-first ordering to reduce thermal/order bias.
+5. Repeat the complete matrix five times. Retain raw JSON rather than only
+   copied summary values.
+6. Report the median of run-level throughput and latency percentiles plus the
+   observed range. Investigate all errors and pool waits before comparing speed.
+
+Example extended invocation for one matrix point:
+
+```bash
+uv run python -m benchmarks.db prepare --records 10000
+uv run python -m benchmarks.db run --mode both --iterations 10000 --warmup 500 --concurrency 25 --pool-size 5 --max-overflow 0 --output results/c25-run1.json
+uv run python -m benchmarks.db cleanup
+```
+
+## Interpretation and adoption gate
+
+Throughput alone is insufficient. Compare p95/p99 latency, acquisition wait,
+peak pool use, error rate, CPU, memory, and transaction correctness. Direct
+equivalence ends above the driver/session layer: the synchronous production
+stack can use multiple worker processes, while one async process multiplexes
+database waits on an event loop. Any worker-level comparison must document that
+difference rather than presenting it as a driver-only result.
+
+Open a dedicated async-migration issue only if representative extended runs
+show a repeatable improvement large enough to matter to the service SLO (a 20%
+change is a useful review trigger, not a universal threshold), with equal error
+rate, bounded connection pressure, passing rollback/security tests, and a clear
+operational plan. Otherwise retain the simpler synchronous path or consider
+async only for a narrowly isolated I/O-heavy component.
+
+## Current comparison record
+
+| Evidence | Sync | Async prototype | Architectural weight |
+| --- | --- | --- | --- |
+| CI PostgreSQL smoke | Required and artifacted | Required and artifacted | Correctness only |
+| Transaction rollback | Existing production tests | Explicit rollback probe | Must pass |
+| Production integration | Current maintained path | Not integrated | Favors sync |
+| Representative extended measurements | Not yet supplied | Not yet supplied | No migration evidence |
+
+The absence of representative measurements is recorded explicitly rather than
+replaced with fabricated numbers. Under the issue's decision guardrails, that
+evidence supports keeping sync for v1.3.0 and deferring any full migration.
+

DEVELOPMENT.md

@@ -100,6 +100,15 @@ The test database is flushed before and after these tests. Never point
 Worker concurrency tests require PostgreSQL because SQLite does not implement
 `FOR UPDATE SKIP LOCKED`. The standard CI job runs these tests on PostgreSQL.
 
+## Database performance evaluation
+
+The opt-in benchmark compares the maintained synchronous engine with a separate
+asyncpg prototype without changing application code. It requires an explicitly
+named local/test database and owns only the `fastapi_benchmark` schema. Run the
+short smoke workflow or the controlled extended matrix in
+[DATABASE_BENCHMARKS.md](DATABASE_BENCHMARKS.md). Never use production
+credentials or infer performance regressions from CI runner timings.
+
 ## Typical contribution workflow
 
 ```bash
@@ -233,4 +242,4 @@ issuer in the key.
 Redis-backed integration tests require `REDIS_TEST_URL`. CI supplies a
 dedicated Redis database so multi-instance sharing, TTL behavior, invalidation,
 outage fallback, and refresh-lock behavior can be exercised without touching
-unrelated Redis data.
+unrelated Redis data.

README.md

@@ -77,6 +77,7 @@ contributor workflows, and local troubleshooting.
 | --- | --- |
 | [API examples](API_EXAMPLES.md) | Register, authenticate, rotate tokens, call admin routes, and inspect operations endpoints |
 | [Architecture](ARCHITECTURE.md) | Understand module boundaries, request flow, authentication, transactions, and extension points |
+| [Database benchmarks](DATABASE_BENCHMARKS.md) | Reproduce sync/async PostgreSQL comparisons and understand the v1.3 architecture decision |
 | [Local development](DEVELOPMENT.md) | Set up a checkout, run common commands, contribute, or troubleshoot locally |
 | [Deployment](DEPLOYMENT.md) | Configure a production host, release safely, terminate TLS, and operate the service |
 | [Monitoring](MONITORING.md) | Configure probes, Prometheus, multi-worker metrics, alerts, logs, and incident diagnosis |
@@ -244,6 +245,11 @@ Pytest measures statement and branch coverage for the application packages and
 fails below 90%. CI also publishes `coverage.xml` as a workflow artifact for
 review and downstream reporting.
 
+CI also runs a bounded PostgreSQL correctness smoke test for the isolated sync
+and async database benchmark and uploads its machine-readable JSON artifact.
+See [DATABASE_BENCHMARKS.md](DATABASE_BENCHMARKS.md); shared-runner timings are
+not used as performance gates.
+
 CI runs against PostgreSQL 17 rather than silently substituting SQLite. It also
 verifies that the built wheel contains and can import the application.
 

ROADMAP.md

@@ -58,7 +58,7 @@ Status: in progress
 - Redis-backed rate limiting (completed)
 - Transactional outbox and background worker processing (completed)
 - Safe OIDC discovery/JWKS caching and invalidation guidance (completed)
-- Async database evaluation and performance benchmarks
+- Async database evaluation and performance benchmarks (completed; sync retained)
 - Load-testing examples
 - OpenTelemetry tracing example (completed)
 

benchmarks/__init__.py

@@ -0,0 +1 @@
+"""Opt-in database benchmarks that are not part of the production package."""