SEP-1820: Hold Celery beat until the HTTP API answers /health

[marcuscruz-percona]

Summary

• Under python -m app.main --start-celery, beat's schedule is persisted by the sqlalchemy scheduler, so a restart dispatches anything already overdue about a second in — before uvicorn.run has opened its listening socket. A periodic task whose first act is to call SEP's own inventory or tasks API failed on connect through no fault of its own. start_celery_beat now polls the local health path until it answers 200 before constructing beat.
• The worker is deliberately left ungated: it idles harmlessly until a task arrives, so delaying it would regress startup for nothing. The gate is best-effort — a timeout is logged at ERROR and beat starts anyway, because running no periodic task at all until an operator notices is the worse failure.
• The probe lives in app/core/health.py beside the route it dials, so HEALTH_PATH cannot drift from the registered path. It uses http.client so no proxy environment variable can divert it, no redirect can lead it away from the local listener, and no request-level retry can hide the poll count. It never reads the response body and caps each attempt by the time left in the budget — draining a response refreshes the socket timeout per chunk, so a listener dribbling an endless body would otherwise hold beat well past its deadline.

Two deliberate calls worth a reviewer's eye:

• The readiness knobs (API_READINESS_TIMEOUT 60s, API_READINESS_POLL_INTERVAL 0.5s, API_READINESS_REQUEST_TIMEOUT 2s) are module constants rather than settings fields. Nothing operational is known to need them tuned per deployment; promoting them later is additive.
• Where ALLOWED_HOSTS restricts the Host header, the probe borrows a configured hostname — synthesizing one from a wildcard pattern when that is all the allow-list offers — so TrustedHostMiddleware admits it instead of answering 400 for the whole deadline. One recorded limitation: a deployment binding an IPv6 literal it also lists verbatim still gets 400, because Starlette compares host.split(":")[0], which no IPv6 literal can satisfy. That case is pinned as a test row against Starlette's own middleware rather than papered over.

Tested

• python -m app.main --start-celery with a periodic task left overdue in the DB: beat waits, logs Waiting up to 60.0s for the HTTP API at 127.0.0.1:8000 to answer /health, then is ready after Ns, and the task's first run reaches the API instead of failing on connect
• Same start with the database stopped so /health answers 503 throughout: the wait ends at the deadline, logs the ERROR with the last outcome, and beat still starts
• Ctrl-C during the wait: the beat process exits quietly with no traceback
• python -m app.main (no --start-celery) is unchanged — no wait, no new log lines
• A container start (FASTAPI_ENV=production_docker, UVICORN_HOST=0.0.0.0) still reaches beat, confirming the bind-address translation dials loopback

Checklist

• New/modified functions have type hints and rST docstrings
• New tests added for new features or bug fixes
• All tests pass locally (make test)
• Pre-commit hooks pass (make run-pre-commit)
• Database migrations generated if models changed (make makemigrations)
• User-facing changes documented (README, inline help, UI text)
• Configuration changes documented with examples
• Changelog fragment added under changelog.d/ if the change is user-facing (make changelog-add), or confirmed N/A (internal-only change, or a same-release-cycle fix for an unreleased sibling ticket)

[Copilot]

Pull request overview

This PR prevents Celery beat from dispatching overdue periodic tasks before the FastAPI server is actually reachable by adding a local readiness probe that polls GET /health and gating start_celery_beat() on that probe (best-effort, with a logged timeout fallback).

Changes:

• Add wait_for_api_ready() and supporting helpers/constants in app/core/health.py to probe the local HTTP listener safely (no proxy/redirect influence, bounded per-attempt timeout).
• Update app/main.py::start_celery_beat() to configure logging, wait for API readiness, and only then construct beat (while leaving the worker ungated).
• Add extensive unit + real-socket tests for the probe and the beat startup ordering, plus a reusable HealthProbeServer fixture, and a changelog fragment.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
app/core/health.py	Introduces the readiness probe (wait_for_api_ready) and consolidates the /health route path via HEALTH_PATH.
app/main.py	Gates Celery beat startup on the readiness probe (best-effort) and configures logging in the beat process.
tests/app/test_main.py	Adds beat readiness gate tests, including end-to-end ordering assertions using a real listener fixture.
tests/app/core/test_health.py	Adds unit + socket-level coverage for probe host/host-header resolution, polling behavior, and timeout semantics.
tests/app/conftest.py	Adds HealthProbeServer fixture to drive a real loopback /health listener for tests.
changelog.d/SEP-1820.fixed.md	Documents the user-visible behavior change in Celery beat startup.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
app
main.py
app/core
health.py
Project Total

This report was generated by python-coverage-comment-action

[marcuscruz-percona]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Done in 690c47d — I merged main into this branch and resolved the conflict in tests/app/conftest.py.