Persist RunPod job id and reattach to in-flight jobs after a daemon interruption

[quevon24]

Closes #127.

Problem

When the daemon is killed mid-pipeline (SIGTERM on deploy/eviction, SIGKILL/OOM) or a scan times out, the in-flight scan is re-queued and the pipeline re-runs. The RunPod job_id lived only as a local variable inside runpod_client._submit/_poll, so on restart the daemon couldn't tell a job was already running. Two costs, both seen live on scan 1940 (2026-07-20): the killed daemon never cancelled the in-flight job (only the timeout path did), so it ran orphaned for 57m until cancelled by hand, while the re-queue submitted a duplicate job that ran concurrently; and detect/OCR was redone from scratch even when the previous job had finished.

Fix

Persist the RunPod job handle on the Scan and, on re-entry, reattach to the existing job instead of submitting a duplicate.

• Persist + reattach. New runpod_job_id / runpod_job_action / runpod_job_submitted_at fields. _invoke records the id after submit and clears it on any terminal outcome; on a hard kill it stays set, so the next tick reattaches: a COMPLETED job's result is reused (GPU step skipped), an IN_QUEUE/IN_PROGRESS job is picked up without duplicating. A job aged out of RunPod's retention window returns 404, which the existing _poll path turns into a re-queue that submits fresh, so recovery is always safe.
• Cancel orphaned jobs on shutdown. Scans that the shutdown flags ERROR_INTERRUPTED are terminal and will never reattach, so their jobs are cancelled to stop billing and their persisted handle is cleared (so a later re-queue or a frontend re-detect/revalidate submits fresh rather than reattaching a dead job). Re-queued scans keep their handle so they still reattach.

Follow-ons in this PR

• Page-aware request timeout. The wall-clock ceiling now scales with page count (RUNPOD_REQUEST_TIMEOUT + RUNPOD_REQUEST_TIMEOUT_PER_PAGE * page_count), so a large volume (e.g. 1300 pages x 3 detect models) no longer false-times-out against a flat base. A genuine timeout stays terminal with no auto-retry.
• Aligned stale cutoff. _recover_stale uses a per-scan threshold of max(DAEMON_PROCESSING_TIMEOUT, 2x the page-aware ceiling) instead of a single global timeout, so raising the request timeout can't leave stale recovery cutting off a large scan whose job is still legitimately running.
• Skip re-detection on resume. _run_yolo pushes detections.json to S3 the moment detection finishes, and run_full_pipeline skips detection when it already exists (restored from S3 on a resume), mirroring the existing OCR-PDF guard. So an interruption during a later stage no longer redoes detection.
• Frontend visibility. An error banner on the scan list surfaces ERROR scans (likely timeouts/failures), linking superusers to the admin changelist filtered to ERROR for review and re-queue, alongside the existing retry-cap and interrupted banners.

Notes on behavior

• The biggest win (skip completed GPU work) is realized for graceful SIGTERM shutdowns, which re-queue immediately. Hard kills (SIGKILL/OOM) wait for stale recovery, by which time a COMPLETED result may have aged out (30-min retention), so they mostly get the "don't duplicate a running job" win.
• Deliberate regenerate actions (re-detect, revalidate, reprocess) are unaffected: they run through separate functions that submit unconditionally, and reviewable scans have their handle cleared, so they always submit fresh.
• A successful reattach does not un-bump interruption_count; the interruption still happened and the cap is about daemon churn, not job outcome.

Testing

DEVELOPMENT=True DB_HOST=localhost DB_SSL_MODE=prefer python manage.py test scanning.tests — 353 tests pass. New coverage: persist/clear/reattach across every /status state, cancel_job, page-aware timeout, aligned stale cutoff, detect stage-skip and S3 persist, cancel-only-flagged on shutdown with handle clearing, and the error banner.