[Feat] Durable retry scheduling: parked Fast turns resume at their retry time

[daniel-lxs]

Summary

Phase 2 of the durable Fast-turn execution work (Phase 1 was #2016): inference retry backoff no longer lives in a setTimeout owned by the API process.

When a replay-safe Fast turn hits a retryable inference failure, the owning process now:

1. records the retry notice as before (hidden marker, or a visible "Retrying in Ns" post once the silent window is exceeded),
2. parks the turn on its durable row: retry_at = the backoff deadline, inference_retries = retries consumed so far, claim released,
3. queues a delayed BullMQ wakeup for retry_at and exits without an outcome.

The parent-event queue re-runs the same turn at retry_at on whichever process is alive then. The resumed run continues the same bounded retry budget (per-turn cap holds across handoffs) and inherits the active retry notice for its turn, so the eventual answer edits that notice in place instead of reconciling it into an interruption and posting beside it. A deploy or crash during backoff therefore needs nothing special: the row waits, the recovery sweep re-adds the delayed wakeup if Redis lost it, and the next owner picks it up.

Turns that already performed a non-replayable action (task launch, message, memory write, mutating MCP call, closeout) keep their in-process backoff exactly as before; they cannot be re-run elsewhere until the Phase 3 side-effect journal exists.

Changes

• fast_agent_parent_events: nullable retry_at, inference_retries (default 0). Migration 0072. The previous release ignores both columns; a rollback would run a parked retry immediately rather than at its time, which is safe (N-1).
• Repository: scheduleFastAgentDurableTurnRetry (pending-row guarded), findFastAgentActiveInferenceRetryNotice (for the resumed run), and the expired-lease reconciler skips conversations whose inline row has a future retry_at or a live claim.
• Queue: pending predicate excludes rows before their retry_at; wakeFastAgentParentEventAt adds a delayed job keyed by event key + time; the recovery sweep re-adds delayed wakeups for scheduled rows; the drain passes resumedAfterInferenceRetry, the consumed retry count, and both wake hooks to the resumed run (this also closes a Phase 1 gap: worker-side resumed runs had no wake hook).
• Service: deferRetry step in the retry loop after the notice is recorded; FastAgentDurableRetryScheduledError ends the execution without settling the row, the notice, or the responding lease; resumed runs seed the retry budget and adopt the predecessor's notice; new <resumed_turn> marker text for the retry case.
• Service, OpenCode-owned backoff: on develop OpenCode runs the provider retries itself (3 attempts) and marks the final error non-retryable, so Roomote's outer loop rarely sees 5xx or network failures; the in-process wait a restart interrupts is OpenCode's internal backoff. When OpenCode reports a scheduled retry with its next attempt time, a replay-safe durable turn now parks at that time and aborts the prompt with the park as its reason. Without this the feature would not fire in practice.
• Retry notice continuity across parks: the marker keeps the episode's first-failure timestamp (so the 30s silent window counts across handoffs) and keeps its platform message id and visibility on every write (the upsert replaces metadata), so exactly one visible notice appears and the eventual answer edits it in place.
• Queue-side adapters: the parent-turn adapters the queue runs resumed turns through had no replaceReply and their Slack/Discord postReply returned no handle. Both are fixed; the per-surface edit-in-place replacers now live in fast-agent-reply-replacement.ts and are shared with the surface-reply builder (which gains a Discord replacer). The webhook handlers keep their inline versions.
• Retry policy for parked turns: the first short in-process retry stays in place (a one-off blip is cheapest to ride out where it happened; a wait of 10s or more parks at once), parks start from the second attempt, park delays back off across handoffs (2s, 4s, 8s, ... capped at 60s, never shorter than a provider-issued wait), and the episode is bounded by a 15-minute horizon from the first failure plus a safety cap of 30 handoffs. Past that, OpenCode's normal in-process retries run and the honest terminal failure is posted. The in-process retry budget is no longer seeded from the row.
• Drain: a resumed run settles its own row (the drain no longer stamps delivered over a replay-withdrawn row), and a run that parks itself ends the drain quietly instead of failing the BullMQ job.
• Handlers (Slack, Discord, web, shared surface reply) bind requestDurableRetry next to requestDurableResume; Slack and web log the parked turn at info level instead of as a failure.
• Kill switch: R_FAST_DURABLE_RETRY_DISABLED (implied by R_FAST_DURABLE_ADMISSION_DISABLED).

Tests

• Service: parks a replay-safe turn (schedule + wakeup, nothing settled), keeps the retry in process after a non-replayable action, falls back in process when the schedule does not land, continues the inherited budget on a resumed run, edits the inherited notice instead of reconciling it, retry-resume marker.
• Repository (real DB): schedule guarded by pending row, notice lookup, reconciler skips scheduled retries and reconciles once the row settles.
• Queue: drain passes retry resumption + hooks, pending predicate honors retry_at, recovery re-adds delayed wakeups.

Smoke (local, per the PR smoke-testing skill)

Setup: PM2 stack from this branch at 6b45d46d (watchman rebuilds verified by PID changes), mock Slack harness, and a local fault proxy in front of OpenRouter (provider.openrouter.options.baseURL via OPENCODE_CONFIG_CONTENT) that injects 503, 429 with Retry-After: 45, or socket resets on chat completions from a mode file. Evidence layers: mock Slack thread state, fast_agent_parent_events row, fast_agent_messages rows, API and worker logs.

• Baseline (pass-through): two provider calls through the proxy, pong in the thread, row settled.

• Park and resume under persistent 503: the API parked on the first OpenCode retry event (claim released, retry_at about 2s out, inference_retries 1); the worker's delayed job resumed the turn on time and parked again with the count climbing 2, 3, 4, 5 (each visible as Parked the turn in the worker log); after lifting the fault at 5 the resumed run answered pong, the hidden marker was rewritten as the answer, row delivered. One visible transcript row.

• Restart during backoff (429, 45s): visible notice posted at once; both the API and worker runtimes were SIGKILLed 1s after the park and restarted; the resumed run started at exactly retry_at (derived from its own duration fields), the notice was edited in place to pong (same Slack ts), one visible transcript row, row settled.

• Budget exhaustion (503 never lifted): 12 parks at the per-turn cap, then OpenCode's in-process retries ran and the honest terminal failure replaced the marker; no interruption text, no loop.

• Notice continuity across parks (503, lifted at 6): hidden marker for parks 1 to 3, one visible notice at park 4 once the cumulative wait passed 30s, its message id preserved through parks 5 and 6, and the answer edited it in place; thread ends with a single pong, one visible transcript row. (Two earlier runs of this scenario found and fixed the missing adapter handle and the metadata wipe described above.)

• Final policy run (503, lifted at park 4): first retry absorbed in process with no park, parks 1 to 4 scheduled 2s, 4s, 8s, 16s apart, one visible notice once the cumulative wait passed 30s, the resumed run after the fault lifted answered by editing that notice, single transcript row, row settled.

Not exercised live: the non-replayable branch (a turn that already launched or messaged keeps its in-process backoff) and Discord, Teams, Telegram, web surfaces; those are covered by unit tests only. No browser pass (dev login is not enabled on this checkout); transcript visibility was verified from the canonical rows.

Kill switches

R_FAST_DURABLE_RETRY_DISABLED=true keeps backoff in process (parks stop, everything else unchanged). R_FAST_DURABLE_ADMISSION_DISABLED=true implies it.

[roomote-community]

No code issues found. See task

• apps/api/src/handlers/discord/fast-agent.ts:333 treats a successfully parked durable retry as a failed Fast response.

_{Reviewed 1fe9f55}