[Feat] Durable admission: interrupted Fast turns resume on the next process by daniel-lxs · Pull Request #2016 · RooCodeInc/Roomote

daniel-lxs · 2026-09-01T22:19:20Z

Problem

An ordinary human Fast turn exists only in the memory of the API process that accepted it. When that process is restarted or loses its conversation lock, the turn is gone: the user gets a resend closeout and has to ask again. Deploys are the most common trigger. #2004, #2006, #2007, and #2010 shrank the blast radius (attribution, drain, no false stamps, the next nudge resumes the request), but nothing made the turn itself survive the process. This is Phase 1 of the durable-execution plan: durable admission with the accepting process as first claimant.

Change

Every human Fast turn on every surface (Slack, Discord, web, Teams, Telegram) is persisted as an inline-admitted fast_agent_parent_events row under a claim lease before the accepting process runs it. The existing parent-event queue, which already runs busy follow-ups through the same turn path on every surface, becomes the recovery executor.

Persist before run. persistFastAgentInlineHumanTurn (and admitFastAgentHumanFollowUp on its idle path) writes the row with admission='inline' and a 15-minute claimed_until in one transaction with the retirement of any older pending inline row for the conversation (a newer human message supersedes an interrupted one). The queue is not woken; the live owner runs the turn.
Claim lease. The turn renews the claim on the responding-lease tick. Drain and recovery sweep skip rows whose claim is current, so a live turn is never run twice.
Replay safety, recorded before the act. The turn revokes replay durably (row discarded) immediately before the first action a re-run could duplicate: task launch, task message, cancel, memory write, any terminal closeout, and any MCP call that is not a known Roomote read. Acks, progress notes, reactions, skill loads, and spill reads keep replay open. If the revocation write does not land, or the row is no longer pending, the action is refused rather than run unrecorded, and a skipped terminal closeout hands the turn back to the queue instead of settling it.
Interruption hands off instead of failing. On a restart or lost lock while still replay-safe, the turn releases its claim, wakes the queue, and posts nothing. The shutdown path also releases the claim of every bound row, so a turn interrupted before it reached inference is handed off too. The queue re-runs the turn through the existing delivery path with resumedAfterInterruption (a <resumed_turn> marker plus a prompt rule not to re-acknowledge) and with the row as its own durable admission, so the resumed run fences itself the same way. Deliberate cancellations and post-action interruptions keep today's behavior (honest restart closeout, row discarded).
Kill switch. R_FAST_DURABLE_ADMISSION_DISABLED=true stops persisting turns; everything else goes inert and turns run exactly as before this PR.
The turn's own human-steer poll ignores inline-admitted rows (they are whole turns, never steers).

Schema: two nullable columns on fast_agent_parent_events (additive, N-1 safe). No new tables. Also includes a mock Slack harness fidelity fix: unknown Web API methods now return ok:false unknown_method like Slack instead of a 404 the WebClient retries for minutes.

Validation

Unit: cloud-agents fast-agent suites, SDK queue/admission/delivery/surface-reply suites, api Slack+Discord handler suites, web fast-sessions suites, all green; lint:fast, check-types:fast, knip pass. Six Roomote review rounds, each closing a real gap in the fail-closed chain.

Local smoke (exact head, PM2 from this checkout verified by cwd, mock Slack as the only external boundary with the real API, Postgres, Redis turn lock, BullMQ worker and recovery sweep in path, owner PID validated by port, command line, and cwd before every signal, drain set to 0 so SIGTERM is the deploy transition):

SIGTERM mid-inference: claim released and queue woken in 7s, worker re-ran the turn, exactly one reply in the thread (the real answer), no restart closeout, row settled.
SIGTERM during setup, before inference: found a gap (claim stayed held); fixed here by releasing bound claims from the shutdown path. Re-run: claim released in 1s, the real 60s sweep resumed it 36s later, one correct answer.
SIGKILL (hard crash): row stayed claimed and untouched; after expiring the claim and the Redis turn lock by hand (standing in for the lease TTLs), the real sweep resumed it in 54s with one answer.
Non-replayable turn (memory write, then SIGTERM): replay revoked before the tool ran, honest restart closeout posted with interruptionReason=api_shutdown, nothing resumed.
Continuation and UI: follow-ups in recovered threads answered by the restarted API; the recovered session renders the prompt plus one answer with status ready.

Not exercised

A staging deploy-mid-turn pass; it should run on the first deploy of this change.
Killed API or worker runtimes do not release their Redis turn locks, so queued work in those conversations waits up to the 600s lock TTL after a hard crash. By design today; owner-aware leases are the next step.
Owner/generation fencing on the session lease (from the [Fix] Live Fast turns get stamped as interrupted after 15 minutes #2007 review) is still future work.

roomote-community · 2026-09-01T22:19:36Z

No code issues found. See task

Make inline admission and supersession atomic so recovery cannot run an older superseded turn.
Fail closed when durable replay revocation cannot be persisted before a side effect.
Preserve replay protection when the queue resumes an interrupted inline turn.
Admit the first human web Fast turn durably, not only later replies.
Revoke durable replay before every terminal closeout, including cancellation retry-notice replacements, so recovery cannot duplicate replies.
Keep the row recoverable when terminal revocation fails during a queue-resumed turn; the queue must not immediately mark the released row delivered.
Do not discard a durable turn when the new acknowledgement gate rejects a pre-ack action (packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts:2471).
Wake the durable parent-event queue after shutdown releases a claim before the turn reaches answer handling (packages/cloud-agents/src/server/fast-agent/fast-agent-turn-lock.ts:80).

_{Reviewed 4226a14}

Every human Fast turn (Slack, Discord, web, Teams, Telegram) is now persisted as an inline-admitted parent-event row under a claim lease before the accepting process runs it. The turn renews the claim as it works, revokes replay durably before the first action a re-run could duplicate (task launches, task messages, cancellations, memory writes, closeouts, non-read MCP calls), and settles the row on completion. When the process is interrupted by a restart or a lost conversation lock while the turn is still replay-safe, it releases the claim and posts nothing; the parent-event queue re-runs the turn through the same path busy follow-ups already use, with a resumed-turn marker so the model continues without re-acknowledging. A newer human message supersedes a pending interrupted turn. Interruptions after a non-replayable action keep the honest restart closeout.

…doff The turn's own human-steer poll treated its inline-admitted row as a pending follow-up, injected it as a steer, and settled it, leaving an interruption with nothing to resume. Inline rows are whole turns and are now excluded from steering. The abort path also logs whether the claim release and queue wake landed, which is what proves a handoff in production logs.

- The resumed queue run now receives its row as durableAdmission, so it revokes replay before any non-replayable action and a worker death after such an action cannot drain the row again. - Replay revocation fails closed: if the discard write does not land, the native or MCP action is refused with a tool error instead of running unrecorded. - Inline admission and supersession of an older pending inline row commit in one transaction, so recovery never sees the new turn without the superseded one retired. - The initial human turn of a new web Fast session is durably admitted, not only replies.

… pending A guarded revocation that matches no pending row means another executor already settled this turn, so this execution is a stale duplicate and must not act. Propagate that result and reject the tool call.

Plain-text and error closeouts post through the adapter directly rather than a tool call, so they never revoked replay. A restart in the gap between that post and settlement could re-run the turn and post the closeout twice. Every terminal closeout path now revokes first and is skipped when the revocation does not land.

…rable The deliberate-cancellation branch revoked replay but ignored the result and posted the interruption closeout anyway; if the revocation had not landed, the still-recoverable row could re-run the turn after that closeout. The closeout is now gated on a successful revocation.

…withdrawn from replay Skipping a terminal or error closeout because the revocation did not land left settlement marking the row delivered, so the queue could not resume it and the user got no answer. Settlement now releases the claim and wakes the queue in that case instead of marking delivery, so the resumed run delivers the outcome.

The drain marked every delivered row as delivered, including an inline row whose resumed run had just released its claim because its terminal revocation did not land. That settled a row meant to stay recoverable. For inline rows the drain now checks the row after delivery and stops the pass when it is still pending, leaving it for the next sweep.

… needs after rebase

A pre-acknowledgement work call is refused by the gate, so it must not discard the durable row first; otherwise a restart before any real side effect could no longer resume the turn.

… setup A turn interrupted before it reached inference never ran its own abort handling, so its durable claim stayed held and the row waited out the full claim lease before recovery. Handlers now bind the durable row to the turn lock, and abortActiveFastAgentTurns releases every bound claim after aborting; the release is a guarded no-op for rows the turn already revoked or settled. Found by the local smoke: a SIGTERM during setup now hands the turn to the recovery sweep within seconds.

Unknown Web API methods now return HTTP 200 with ok:false and unknown_method, as Slack does, instead of a 404 that the WebClient retries with backoff for minutes while holding Fast turn locks. The agents.sessions status and rename calls made by the Fast session activity adapter are accepted and ignored.

R_FAST_DURABLE_ADMISSION_DISABLED=true stops persisting human turns for automatic resumption; turns run inline exactly as before this change and the queue's claim logic stays inert. Default remains enabled.

A turn interrupted before it reached its own abort handling had its claim released by the shutdown path but nothing woke the queue, so recovery waited for the periodic sweep. Handlers now bind the wake alongside the row id, and the shutdown path wakes the queue after a successful release.