Skip to content

[Fix] Resume a Fast turn whose admission write failed instead of asking the user to resend - #2185

Open
mrubens wants to merge 2 commits into
developfrom
fix/fast-nondurable-interruption-closeout
Open

[Fix] Resume a Fast turn whose admission write failed instead of asking the user to resend#2185
mrubens wants to merge 2 commits into
developfrom
fix/fast-nondurable-interruption-closeout

Conversation

@mrubens

@mrubens mrubens commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

What the user sees today

Almost every Fast turn survives a restart: it is admitted to a durable row before it runs, and the next process resumes it (#2016, #2145, #2156, #2179). Because of that, #2156 removed the "Roomote restarted, please send it again" closeout.

The one turn that could not resume was a turn whose durable admission write failed (a database hiccup at admission time). Admission is best effort by design: the turn still runs. If a deploy then landed on it, nothing re-ran it, and since the restart notice was gone it ended in silence. The thread went quiet, the Session showed "responding" until the lease expired, and the user had no way to know the request was dropped.

Change

We should never ask the user to retry something we can retry for them, so the turn is now admitted late and resumed.

  • Late durable admission. FastAgentTurnAdapter gains requestLateDurableAdmission. When a shutdown interrupts a turn that has no durable row, is not a queue-delivered row, and is not a platform event, the service calls it. The hook (handOffFastAgentInterruptedTurn in the SDK) persists the row now, releases its claim, and wakes the queue. The turn's prompt and actions are already in the transcript under its turn id, so the next process resumes it exactly as it would a turn admitted up front, inheriting any visible retry notice.
  • Every inline surface passes the hook when its up-front admission failed: Slack messages and reactions, Discord, web, and the shared surface-reply path (Teams, Telegram, Linear, source control).
  • Fallback. Only if the late admission also fails (or the same message already settled) does the turn post the recorded closeout "Roomote restarted while working on this request. Please send it again." That is the case where nothing will re-run it.

What I checked before scoping this

The review flagged setup-session turns as a bigger non-durable population, since web skips durable admission when a turn carries adapter extensions or a setup snapshot. They do not need this and must not get a closeout:

  • They are platform events, and platform events already end silently on every other error path.
  • The setup scheduler re-runs any setup event that never completed (skipIfTurnCompleted plus reconcileSetupPlatformEvents). A closeout would be recorded as the terminal response and would stop that re-run.

Every other web turn passes durableSessionId; the comment saying kickoffs omit it was stale and is corrected.

Queue-delivered human follow-ups (steered messages the queue runs as whole turns) also reach this path with no durableAdmission, and since #2179 the worker aborts them on shutdown. The queue re-runs them itself (BullMQ retry, then the recovery sweep), so they stay quiet.

What the user sees after

A deploy that lands on the rare unrecordable turn looks the same as one that lands on any other turn: nothing posted, the answer arrives from the next process. Only a double failure produces a message, and that message is honest.

Testing

  • fast-agent-service.test.ts: a turn with no row is admitted late and posts nothing; when late admission fails it posts the recorded restart closeout with the shutdown reason; a queue-delivered follow-up stays silent. Full suite (177) passes.
  • fast-agent-human-follow-up.test.ts: the hand-off persists, releases the claim, and wakes the queue; reports the hand-off even when only the wakeup fails (the sweep covers it); reports no hand-off when the message already settled.
  • Surface-reply, source-control, Slack reaction, and web fast-sessions suites pass. pnpm lint:fast, pnpm check-types:fast, pnpm knip pass.

Not exercised live; the resume path the late admission feeds into is the one #2016 and #2145 smoke-tested.

…esume

Every admitted turn resumes on the next process, so #2156 removed the
restart notice. A turn whose durable admission write failed has no row to
resume from, and it ended in silence with the Session left showing
responding. That turn now posts the recorded restart closeout again.
Queue-delivered follow-ups stay quiet because the queue re-runs them, and
platform events keep their silent handling; setup platform events are
re-run by their own scheduler and a closeout would mark them complete.
@roomote-community

roomote-community Bot commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

1 issue outstanding. See task

  • Retry notices on adapters without replacement suppress the restart closeout (packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts:4655).

Reviewed 6a75a27

undefined,
interruptionReason,
);
} else if (restartedWithoutRecovery && !isInstructionClosed()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

inferenceRetryReply only means the retry notice has a message ID; it does not mean this adapter can replace it. Linear, for example, returns a handle from postReply but has no replaceReply. In that case replaceInferenceRetryReply records a hidden replacement and returns false, but the preceding branch ignores that result and this direct-closeout path is skipped. A non-durable Linear turn interrupted after a visible retry notice therefore leaves the stale retry message up and never tells the user to resend. Only take the replacement branch when replacement is supported, or fall through here when it returns false.

…the user to resend

When a shutdown interrupts a turn that has no durable row, the handler's
requestLateDurableAdmission hook persists the row now, releases its claim,
and wakes the queue, so the next process resumes the turn. The restart
closeout is posted only when that late admission fails too.
@mrubens mrubens changed the title [Fix] Tell the user when a restart cuts off a Fast turn that cannot resume [Fix] Resume a Fast turn whose admission write failed instead of asking the user to resend Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant