[Fix] Resume a Fast turn whose admission write failed instead of asking the user to resend - #2185
Open
mrubens wants to merge 2 commits into
Open
[Fix] Resume a Fast turn whose admission write failed instead of asking the user to resend#2185mrubens wants to merge 2 commits into
mrubens wants to merge 2 commits into
Conversation
…esume Every admitted turn resumes on the next process, so #2156 removed the restart notice. A turn whose durable admission write failed has no row to resume from, and it ended in silence with the Session left showing responding. That turn now posts the recorded restart closeout again. Queue-delivered follow-ups stay quiet because the queue re-runs them, and platform events keep their silent handling; setup platform events are re-run by their own scheduler and a closeout would mark them complete.
Contributor
| undefined, | ||
| interruptionReason, | ||
| ); | ||
| } else if (restartedWithoutRecovery && !isInstructionClosed()) { |
Contributor
There was a problem hiding this comment.
inferenceRetryReply only means the retry notice has a message ID; it does not mean this adapter can replace it. Linear, for example, returns a handle from postReply but has no replaceReply. In that case replaceInferenceRetryReply records a hidden replacement and returns false, but the preceding branch ignores that result and this direct-closeout path is skipped. A non-durable Linear turn interrupted after a visible retry notice therefore leaves the stale retry message up and never tells the user to resend. Only take the replacement branch when replacement is supported, or fall through here when it returns false.
…the user to resend When a shutdown interrupts a turn that has no durable row, the handler's requestLateDurableAdmission hook persists the row now, releases its claim, and wakes the queue, so the next process resumes the turn. The restart closeout is posted only when that late admission fails too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What the user sees today
Almost every Fast turn survives a restart: it is admitted to a durable row before it runs, and the next process resumes it (#2016, #2145, #2156, #2179). Because of that, #2156 removed the "Roomote restarted, please send it again" closeout.
The one turn that could not resume was a turn whose durable admission write failed (a database hiccup at admission time). Admission is best effort by design: the turn still runs. If a deploy then landed on it, nothing re-ran it, and since the restart notice was gone it ended in silence. The thread went quiet, the Session showed "responding" until the lease expired, and the user had no way to know the request was dropped.
Change
We should never ask the user to retry something we can retry for them, so the turn is now admitted late and resumed.
FastAgentTurnAdaptergainsrequestLateDurableAdmission. When a shutdown interrupts a turn that has no durable row, is not a queue-delivered row, and is not a platform event, the service calls it. The hook (handOffFastAgentInterruptedTurnin the SDK) persists the row now, releases its claim, and wakes the queue. The turn's prompt and actions are already in the transcript under its turn id, so the next process resumes it exactly as it would a turn admitted up front, inheriting any visible retry notice.What I checked before scoping this
The review flagged setup-session turns as a bigger non-durable population, since web skips durable admission when a turn carries adapter extensions or a setup snapshot. They do not need this and must not get a closeout:
skipIfTurnCompletedplusreconcileSetupPlatformEvents). A closeout would be recorded as the terminal response and would stop that re-run.Every other web turn passes
durableSessionId; the comment saying kickoffs omit it was stale and is corrected.Queue-delivered human follow-ups (steered messages the queue runs as whole turns) also reach this path with no
durableAdmission, and since #2179 the worker aborts them on shutdown. The queue re-runs them itself (BullMQ retry, then the recovery sweep), so they stay quiet.What the user sees after
A deploy that lands on the rare unrecordable turn looks the same as one that lands on any other turn: nothing posted, the answer arrives from the next process. Only a double failure produces a message, and that message is honest.
Testing
fast-agent-service.test.ts: a turn with no row is admitted late and posts nothing; when late admission fails it posts the recorded restart closeout with the shutdown reason; a queue-delivered follow-up stays silent. Full suite (177) passes.fast-agent-human-follow-up.test.ts: the hand-off persists, releases the claim, and wakes the queue; reports the hand-off even when only the wakeup fails (the sweep covers it); reports no hand-off when the message already settled.pnpm lint:fast,pnpm check-types:fast,pnpm knippass.Not exercised live; the resume path the late admission feeds into is the one #2016 and #2145 smoke-tested.