[Fix] Deploys interrupt in-flight Fast turns instead of letting them finish - #2006
Merged
Conversation
Shutdown previously aborted every active Fast turn as its first act, so each deploy interrupted whatever users had in flight and posted a resend closeout. The API now closes admissions and stops accepting connections first, then gives active turns a bounded window (R_API_SHUTDOWN_DRAIN_MS, default 20s, 0 restores abort-immediately) to finish on their own; only the stragglers still active at the deadline are aborted through the existing shutdown path. The drain wait resolves the moment the last turn settles, so idle shutdowns stay instant, and a hard deadline keeps a wedged turn from hanging shutdown.
Contributor
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On SIGTERM the API aborts every active Fast turn as its first act, before anything else happens. Every deploy therefore kills whatever users have in flight and posts a restart closeout asking them to resend. Attribution work in #2004 identified this as the largest producer of interrupted-turn messages: the frequency is roughly deploys times concurrently active turns, and most Fast turns would have finished within seconds if given the chance.
Change
Shutdown now drains before it aborts:
beginFastAgentTurnDraincloses admissions immediately (new lock acquisitions are refused) without touching active turns.server.close()stops accepting connections in parallel.waitForActiveFastAgentTurnsToSettlegives in-flight turns a bounded window to finish on their own. It resolves the moment the last turn settles, so idle shutdowns stay instant, and the hard deadline means a wedged turn cannot hang shutdown (preserving the [Fix] API shutdown can hang before Fast answers start #1958 guarantee).The window is
R_API_SHUTDOWN_DRAIN_MS(default 20s, sized to leave room for the straggler abort, closeout delivery, and Sentry flush inside a typical 30s SIGTERM-to-SIGKILL grace). Setting it to 0 restores the previous abort-immediately behavior as a kill switch. Documented in.env.production.example.Validation
@roomote/cloud-agents(fresh module instance per test): admissions refused during drain without aborting active turns, settle-wait resolves on last release, stragglers reported at the deadline, and stragglers abort with the reason the drain began with. Existing turn-lock suite unchanged and passing (11 tests total across both files).@roomote/apigraceful-shutdown suite rewritten for the drain ordering: drain precedes abort, quiet path logs nothing, close-failure exit code and double-signal force-exit behavior preserved, env parsing covered (7 tests).pnpm lint:fast,pnpm check-types:fast,pnpm knipall pass.Context
PR 2 of the interruption work started in #2004. Next: renewing the Fast responding lease mid-turn so the scheduled reconciler stops stamping turns that are still running.