Skip to content

[Fix] Deploys interrupt in-flight Fast turns instead of letting them finish - #2006

Merged
daniel-lxs merged 1 commit into
developfrom
fix/api-shutdown-drain
Sep 1, 2026
Merged

[Fix] Deploys interrupt in-flight Fast turns instead of letting them finish#2006
daniel-lxs merged 1 commit into
developfrom
fix/api-shutdown-drain

Conversation

@daniel-lxs

Copy link
Copy Markdown
Member

Problem

On SIGTERM the API aborts every active Fast turn as its first act, before anything else happens. Every deploy therefore kills whatever users have in flight and posts a restart closeout asking them to resend. Attribution work in #2004 identified this as the largest producer of interrupted-turn messages: the frequency is roughly deploys times concurrently active turns, and most Fast turns would have finished within seconds if given the chance.

Change

Shutdown now drains before it aborts:

  1. beginFastAgentTurnDrain closes admissions immediately (new lock acquisitions are refused) without touching active turns.
  2. server.close() stops accepting connections in parallel.
  3. waitForActiveFastAgentTurnsToSettle gives in-flight turns a bounded window to finish on their own. It resolves the moment the last turn settles, so idle shutdowns stay instant, and the hard deadline means a wedged turn cannot hang shutdown (preserving the [Fix] API shutdown can hang before Fast answers start #1958 guarantee).
  4. Only the stragglers still active at the deadline are aborted through the existing shutdown path, with the same closeout semantics as before, plus a log line stating how many turns were aborted after the drain.

The window is R_API_SHUTDOWN_DRAIN_MS (default 20s, sized to leave room for the straggler abort, closeout delivery, and Sentry flush inside a typical 30s SIGTERM-to-SIGKILL grace). Setting it to 0 restores the previous abort-immediately behavior as a kill switch. Documented in .env.production.example.

Validation

  • New drain suite in @roomote/cloud-agents (fresh module instance per test): admissions refused during drain without aborting active turns, settle-wait resolves on last release, stragglers reported at the deadline, and stragglers abort with the reason the drain began with. Existing turn-lock suite unchanged and passing (11 tests total across both files).
  • @roomote/api graceful-shutdown suite rewritten for the drain ordering: drain precedes abort, quiet path logs nothing, close-failure exit code and double-signal force-exit behavior preserved, env parsing covered (7 tests).
  • pnpm lint:fast, pnpm check-types:fast, pnpm knip all pass.

Context

PR 2 of the interruption work started in #2004. Next: renewing the Fast responding lease mid-turn so the scheduled reconciler stops stamping turns that are still running.

Shutdown previously aborted every active Fast turn as its first act, so
each deploy interrupted whatever users had in flight and posted a resend
closeout. The API now closes admissions and stops accepting connections
first, then gives active turns a bounded window (R_API_SHUTDOWN_DRAIN_MS,
default 20s, 0 restores abort-immediately) to finish on their own; only
the stragglers still active at the deadline are aborted through the
existing shutdown path. The drain wait resolves the moment the last turn
settles, so idle shutdowns stay instant, and a hard deadline keeps a
wedged turn from hanging shutdown.
@roomote-community

roomote-community Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

No code issues found. See task

Reviewed 0d6ce08

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant