[Fix] bullmq drains and aborts resumed Fast turns before shutting down

[mrubens]

What the user sees today

You send a Fast message and a deploy lands while it is running. The API side is handled: it gets a 20s drain, and a turn that is not done hands itself off silently and picks up again on the queue worker within seconds (#2006, #2016). But the process that resumes it, the bullmq service, is being redeployed too, and it has no drain. When the platform kills it, the turn dies mid-run with its conversation lock and durable claim still held.

From the user's side the thread just goes quiet. The Session shows "responding" with no reply, no retry notice, and nothing to act on. Sending another message does not help, because the conversation lock belongs to a dead process. After about 10 minutes the lock expires and the turn resumes on its own, or after 15 minutes if the run had been going long enough to renew its claim. Any visible "Retrying in 30s" notice stays up for the whole wait.

This affects every deploy that lands during a Fast turn, which is exactly the population the drain work was built to protect. Without this fix, a good share of the turns the API path "saved" were dying a second time and looking hung.

Why

Durable admission (#2016, #2145, #2156) makes an interrupted Fast turn resume on the next process. The API hands the turn back on SIGTERM: it closes admissions, drains, aborts the stragglers, and each aborted turn releases its durable claim and wakes the queue.

The queue's consumer is the bullmq service, and it runs the resumed turn in-process. Its shutdown closed about twenty workers one after another with a waiting close() and never called the drain or abort, with the Fast worker nineteenth in the sequence. On a deploy both services restart:

1. API drains, aborts the straggler, releases the claim, queues the wake.
2. Old bullmq, mid-shutdown but still fetching, picks the job up and starts the resumed turn.
3. The platform's grace window ends and the process is SIGKILLed with the turn in flight.
4. The Redis conversation lock lingers for up to 600s and the durable claim, if the run reached a lease tick, for up to 15 minutes. The recovery sweep keeps hitting the busy lock and moving the job to a 1-second delayed retry until the dead owner's lease expires.

The guarantee in practice covered one restart, not a deploy.

Change

• Shared sequence. drainAndAbortFastAgentTurns and resolveFastAgentShutdownDrainMs in @roomote/cloud-agents hold the drain-then-abort ordering both services must follow. The API's graceful-shutdown.ts delegates to it with the same options and log line as before.
• bullmq drains and aborts. New apps/bullmq/src/graceful-shutdown.ts: close admissions, close the Fast parent-event worker as the drain starts so no new wakeups are fetched, let in-flight turns finish inside the window, abort the stragglers so each hands its row back, then close everything else. Jobs that arrive during the drain hit the refused lock, throw the busy error, and land in the delayed state for the new process. A worker that ignores its abort cannot hold shutdown open: the post-abort wait for its close is bounded (5s), since SIGKILL is coming anyway. Double signal forces exit, as in the API.
• Queue-resumed rows are bound to the turn lock. The drain now sets durableRowId and durableResume on the lock for each inline row before delivery and clears them after, so a shutdown that aborts a resumed turn during its setup (before the turn's own abort handling) can still release the claim and wake the queue. (From review.)
• Window. R_BULLMQ_SHUTDOWN_DRAIN_MS, falling back to R_API_SHUTDOWN_DRAIN_MS, then 20s. Documented in environment-variables.mdx and .env.production.example.
• FastAgentProcessShutdownError now says "process shutdown" rather than "API shutdown" and exposes signal.

What the user sees after

The worker gets the same window to finish, and anything still running is aborted so it hands itself back immediately. The next process resumes it within seconds, the same as the API path. Worst case is a slightly slower answer with no visible interruption, instead of a silent 10 to 15 minute hang.

Follow-up for operators

The bullmq service needs a SIGTERM-to-SIGKILL grace window longer than its drain window, the same adjustment the API service received. That lives in the deployment configuration, not in this repo.

Testing

• fast-agent-turn-shutdown.test.ts: resolver defaults, kill switch, key fallback; the sequence closes admissions, starts the parallel close, waits, then aborts, and stays quiet when everything settles.
• apps/bullmq/src/graceful-shutdown.test.ts: Fast worker closed as the drain starts and the rest closed only after the abort; bounded wait on a worker that never closes; exit still happens when the remaining closes fail; double-signal force exit.
• fast-agent-parent-event-queue.test.ts: a resumed inline row is bound to the lock during delivery, its wake callback queues the right job, and the binding is cleared afterwards.
• API graceful-shutdown.test.ts unchanged in substance (7 tests) and now runs against the real shutdown error instead of a module mock.
• Existing turn-lock and turn-drain suites, bullmq parent-event queue suite, pnpm lint:fast, pnpm check-types:fast, pnpm knip all pass.

Not exercised: a live deploy mid-resumed-turn. The mechanics on the turn side (abort → hand-back → wake) are the ones the API path already smoke-tested in #2016 and #2145; this change only makes bullmq trigger them.

[roomote-community]

No new code issues found. See task

• apps/bullmq/src/graceful-shutdown.ts:101 — Queue-resumed Fast turns do not bind their durable row to the turn lock, so shutdown can leave a renewed claim until its lease expires.

_{Reviewed d45866c}