-
Notifications
You must be signed in to change notification settings - Fork 27
Description
What happened
A container was killed by the sleepAfter alarm while containerFetch() was actively awaiting a response from the container. The operation (a matplotlib chart generation via Sandbox SDK's exec(), which uses containerFetch internally) had been running for approximately 6 seconds when the container received SIGTERM.
Client-side error:
Error: ReadableStream received over RPC disconnected prematurely.
Container-side error:
SessionDestroyedError: Session 'main' was destroyed during command execution
Note: the exact error messages come from Sandbox SDK, but sharing it here to be illustrative.
Reproduction
- Create a DO subclass extending
Container - Set
sleepAfterto a short duration (e.g.,"10s") - Call
containerFetch()with a request that takes longer thansleepAfterto complete (e.g., a command execution that runs for 15+ seconds) - The alarm fires while
containerFetch()is still awaiting the response, andonActivityExpired()kills the container
This also affects back-to-back operations: if several quick operations consume most of thesleepAfterwindow, a subsequent longer operation can be killed even though it just started, becauserenewActivityTimeout()resets the deadline tonow + sleepAfter, notnow + sleepAfterfrom when the operation finishes.
Expected behaviour
The sleepAfter timer should not fire while containerFetch() is actively awaiting a response. The container is clearly in use when there's an in-flight request.
Current workaround
Before each long-running operation, extend the sleepAfter property and refresh the timeout:
this.sleepAfter = estimatedDurationSeconds + 30;
this.renewActivityTimeout();This works but requires subclasses to predict operation duration and manage the timeout around every containerFetch() call.
Potentially related issues
- Container terminated mid-processing after ~8-10 minutes of active work #138: Container killed mid-processing (likely same root cause)
- WebSocket connections don't renew activity timeout - container sleeps despite active connections #147: WebSocket connections don't trigger activity renewal (related: no refresh at all for non-
containerFetchpaths) - "SleepAfter" is not effective #127:
sleepAfterappears ineffective
Environment
Observed in production using v0.1.1 of @cloudflare/containers