Skip to content

sleepAfter alarm fires during long-running containerFetch() operations #162

@ghostwriternr

Description

@ghostwriternr

What happened

A container was killed by the sleepAfter alarm while containerFetch() was actively awaiting a response from the container. The operation (a matplotlib chart generation via Sandbox SDK's exec(), which uses containerFetch internally) had been running for approximately 6 seconds when the container received SIGTERM.

Client-side error:

Error: ReadableStream received over RPC disconnected prematurely.

Container-side error:

SessionDestroyedError: Session 'main' was destroyed during command execution

Note: the exact error messages come from Sandbox SDK, but sharing it here to be illustrative.

Reproduction

  1. Create a DO subclass extending Container
  2. Set sleepAfter to a short duration (e.g., "10s")
  3. Call containerFetch() with a request that takes longer than sleepAfter to complete (e.g., a command execution that runs for 15+ seconds)
  4. The alarm fires while containerFetch() is still awaiting the response, and onActivityExpired() kills the container
    This also affects back-to-back operations: if several quick operations consume most of the sleepAfter window, a subsequent longer operation can be killed even though it just started, because renewActivityTimeout() resets the deadline to now + sleepAfter, not now + sleepAfter from when the operation finishes.

Expected behaviour

The sleepAfter timer should not fire while containerFetch() is actively awaiting a response. The container is clearly in use when there's an in-flight request.

Current workaround

Before each long-running operation, extend the sleepAfter property and refresh the timeout:

this.sleepAfter = estimatedDurationSeconds + 30;
this.renewActivityTimeout();

This works but requires subclasses to predict operation duration and manage the timeout around every containerFetch() call.

Potentially related issues

Environment

Observed in production using v0.1.1 of @cloudflare/containers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions