Skip to content

Absorb user-DO resets in the Worker - #133

Open
ndisidore wants to merge 1 commit into
mainfrom
chore/handle-do-resets-unified
Open

Absorb user-DO resets in the Worker#133
ndisidore wants to merge 1 commit into
mainfrom
chore/handle-do-resets-unified

Conversation

@ndisidore

@ndisidore ndisidore commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Routine Durable Object resets — storage timeouts, overload aborts; ~3k over a recent 4-day window in production logs — were wedging sessions: a stub is permanently broken once its incarnation resets, so a cached stub turned one reset into every subsequent call failing until reload, surfacing as scary terminal errors.

terminal error screenshot terminal error screenshot

What changed

Fresh stub per call. The session re-resolves its user-DO stub on every access, so a post-reset call simply restarts the object

Deliberately no retries. With fresh stubs, the only thing a reset can still break is the call in flight at that moment; workerd's structured reset flags reach the client

Deliberately deferred / out of scope

  • Subscription self-heal: the DO holds subscriber registrations in memory only, so a reset still silently kills them while the browser's WebSocket stays healthy
  • Workspace (Overseer) DO resets keep their pre-existing session-abort handling

Testing

  • Integration (abortAllDurableObjects(), the non-graceful teardown): the same session recovers transparently after a reset via fresh stubs, and the local-aborts-reject-flagless shape is pinned so the flag path can graduate from synthetic unit tests to real-reset integration tests if a future pool upgrade attaches the production flags.
  • Unit: the reset-flag predicate against production-shaped synthetic errors. 284 backend unit + integration green; no frontend or shared-API changes in this diff.
  • Manual fault injection (chrome-devtools against a local dev stack, temporary debugAbort() injector, composed with the Classify RPC errors client-side and quiet recoverable failures #110 client): run on the pre-split branch — 72 aborts across idle, reload-race, and SPA-navigation-storm scenarios with every page functional and zero application console output; a chat send racing a workspace-DO abort showed the inline hint (Classify RPC errors client-side and quiet recoverable failures #110's UI), preserved the draft, and did not double-send. The subscription-specific results move to the follow-up PR.

ask-bonk[bot]

This comment was marked as outdated.

@ask-bonk

This comment was marked as outdated.

@ndisidore
ndisidore force-pushed the chore/handle-do-resets-unified branch from 4b76f09 to 58866bc Compare August 10, 2026 21:58
ask-bonk[bot]

This comment was marked as outdated.

@ask-bonk

This comment was marked as outdated.

@Maximo-Guk

Maximo-Guk commented Aug 10, 2026

Copy link
Copy Markdown
Member

GPT comments:

1. Medium: do-retry.ts:19 retries errors with both overloaded and reset flags. Cloudflare explicitly says overloaded errors must not be retried; this can amplify an overload. Check overloaded before retrying.
2. Medium: server.ts:125-152 allows queries to overtake earlier commands because each uses a fresh stub and queries bypass #commandChain. This regresses RPC e-order/pipelining. Queries should await a snapshot of prior commands.
3. Low: server.ts:811,861 retries authentication resets without emitting the new retry/surfaced telemetry, undercounting resets.

Routine user-DO resets (storage timeouts, overload aborts — the errors seen
in production logs) were wedging sessions: a stub is bound to one incarnation
and permanently broken once it resets (DO error-handling docs), so a cached
stub turned one reset into every subsequent call failing until reload.

- Fresh stub per call: #user re-resolves on every access, so a post-reset
  call simply restarts the object and the session self-heals structurally.
- Every user-DO RPC goes through the #userCall / #userCommand choke points —
  a naked this.#user.x() is a review defect. Typed #query/#command Proxies
  route the ~35 plain delegation sites through them while deriving the
  telemetry operation name from the DO method. The split is about ordering:
  commands are effectful and serialize on a per-session chain (per-call
  stubs forfeit cross-stub e-order, and overlapping optimistic writes like
  rapid pin toggles must arrive in issue order); queries stay concurrent.
- Deliberately NO retries: fresh stubs already fix the wedged-session mode,
  what remains is only the call in flight at the reset moment, workerd's
  structured reset flags reach the client (which classifies and quiets
  them, #110), and the DO platform is moving toward transparent recovery.
  do-reset.ts keeps the flag predicate (durableObjectReset/retryable, never
  bare overloaded) purely to classify surfaced resets for telemetry
  (user_do.reset.surfaced) — the volume check on this design's thesis.
- authenticate() rejects corrupt base64 tokens as coded auth failures
  instead of leaking the decoder's SyntaxError.

Integration tests pin that local vitest-pool-workers aborts reject FLAGLESS
(so the flag predicate is unit-tested with production-shaped synthetic
errors, and a pool upgrade that adds real flags fails loudly) and cover
same-session recovery across abortAllDurableObjects().

Deliberately deferred: self-healing of connected-accounts subscriptions
across resets (the DO's in-memory registrations die with the incarnation,
so a subscribed browser silently stops receiving updates until it
re-subscribes or the socket reconnects). A TODO(deferred) at the subscribe
chokepoint marks the gap; the implementation is split out to the follow-up
branch feat/do-reset-subscription-self-heal.
@ndisidore
ndisidore force-pushed the chore/handle-do-resets-unified branch from 58866bc to 1b25391 Compare August 10, 2026 22:40
@ask-bonk

ask-bonk Bot commented Aug 10, 2026

Copy link
Copy Markdown

Re: the GPT comments (Maximo-Guk)

I traced all three against the actual diff. All three describe the superseded #104 changeset (which shipped a do-retry.ts with a real one-shot retry layer), not this PR — the retry layer was built and then removed on review, as the PR body notes. There is no retry anywhere in this diff; do-reset.ts is a pure telemetry classifier. Details:

1. "do-retry.ts:19 retries overloaded errors" — not applicable.
There is no do-retry.ts in this branch. The file is do-reset.ts, and isDoResetError (do-reset.ts:15-18) is used only for telemetry (user_do.reset.surfaced) — #userCall observes the flag and rethrows unchanged (server.ts:157-160). Nothing retries. Note also the predicate already excludes overloaded alone (do-reset.ts:11-14): an overloaded-but-alive object is not classified as a reset. So even the classification the comment worries about is already the conservative one.

2. "queries overtake earlier commands, regressing e-order" — deliberate and documented.
This is the central, intentional trade of the fresh-stub design, called out explicitly at server.ts:108-110 and 143-144: per-call stubs forfeit per-stub e-order; #userCommand re-establishes command→command issue order via #commandChain, and command→query ordering is deliberately not restored ("a call site that depends on a prior user-DO call must await it"). Making every query await a snapshot of prior commands would serialize slow vendor-bound queries (e.g. getCloudflareUsage, listConnectedAccounts) behind unrelated commands — the opposite of the concurrency this split is designed to preserve. The optimistic-UI ordering hazard the comment implies (reversed writes) applies only to command→command, which is ordered.

3. "auth retries resets without emitting telemetry" — no auth retries exist.
authenticate (server.ts:796), authenticateFromCfAccess (server.ts:814), login (server.ts:841), and createAccount (server.ts:870) are all single naked this.users.get(id).<method>() calls with no retry wrapper. A reset there surfaces to the client (classified/quieted by #110) exactly once. There is nothing to under-count.

On the earlier review nits

  • The delayMs: undefined item is already resolved on the current revision: #onUserDoReset (server.ts:187-194) emits only event, operation, durableObjectId, error — no delayMs field is passed at all.
  • Comment-block line lengths (server.ts:116-125) are all ≤95 cols now, in-band with the surrounding block.

Net: no in-scope code change needed. The three GPT items are all stale against #104's retry design; the fresh-stub-no-retry design here is intentionally free of the idempotency/overload/telemetry hazards they raise.

@ask-bonk

ask-bonk Bot commented Aug 10, 2026

Copy link
Copy Markdown

Posted my analysis to PR #133.

Summary

I reviewed the three actionable GPT comments (from Maximo-Guk) against the actual diff. All three describe the superseded #104 changeset, not this PR — the retry layer was built and removed on review, so none of the hazards apply:

  1. do-retry.ts:19 retries overloaded errors — No such file exists here. do-reset.ts:15-18 is a telemetry-only classifier; #userCall observes the flag and rethrows unchanged (server.ts:157-160). No retry. And it already excludes overloaded-alone.

  2. Queries overtake commands, regressing e-order — Deliberate and documented (server.ts:108-110, 143-144). #userCommand orders command→command via #commandChain; command→query ordering is intentionally not restored to avoid serializing slow vendor-bound queries behind unrelated commands. The reversed-write hazard only applies to command→command, which is ordered.

  3. Auth retries resets without telemetry — The auth methods (server.ts:796, 814, 841, 870) are single naked DO calls with no retry wrapper. Nothing to under-count.

I also verified the two stale ask-bonk nits: delayMs: undefined is already gone (#onUserDoReset at server.ts:187-194 never passes it), and the comment-block lines are all ≤95 cols.

No in-scope code changes were warranted — making spurious edits would have been the wrong call given the comments are all stale against #104's retry design.

github run

@ndisidore ndisidore added the kernel Changes to the Workshop kernel label Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kernel Changes to the Workshop kernel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants