fix(chat): batch replayed chunks on stream resume - #2050
Open
ben-reitz wants to merge 4 commits into
Open
Conversation
🦋 Changeset detectedLatest commit: 10a80c4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
agents
@cloudflare/ai-chat
@cloudflare/codemode
create-think
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
ben-reitz
force-pushed
the
fix/replay-burst-update-depth
branch
2 times, most recently
from
August 7, 2026 07:24
2d68d7c to
4192570
Compare
ben-reitz
added
the
ready-for-review-and-merge
Resuming a stream replays every stored chunk of the active turn as its own `cf_agent_use_chat_response` frame. Both transport-owned resume paths forwarded each frame into the UI-message stream individually, so chat state was rebuilt once per chunk. Two costs followed: each rebuild deep-clones the whole assistant message, making replay quadratic in turn length; and each rebuild schedules a synchronous React render, so a long turn's burst outpaced React's commit loop and tripped its nested-update guard. The resulting "Maximum update depth exceeded" escaped `useAgentChat` as a false terminal `status: "error"` for a turn the server had completed fine. `ReplayChunkBatch` buffers `replay: true` chunks and merges consecutive deltas belonging to the same part, and `applyChatResponseFrame` delivers the compacted batch once at the replay boundary — `replayComplete`, `done`, or the first live chunk. Applying a replayed prefix is now proportional to the number of message parts rather than the number of chunks. Invariants preserved: - Only adjacent deltas of the same part merge, so cross-part ordering is exact. - Deltas carrying `providerMetadata` are kept whole. - Both burst terminators flush: live streams end replay with `replayComplete`, finalized or orphaned streams end with `done` and never send it. - An errored resumed turn still delivers its replayed content before the terminal error (#1575). `controller.error()` would discard the queued batch, so `failChatStream` enqueues an `error` chunk behind the flush instead, which `processUIMessageStream` rethrows into the same terminal-error path. Tested at both layers: `replay-batch.test.ts` covers merge and boundary semantics, and `resume-replay-burst.test.tsx` runs the real hook in the react project, where React's update-depth throw is actually observable. All four react cases fail without the transport change. Refs #1913 Co-authored-by: Horcrux <7693239+magicismight@users.noreply.github.com>
The first version of the replay batch held chunks until a burst terminator arrived. That was too long, and broke two behaviours in packages that consume the transport. A burst with no terminator never reached the UI. `replayComplete` and `done` are the documented terminators, but a resumed stream is not obliged to send either promptly, and tests in `@cloudflare/ai-chat` correctly asserted that replayed chunks appear on arrival. Worse, two replay passes of the same turn were concatenated into one batch and produced a duplicated assistant message. The hook repairs a second replay (#1733) in `resetMatchingHydratedAssistantForReplay`: on each replayed `start` frame it wipes the parts of the matching trailing assistant, so the pass can rebuild it. That repair reads messages the previous pass already applied. With the batch held open, nothing had been applied when the second `start` arrived, the reset matched nothing, and both passes then applied back to back. The batch is now a coalescing window over a single event-loop turn: it opens on the first buffered chunk and closes at the end of that turn, or earlier at a burst boundary. This keeps the property that fixes #1913 — a burst delivered in one turn is still merged to roughly one chunk per part — while making any frame sequence separated in time behave exactly as it did before. A burst spread over several turns cannot trip React's guard in the first place, so nothing is lost. The window is injectable, so its behaviour is asserted directly rather than by racing a timer. `ReplayChunkBatch` now owns the stream controller, so every write to the stream goes through it and nothing can overtake replayed content. Tests: three cases for the window in `replay-batch.test.ts`, and a react case asserting a burst with no terminator still appears. Both fail without this change. Suites: agents chat 530, react 132, workers 1801; ai-chat 737 + react 82; think 1000 across five projects.
… passes Two review findings on the replay batch. Buffered content was lost when a stream was closed rather than terminated by a frame. `onClose` and the detach path in both resume streams closed the controller without flushing, and the pending window timer then enqueued into a closed stream. A socket that died in the same task as the burst dropped the whole burst: a react case measures 0 characters where `main` shows 186 (`main` only truncates because the burst also throws). Both paths now flush first, and `flush()` never throws, so a caller on its way to closing a stream always gets to close it. Correctness also leaned on the flush timer winning a race it is not guaranteed to win. `setTimeout(0)` is a macrotask, not an end-of-turn hook, so two replay passes of one turn could in principle land in one batch and be concatenated — the duplication the hook repairs (#1733) when the passes are applied separately. A replayed `start` for a message already buffered now supersedes the buffered pass: every replay rebuilds from its first chunk, so the newer pass is a superset and the older one has reached nobody. Continuation replays are excluded, since they append rather than rebuild. The window is now only a latency device, and the comments say so instead of claiming an end-of-turn guarantee. Tests: three cases covering the close path, the supersede rule, and that a continuation replay is never superseded. Each fails without its fix. Suites: agents chat 532, react 133, workers 1801; ai-chat 737 + react 82; think 1000.
`setTimeout(0)` is a fresh macrotask. It runs no earlier than the end of the current task, but other queued tasks can run before it, so "end-of-turn" claimed an ordering the scheduler does not give. The comments now describe the window as best-effort coalescing, and say plainly that correctness rests on the three order-independent rules instead: a burst boundary flushes, close and detach flush, and a newer replay supersedes a buffered pass. Also trims the commentary, which had grown heavier than the logic: comment lines 102 -> 89. Removed restatements of what the code says (`closeNow`, `errorNow`, the `bufferedStartMessageId` getter) and kept the parts that carry a reason a reader cannot recover from the code — why a flush must not throw, why an errored turn flushes before erroring (#1575), and why a superseded pass is safe to drop (#1733). Renames the test helper `endTurn` to `fireWindow` for the same reason. No behaviour change. chat 18, react 6, `pnpm run check`.
ben-reitz
force-pushed
the
fix/replay-burst-update-depth
branch
from
August 10, 2026 15:00
4192570 to
10a80c4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1913. Thanks to @magicismight for the diagnosis and the fork branch that inspired this fix.
What users see
Your WebSocket drops in the middle of an answer and reconnects a second later. Instead of the answer carrying on, the console fills with
Maximum update depth exceededand the chat shows an error - even though the server finished the answer fine and has it stored. Only a reload clears it.Why it happens
When a client reconnects mid-turn, the server replays the answer so far: every chunk it has stored, back to back, as fast as the socket will carry them.
The transport handed each chunk straight to the AI SDK. Every chunk rewrites the chat state, and every rewrite is a React render. React allows 50 renders in an unbroken row before it assumes something is looping and throws - and a replayed answer is easily 400 chunks, arriving with nothing in between to break the chain.
The throw lands deep inside the SDK's stream loop, which can't tell a rendering failure from a dead connection, so it marks the turn as failed and stops reading.
flowchart LR S["server replays<br/>400 stored chunks"] --> T{transport} T -->|"before: one at a time"| B1["400 chat-state writes"] --> B2["renders 1 to 50"] --> B3["render 51 throws<br/>status: error"] T -->|"after: collected and merged"| A1["about 5 writes"] --> A2["5 renders"] --> A3["answer restored<br/>status: ready"]The fix
The transport now catches replayed chunks in a short window instead of forwarding them one by one, and glues together the ones belonging to the same piece of text. A few hundred chunks become a handful of updates - roughly one per part of the message. The replayed text also appears at once, rather than visibly re-typing itself.
Holding chunks is only safe if they can never be stranded. Three rules take care of that:
It's one new module,
chat/replay-batch.ts, wired into the two resume paths inws-chat-transport.ts. Nothing changes in the server, the wire format, orchat/react.tsx.Tests
18 unit tests in
chat/__tests__/replay-batch.test.tscover the merge rules, the window, the boundaries and an errored turn. Six tests inreact-tests/resume-replay-burst.test.tsxdrive the real hook with real wire frames - all six fail onmain, and that is the only layer where the React throw is visible at all.Suites: agents chat 532 · react 133 · workers 1801 · ai-chat 737 + react 82 · think 1000 ·
pnpm run check.Next steps
This massively reduces the chance of the bug appearing in the wild, but a few things remain for a follow-up:
For 1 & 2, setting a default value for
throttleonuseAgentChatavoids the crash outright - which is what 25 of our 27 examples already do. Am going to raise a followup PR after this to set a default value for it inuseChatAgentrather than leaving it unthrottled.