Skip to content

fix(runtime-host): break reconnect failure cascades - #3917

Merged
M4n5ter merged 4 commits into
mainfrom
fix/runtime-host-flapping-root
Aug 26, 2026
Merged

fix(runtime-host): break reconnect failure cascades#3917
M4n5ter merged 4 commits into
mainfrom
fix/runtime-host-flapping-root

Conversation

@M4n5ter

@M4n5ter M4n5ter commented Aug 26, 2026

Copy link
Copy Markdown
Member
English

Summary

This change removes three independent causes that turned an ordinary Runtime Host replacement into a prolonged failure cascade:

  • a reconnect restoring many subscriptions could abort a healthy server connection because each live stream was activated before its subscription.open response had left the connection-wide writer;
  • Desktop retirement could fail solely because it called quiesce() during the short interval with no current candidate;
  • renderer IPC invocations could remain alive across unlimited candidate replacements because handler waits had no invocation-wide lifetime.

The repair does not increase queue capacity, add another retry loop, or weaken the liveness watchdog. It changes ownership and ordering at the three earliest faulty boundaries.

Root causes and fixes

Flush subscription ownership before activating live streams

All subscription responses and live frames share one writer bounded at 2 MiB. The previous order enqueued an open response, activated that subscription immediately, and only then waited for the response to flush. During restoration, sixteen legal large subscriptions could therefore place sixteen open responses and sixteen first live snapshots in the same queue. The writer treated that as overload and closed the socket; the client observed read_eof, reconnected, restored the same subscriptions, and could repeat the same abort.

The server now flushes each open response before activating its stream. Subscriber-local queues already retain events before activation, so this removes the overlap without losing data or adding buffering.

Quiesce the candidate that reconnects

quiesce() previously required a current candidate and threw during a reconnect gap. It is now asynchronous: it reuses the lifecycle's existing waitForCurrent() authority, then freezes replacement synchronously around the recovered candidate. Desktop retirement simply awaits that lease.

Give one IPC invocation one replacement lifetime

The router previously bounded reconciliation only. Initial handler absence could hold a non-replayable command forever, and every failed reconnectable-read replacement could begin another unbounded wait.

All handler replacement paths now share one lazy, monotonic deadline per renderer invocation. The budget starts only when a replacement is first required and never resets as candidates flap. Reconciled controls retain their existing unavailable fallback; reads and commands that never reach a handler fail explicitly.

This consolidates the bounded-handler work from #3795 and the shared monotonic read deadline from #3847 into one replacement authority. Their authors are credited in the commit trailers.

Closes #3458. Folds in the now-merged #3795 and supersedes #3847.

Relationship to #3280

#3280 fixes the downstream transcript identity defect: durable transcript ranges are scoped by Session and Host epoch rather than a Desktop replica generation. This PR addresses the upstream disconnect/reconnect causes and the independent quit-time race that #3280 intentionally did not cover.

I did not change the host.status watchdog. After removing the proven server-side abort and the unbounded retained invocations, the captured timeout is explained as a liveness signal inside the cascade; I did not find evidence that a healthy, normally draining local Host independently exceeds that deadline.

Validation

  • A production-session regression constructs sixteen wire-valid large subscription restores entirely in memory. Restoring the old activate-before-flush order makes the transport abort; the fixed order stays open and flushes all 16 responses plus all 16 first projections. The hypothetical simultaneous population is larger than 2 MiB, while the response-only population is below the bound.
  • 16/16 reconnecting-client lifecycle tests pass.
  • 49/49 Desktop manager and reconnecting IPC tests pass. A committed regression
    verifies that 200 concurrent invocations all settle after the same bounded
    replacement window instead of remaining rooted while the target stays active.
  • The new reconnect-gap retirement test reaches the replacement candidate exactly once.
  • A synthetic 15 ms monotonic budget stops a twenty-candidate read-flapping sequence after four attempts, proving that candidate replacement cannot reset the invocation lifetime.
  • A local, in-memory forced-GC probe ran 20 flapping rounds with 200 large-argument
    invocations per round (4,000 total). All 4,000 settled, zero argument markers
    remained reachable after collection, and the post-GC heap peak changed by about
    148 KiB between the first and final five rounds. This directly validates the
    retained-waiter path that made prolonged reconnect churn accumulate heap.
  • Core, Storage, MCP, Runtime, Runtime Host, Computer Use, UI, and Desktop main builds pass.
  • Biome and git diff --check pass.

The socket-backed remainder of the Runtime Host connection-session suite is left to hosted CI; the new regression itself uses an in-memory transport.

Generative tool disclosure

Codex made substantive contributions: it traced the failure chain, implemented the three repairs, and wrote the regression tests. Every contributed commit includes a Generated-by: Codex trailer.

简体中文

概要

这次修改移除了三个相互独立的根因,它们会把一次普通的 Runtime Host 替换放大成长时间故障:

  • 重连恢复多个订阅时,服务端会在 subscription.open 响应真正写出前启动直播流,合法的大恢复集合因此可能击穿连接级 2 MiB 写队列并主动断开连接;
  • Desktop 在短暂没有当前候选者时调用 quiesce() 会直接失败,退出或更新流程因此与重连窗口竞争;
  • renderer IPC 没有统一的调用生命周期,候选者不断替换时,同一次调用可以无限等待并长期保留参数、事件和 Promise。

修复没有提高队列容量、没有增加新的重试循环,也没有削弱存活探针。它在三个最早出错的边界上改正了顺序和权威。

修复

  1. 服务端先冲刷订阅打开响应,再激活对应直播流。订阅自己的队列本来就会保存激活前事件,因此不丢数据,也不需要第二份缓冲。
  2. quiesce() 改为异步复用现有 waitForCurrent():连接恢复后立刻冻结该候选者,Desktop 退休流程只需等待这份 lease。
  3. 所有 IPC handler 替换路径共享同一个按调用建立的单调时钟 deadline。它只在第一次需要替换时开始,并且不会随候选者反复失败而重置。该设计把 fix(desktop): bound Runtime Host handler waits #3795fix(desktop): bound reconnectable read retries #3847 的关键修复合并为一份 replacement authority,并在 commit trailer 中保留作者署名。

本 PR 关闭 #3458,吸收已经合并的 #3795,并取代 #3847

#3280 已修复下游 transcript 身份问题;本 PR 修复它没有覆盖的上游断连/重连根因以及退出竞态。

验证包括:旧顺序下可稳定触发 transport abort、修复后 16 个打开响应和 16 个首帧全部写出;16/16 reconnecting-client 测试、49/49 Desktop manager/router 测试以及八个 workspace 构建全部通过。提交的并发回归确认 200 个调用都会在同一个窗口内结束。另一个本地纯内存 forced-GC 探针执行了 20 轮、每轮 200 个大参数调用,共 4,000 次;回收后没有参数 marker 仍可达,首尾五轮的 GC 后 heap 峰值只相差约 148 KiB。这个结果直接验证了持续重连时会累积 heap 的 retained-waiter 路径。新回归使用纯内存 transport;其余依赖 socket 的 connection-session 测试交给 hosted CI。

Codex 对本 PR 做了实质贡献,包括故障链追踪、三处修复和回归测试;所有相关 commit 都带有 Generated-by: Codex trailer。


Posted by an automated review agent operated by @M4n5ter. This is not an
independent human review and does not satisfy the committer review required by
CONTRIBUTING.md. A human is accountable for this pull request — please push back
if anything here is wrong.

简体中文

本 PR 由 @M4n5ter 运行的自动化审查程序发出。它不构成 CONTRIBUTING.md
所要求的独立人类审查,也不能替代人类审查。有人类对本 PR 负责,如有错误请直接指出。

@M4n5ter
M4n5ter force-pushed the fix/runtime-host-flapping-root branch from 70f366d to a3a4b6b Compare August 26, 2026 09:59

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this head and found no blocking issues.

Fixes three reconnect cascade roots: subscription open flush→activate ordering, quiesce async wait, and monotonic replacement window. Hosted windows_recovery: SUCCESS; test for this head was in progress when author pushed new head b82d6da (task #355 now tracking latest).

No P0-P3 on a3a4b6b.

简体中文该头无阻断,新提交另审。

Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed this head and found no blocking issues.

Subscription open ordering, quiesce wait, and monotonic replacement window all verified; hosted windows_recovery: SUCCESS, test: QUEUED.

No P0-P3 on b82d6da.

简体中文该头无阻断。

Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — b82d6da, no P0-P3 per review.


Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

Co-authored-by: me2seeks <me2seeks@gmail.com>

Co-authored-by: hqhq1025 <1506751656@qq.com>

Generated-by: Codex
@M4n5ter
M4n5ter force-pushed the fix/runtime-host-flapping-root branch from b82d6da to 367260a Compare August 26, 2026 12:24
@M4n5ter
M4n5ter merged commit 5754873 into main Aug 26, 2026
2 checks passed
@M4n5ter
M4n5ter deleted the fix/runtime-host-flapping-root branch August 26, 2026 12:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Desktop main process OOM after prolonged Runtime Host connection flapping (repeated MCP capability alignment failures)

2 participants