fix(runtime-host): break reconnect failure cascades - #3917
Conversation
70f366d to
a3a4b6b
Compare
Astro-Han
left a comment
There was a problem hiding this comment.
I reviewed this head and found no blocking issues.
Fixes three reconnect cascade roots: subscription open flush→activate ordering, quiesce async wait, and monotonic replacement window. Hosted windows_recovery: SUCCESS; test for this head was in progress when author pushed new head b82d6da (task #355 now tracking latest).
No P0-P3 on a3a4b6b.
简体中文
该头无阻断,新提交另审。Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.
Astro-Han
left a comment
There was a problem hiding this comment.
I reviewed this head and found no blocking issues.
Subscription open ordering, quiesce wait, and monotonic replacement window all verified; hosted windows_recovery: SUCCESS, test: QUEUED.
No P0-P3 on b82d6da.
简体中文
该头无阻断。Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.
Generated-by: Codex
Generated-by: Codex
Co-authored-by: me2seeks <me2seeks@gmail.com> Co-authored-by: hqhq1025 <1506751656@qq.com> Generated-by: Codex
Generated-by: Codex
b82d6da to
367260a
Compare
English
Summary
This change removes three independent causes that turned an ordinary Runtime Host replacement into a prolonged failure cascade:
subscription.openresponse had left the connection-wide writer;quiesce()during the short interval with no current candidate;The repair does not increase queue capacity, add another retry loop, or weaken the liveness watchdog. It changes ownership and ordering at the three earliest faulty boundaries.
Root causes and fixes
Flush subscription ownership before activating live streams
All subscription responses and live frames share one writer bounded at 2 MiB. The previous order enqueued an open response, activated that subscription immediately, and only then waited for the response to flush. During restoration, sixteen legal large subscriptions could therefore place sixteen open responses and sixteen first live snapshots in the same queue. The writer treated that as overload and closed the socket; the client observed
read_eof, reconnected, restored the same subscriptions, and could repeat the same abort.The server now flushes each open response before activating its stream. Subscriber-local queues already retain events before activation, so this removes the overlap without losing data or adding buffering.
Quiesce the candidate that reconnects
quiesce()previously required a current candidate and threw during a reconnect gap. It is now asynchronous: it reuses the lifecycle's existingwaitForCurrent()authority, then freezes replacement synchronously around the recovered candidate. Desktop retirement simply awaits that lease.Give one IPC invocation one replacement lifetime
The router previously bounded reconciliation only. Initial handler absence could hold a non-replayable command forever, and every failed reconnectable-read replacement could begin another unbounded wait.
All handler replacement paths now share one lazy, monotonic deadline per renderer invocation. The budget starts only when a replacement is first required and never resets as candidates flap. Reconciled controls retain their existing unavailable fallback; reads and commands that never reach a handler fail explicitly.
This consolidates the bounded-handler work from #3795 and the shared monotonic read deadline from #3847 into one replacement authority. Their authors are credited in the commit trailers.
Closes #3458. Folds in the now-merged #3795 and supersedes #3847.
Relationship to #3280
#3280 fixes the downstream transcript identity defect: durable transcript ranges are scoped by Session and Host epoch rather than a Desktop replica generation. This PR addresses the upstream disconnect/reconnect causes and the independent quit-time race that #3280 intentionally did not cover.
I did not change the
host.statuswatchdog. After removing the proven server-side abort and the unbounded retained invocations, the captured timeout is explained as a liveness signal inside the cascade; I did not find evidence that a healthy, normally draining local Host independently exceeds that deadline.Validation
verifies that 200 concurrent invocations all settle after the same bounded
replacement window instead of remaining rooted while the target stays active.
invocations per round (4,000 total). All 4,000 settled, zero argument markers
remained reachable after collection, and the post-GC heap peak changed by about
148 KiB between the first and final five rounds. This directly validates the
retained-waiter path that made prolonged reconnect churn accumulate heap.
git diff --checkpass.The socket-backed remainder of the Runtime Host connection-session suite is left to hosted CI; the new regression itself uses an in-memory transport.
Generative tool disclosure
Codex made substantive contributions: it traced the failure chain, implemented the three repairs, and wrote the regression tests. Every contributed commit includes a
Generated-by: Codextrailer.简体中文
概要
这次修改移除了三个相互独立的根因,它们会把一次普通的 Runtime Host 替换放大成长时间故障:
subscription.open响应真正写出前启动直播流,合法的大恢复集合因此可能击穿连接级 2 MiB 写队列并主动断开连接;quiesce()会直接失败,退出或更新流程因此与重连窗口竞争;修复没有提高队列容量、没有增加新的重试循环,也没有削弱存活探针。它在三个最早出错的边界上改正了顺序和权威。
修复
quiesce()改为异步复用现有waitForCurrent():连接恢复后立刻冻结该候选者,Desktop 退休流程只需等待这份 lease。本 PR 关闭 #3458,吸收已经合并的 #3795,并取代 #3847。
#3280 已修复下游 transcript 身份问题;本 PR 修复它没有覆盖的上游断连/重连根因以及退出竞态。
验证包括:旧顺序下可稳定触发 transport abort、修复后 16 个打开响应和 16 个首帧全部写出;16/16 reconnecting-client 测试、49/49 Desktop manager/router 测试以及八个 workspace 构建全部通过。提交的并发回归确认 200 个调用都会在同一个窗口内结束。另一个本地纯内存 forced-GC 探针执行了 20 轮、每轮 200 个大参数调用,共 4,000 次;回收后没有参数 marker 仍可达,首尾五轮的 GC 后 heap 峰值只相差约 148 KiB。这个结果直接验证了持续重连时会累积 heap 的 retained-waiter 路径。新回归使用纯内存 transport;其余依赖 socket 的 connection-session 测试交给 hosted CI。
Codex 对本 PR 做了实质贡献,包括故障链追踪、三处修复和回归测试;所有相关 commit 都带有
Generated-by: Codextrailer。Posted by an automated review agent operated by @M4n5ter. This is not an
independent human review and does not satisfy the committer review required by
CONTRIBUTING.md. A human is accountable for this pull request — please push back
if anything here is wrong.
简体中文
本 PR 由 @M4n5ter 运行的自动化审查程序发出。它不构成 CONTRIBUTING.md
所要求的独立人类审查,也不能替代人类审查。有人类对本 PR 负责,如有错误请直接指出。