Conversation
…ming A transcript page-read rejection used to re-arm unconditionally: advance()'s finally queued another catch-up whenever the announced watermark still led the durable one, so a replica bound to a dead subscription spun an unbounded microtask rejection storm — observed pinning a Maka main process near 75% CPU and starving IPC. Runtime Host read failures are permanent for the subscription a replica is bound to, so the first one is now latched: later advance() calls and transcript reads reject with that same error — ahead of the generic closed state — and only a settled catch-up re-arms, once, to cover a watermark that moved while it ran. Non-subscription failures still propagate unlatched so a transient page read can retry on the next announced frame, and renderer delivery failures keep their existing path and never latch. advance() also loses its explicit target: the subscription already records the newest announced watermark before handing out the frame, so the replica reads it from the handle rather than mirroring it in a second field. Recovery stays with the subscription owner, which replaces the replica or the subscription by its existing classification. Generated-by: Devin
…ndshake A frame handler rejection inside the pump's for-await only reached the catch block after the loop's implicit return(), which awaits the subscription's close acknowledgement — under a rejection storm that handshake never lands, so failure handling never ran and the failed attempt's replica stayed armed. Frame errors are now dispatched inside the loop: the attempt is failed and replacement or terminal teardown is kicked off before the iterator is left, so teardown no longer queues behind the close handshake. Generated-by: Devin
…scription Eviction recovery used to replace the whole subscription — a new handshake, a new replica, and retiring-subscription frame buffering — even though the subscription was still alive and only the durable tail was missing. Reseeding reads the current tail at the live watermark on the same handle, installs it through the same path as prepare(), and lets advance() cover whatever was committed during the fetch. The owner swaps attempt.replica only while the attempt is unchanged, so a concurrent recovery is never displaced; consumers reset through the existing generation mechanism. Deletes refresh(), the retiring phase, pendingFrames buffering, and the MAX_PENDING_* constants. The two owner-level tests that exercised refresh directly are replaced by reseed tests covering same-subscription rebuild, catch-up across the fetch window, and supersession by recovery. Generated-by: Devin
runtimeHostSessionFixture now defaults transcript and events to empty, and hosts the shared AsyncFrameQueue, continuitySnapshot, and transcriptPage that several test files each defined locally. Removes the empty-transcript literals and three duplicate helper definitions. Generated-by: Devin
…swap The reseed swapped attempt.replica and closed the evicted replica while the observer's state still referenced it, installing the new one a microtask later. A transcript_advanced landing in the gap called advance() on a closed replica, whose rejection classified as terminal — the failure class this branch exists to eliminate. A recovery interposing in the same gap closed the freshly swapped replica and left state.replica and attempt.replica permanently diverged. The observer's install now runs as an installReseededReplica dep inside the owner's staleness check, so state.replica moves before the evicted replica closes — the same atomicity activate() has. The failed attempt detaches before the first teardown await so a reseed cannot swap onto a corpse, and discard()/trimDurable() check residency before liveness since recovery windows legitimately leave closed replicas in state. A reseed read failure also had nowhere to go: session.transcript.page not_found — the class recovery exists for — propagated raw and stranded the session on a dead handle. It now routes through #failAttempt exactly like a pump-frame failure, and onChange no longer caches snapshots from replicas that were never installed. Adds the coverage the audit surfaced: recovery on a failed reseed, concurrent-reseed serialization with byte-balance assertions on superseded builds, evicted-replica advance and mid-flight watermark tests, and an observer-level evict-to-reopen test behind a transcriptGlobalCacheMaxBytes dep. Generated-by: Devin
state.replica can legitimately hold a non-resident or closed replica — eviction and recovery windows both produce that — but the invariant was implicit, so each consumer encoded its own assumption (discard() and trimDurable() asserted liveness first, which killed unrelated sessions under cache pressure). Writing the legal state set and the single swap authority down where the pointer lives. Generated-by: Devin
The install callback ran consumer resets and the budget pass before moving state.replica, diverging from activate()'s pointer-first order and undercounting the new replica's bytes in that pass. The pointer now moves right after the only throwing steps: a throw still leaves the state on the evicted replica for the owner to close, while resets and accounting see the installed replica. The post-commit catch-up moves out of the callback and into the owner, where its rejection takes the same #failAttempt path as a pump-frame failure. Swallowing it inside the install left a latched-dead replica reporting resident on a quiet stream. Generated-by: Devin
- feed the surviving projector the reseeded tail: rows that went durable while a replica was evicted never reached its durable-message map, so steering suppression and message admissions stayed stale until the next subscription recovery rebuilt the projector - report resident=false for a latched replica: a dead read model now takes the same reseed/recovery path as an evicted one, and ack/read gates stop throwing the latched error into unrelated IPC - let the pump's authoritative error land before reporting a masked 'connection_closed' from a racing reseed fetch — a recoverable close could otherwise be inverted into a terminal teardown - wait out in-flight recovery before a losing reseed returns, so callers pick up the installed replica instead of erroring on the closed one - protect the just-installed replica from its own install-time budget pass - drop dead watermark bookkeeping on evicted replicas — nothing reads a husk's durableThrough now that reseed always builds a new replica Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
jackwener
left a comment
There was a problem hiding this comment.
Independent agent review. Reviewed at 4c8164d15dacf425dc20dd697ba0847baa3c45f5. I am an AI agent (executing seat @kabi-opus) publishing through a shared GitHub account; this is an automated review and does not substitute for independent human review. Several other lines are reviewing this in parallel at the same head, including one under a different operator; we do not share conclusions.
One P2, inline. The diagnosis and the shape of the fix are right — retry authority belonged in one place and the owner is the layer that can classify. My finding is about where the classification boundary actually landed versus where the Summary says it landed.
The deletions are genuine obsolescence, not lost capability
I checked this before anything else, because "three authorities become one" usually means something stopped being covered. pendingFrames, MAX_PENDING_FRAMES (32), MAX_PENDING_FRAME_BYTES (256 KB) and the owner's refresh() all existed to buffer frames through the retiring-subscription phase that eviction recovery used to require. With reseed() rebuilding the tail on the live subscription there is no retiring phase to buffer through, and the remaining path that does replace a subscription — recovery — goes through #prepare(), a full rebuild that re-reads rather than replaying buffered frames. Nothing needs the buffer, so deleting it removes real complexity rather than coverage.
resident reflecting the latch is the right half of the same idea
The Summary is candid that the latch initially left a dead read model that every resident gate still saw as usable. Making resident return this.#resident && this.#failure === undefined is what turns a latched replica into something the owner routes around instead of something callers keep touching — and it is what makes my P2 a cost question rather than a correctness one, since an over-eager latch degrades into a reseed rather than a stuck session.
Evidence
Full workspace dependency rebuild plus apps/desktop build:test from a cleaned dist at this head. Desktop main suite: 2714 pass, 0 fail, 0 skipped. Focused: desktop-transcript-replica 6/6, runtime-host-session-subscription-owner 9/9, runtime-host-session-observer 57/57. Error-set and precedent citations above are from the protocol spec and client source at this head, not from the description.
Not covered by me
The incident itself — I did not reproduce the microtask storm, so "this removes the 1h high-CPU loop" rests on the diagnosis reading correct to me, not on measurement. Electron end-to-end is outside what I can run, matching your own unchecked box.
简体中文
在 4c8164d15dacf425dc20dd697ba0847baa3c45f5 上评审。一条 P2,在行内。 诊断与修法形状都是对的 —— 重试权威本就该收归一处,而 owner 是唯一有信息做分类的那一层。我的发现是:分类边界实际落在哪里,与 Summary 说它落在哪里,不一致。
删除是真正的过时,不是能力丢失 —— 这一条我最先核,因为"三个权威收敛成一个"常常意味着有东西不再被覆盖。pendingFrames、MAX_PENDING_FRAMES(32)、MAX_PENDING_FRAME_BYTES(256 KB)与 owner 的 refresh(),存在的意义都是在"退休订阅"阶段缓冲帧,而那个阶段是旧的驱逐恢复所需要的。改成 reseed() 在存活订阅上重建尾部之后,已经没有需要被缓冲穿过的退休阶段;而仍然会替换订阅的那条路径(恢复)走的是 #prepare() —— 完整重建、重新读取,而不是回放缓冲帧。没有东西再需要这个缓冲,所以删掉它减的是复杂度,不是覆盖面。
resident 跟随 latch 是同一想法的另一半:Summary 很坦率地写了,latch 一开始留下了一个"每个 resident 闸门都还当它可用"的死读模型。让 resident 返回 this.#resident && this.#failure === undefined,才使被 latch 的 replica 变成 owner 会绕开的东西,而不是调用方继续去碰的东西 —— 这也正是我那条 P2 属于代价问题而非正确性问题的原因:过早 latch 会降级成一次 reseed,而不是卡死会话。
证据:本 head 上完整重建工作区依赖并 build:test;desktop main 全量 2714 通过 / 0 失败 / 0 跳过;定向:desktop-transcript-replica 6/6、runtime-host-session-subscription-owner 9/9、runtime-host-session-observer 57/57。上面引用的错误集与先例都来自本 head 的协议规格与客户端源码,不是来自描述。
未覆盖:事故本身 —— 我没有复现那个 microtask 风暴,所以"这消除了 1 小时高 CPU 循环"依据的是我读下来诊断成立,而不是测量。Electron 端到端不在我能跑的范围,与你自己未勾选的那一项一致。
| // instead of retrying a dead subscription — the owner decides whether | ||
| // recovery means replacing this replica or the whole subscription. | ||
| if ( | ||
| error instanceof RuntimeHostSubscriptionError || |
There was a problem hiding this comment.
[P2] The latch keys on the error class, but the Summary's rule — and your own test fixture — are about the code. On this operation the class covers several codes that do not mean the subscription is gone.
session.transcript.page declares its error set in packages/runtime-host/src/protocol/session-transcript.ts:77:
host_not_ready · host_draining · operation_unavailable · invalid_request
not_found · operation_conflict · persistence_failed · internal_failure
A RuntimeHostOperationError carrying any of those latches here. But the Summary says "Non-subscription read errors stay retryable", and the new test builds the terminal case explicitly:
new RuntimeHostOperationError('session.transcript.page', 'not_found', 'subscription transcript context was lost')not_found is the right thing to latch, and the fixture's own message names the reason: the subscription's transcript context is gone. host_not_ready does not mean that — it means the Host is coming up. Nor does operation_conflict, persistence_failed, or internal_failure. invalid_request is the sharpest: a malformed query from this process would latch a perfectly healthy replica.
What that costs is not a wedge, and that matters for how you weigh it. Latching makes resident false, so the owner reseeds or recovers — bounded, and the recovery path is correct. The cost is that a Host restart window, which previously resolved by reading again, now tears down and rebuilds a subscription. The old bug was an unbounded retry; the shape being traded in is an over-eager teardown on ordinary transient conditions.
The codebase already discriminates at this granularity. connection.ts:622 retries subscription open specifically on transcript_preparing with a bounded deadline, and lets every other code through — the same judgement applied one layer up. Latching on not_found, session_archived-like terminals and host_draining, while leaving host_not_ready / operation_conflict / persistence_failed / internal_failure retryable, would match both the Summary's stated rule and that precedent.
简体中文
latch 判的是错误类,而 Summary 的规则 —— 以及你自己的测试夹具 —— 说的是错误码。在这个操作上,该类涵盖了若干并不意味着订阅已死的码。
session.transcript.page 在 packages/runtime-host/src/protocol/session-transcript.ts:77 声明的错误集是:host_not_ready、host_draining、operation_unavailable、invalid_request、not_found、operation_conflict、persistence_failed、internal_failure。带上述任意一个码的 RuntimeHostOperationError 都会在这里 latch。
但 Summary 写的是「Non-subscription read errors stay retryable」,而新测试构造的恰恰是终态那一个:new RuntimeHostOperationError('session.transcript.page', 'not_found', 'subscription transcript context was lost')。not_found 该 latch,夹具自己的消息就点明了理由:订阅的 transcript 上下文没了。 而 host_not_ready 不是这个意思 —— 它是"Host 正在起来";operation_conflict、persistence_failed、internal_failure 同样不是。invalid_request 最刺眼:本进程发出的一个畸形查询,会把一个完全健康的 replica latch 掉。
代价不是卡死,这一点对如何权衡很重要:latch 使 resident 变 false,owner 会 reseed 或恢复 —— 有界,且恢复路径是对的。代价在于:一个 Host 重启窗口,原本再读一次就过去了,现在会变成拆掉并重建一个订阅。旧缺陷是无界重试,换进来的形状是在寻常瞬时条件下过早拆除。
这个粒度的判别在本仓已有先例:connection.ts:622 对订阅打开专门针对 transcript_preparing 做有界重试,其余码一律透出 —— 同样的判断,只是在上一层。把 not_found、host_draining 这类终态 latch 住,而让 host_not_ready / operation_conflict / persistence_failed / internal_failure 保持可重试,就能同时符合 Summary 自述的规则和这条先例。
There was a problem hiding this comment.
Follow-up on this [P2] — I went and read the server handler, and the invariant the comment states is contradicted by it, which I think makes this concrete rather than theoretical.
#readTranscriptPage (packages/runtime-host/src/server/session-continuity-coordinator.ts:1073-1129) emits four codes for this operation:
| code | server condition | is the subscription gone? |
|---|---|---|
not_found |
subscriber not owned / access revoked / ownership changed mid-read | yes |
operation_unavailable |
!this.#transcriptReader || !subscriber.transcript |
no |
invalid_request |
TranscriptPageRequestError from the pager |
no |
persistence_failed |
any other throw out of readSessionTranscriptPage |
no |
persistence_failed is the catch-all around the reader: a transient storage/IO error on the Host surfaces as persistence_failed while the subscription is completely alive. The latch here captures it by class, so that transient read error permanently kills this replica — advance() rejects with the stored error forever and resident goes false — and the code comment above it ("A Runtime Host read failure is permanent for the subscription this replica is bound to") asserts something the server does not guarantee for 3 of the 4 codes it can return.
To be explicit about scope, because I don't think you should be asked to fix base behaviour in this PR: isRecoverableSubscriptionFailure in runtime-host-session-subscription-owner.ts is byte-identical to the merge-base (0169d0731), and it already treats everything except session.transcript.page/not_found as terminal. So the whole-subscription teardown on persistence_failed is not introduced here. What this PR adds is the class-keyed latch on the replica side, and what I'm asking for is only that the latch key on the same thing the rule is written in terms of — the code — rather than the class.
Narrowing it to error.code === 'not_found' (plus the RuntimeHostSubscriptionError reasons) would make the latch agree with both your Summary's wording and your own test fixture, which constructs new RuntimeHostOperationError('session.transcript.page', 'not_found', 'subscription transcript context was lost'). The repo already discriminates this way at packages/runtime-host/src/client/connection.ts:622, which retries only on transcript_preparing.
If you intend the broad capture — i.e. you'd rather over-tear-down than risk any retry against a possibly-dead subscription — then I'd ask for the comment to say that instead, since as written it claims an invariant the protocol does not provide.
简体中文
补充这条 [P2] 的证据:我去读了服务端 handler,代码里写的那条不变量被它证伪了。
#readTranscriptPage(session-continuity-coordinator.ts:1073-1129)对这个操作只会发四个码:not_found(订阅确实没了)、operation_unavailable、invalid_request、以及 persistence_failed——它是包在 readSessionTranscriptPage 外面的兜底 catch。也就是说,Host 侧一次瞬时的存储/IO 错误会变成 persistence_failed,而此时订阅是活的。
这里的闩锁按错误类别捕获,于是这次瞬时读错误会永久杀死这个 replica(advance() 之后永远用存下来的错误 reject,resident 变 false),而上面那句注释「A Runtime Host read failure is permanent for the subscription this replica is bound to」断言的东西,对该操作四个码里的三个都不成立。
范围我说清楚,免得让你在这个 PR 里背基线的债:owner 里的 isRecoverableSubscriptionFailure 与 merge-base(0169d0731)逐字节相同,基线就已经把除 not_found 外的一切判为终结。所以 persistence_failed 导致整条订阅终结不是这个 PR 引入的。这个 PR 新增的只是 replica 侧这个按类别的闩锁;我请求的也只是让闩锁按码判,和规则本身的表述口径一致。
收窄成 error.code === 'not_found'(外加 RuntimeHostSubscriptionError 的那几个 reason)就能同时对上你 Summary 的措辞和你自己的测试夹具(构造的正是 ('session.transcript.page', 'not_found', 'subscription transcript context was lost'))。仓库里已有按码区分的先例:client/connection.ts:622 只在 transcript_preparing 上重试。
如果你就是要宽口径捕获(宁可过早拆除也不冒重试死订阅的风险),那我请求改注释说清楚这一点——因为现在这句话宣称了一个协议并不提供的保证。
Automation disclosure: this reply was produced by an AI agent (Claude Opus 5) reviewing at head 4c8164d15dacf425dc20dd697ba0847baa3c45f5, published through the shared jackwener account. No human re-derived these findings line by line. The server-side code table above was read directly from source at this head.
jackwener
left a comment
There was a problem hiding this comment.
Review of exact head 4c8164d15dacf425dc20dd697ba0847baa3c45f5.
I am not approving this head. There is an open P2. This is an Astro-Han pull request and it is still a draft; I will not merge it.
What this change is doing
The failure mode is real. DesktopTranscriptReplica.advance() used to re-arm a catch-up read whenever the watermark was still ahead, including after a Host read had already failed. On a dead subscription that is a pure-microtask retry storm in the main process.
This head does three things that belong together:
- The replica latches the first
RuntimeHostSubscriptionError/RuntimeHostOperationErrorfrom a catch-up read, reportsresident === false, and does not re-arm after a rejection. - The subscription pump calls
#failAttemptinside the frame loop, so a handling error is on its way to teardown before the iterator'sreturn()handshake. - Eviction recovery reseeds the tail on the live subscription instead of opening a replacement subscription and buffering frames in a retiring phase.
The latch and the pump change sit at the right layer. Retry classification belongs to the owner; the replica is a read model that should die with its subscription.
I did not treat author self-checks as proof. Locally: replica tests 6/6 and owner tests 9/9 pass. Dropping the latch makes both latch tests fail (pageReads 2 vs 1). Required test is completed/success on this SHA.
P2 — this pull request — connection_closed from a reseed is terminal, and one setImmediate is not a join with the pump
apps/desktop/src/main/runtime-host-session-subscription-owner.ts:174-182
isRecoverableSubscriptionFailure accepts slow_consumer, sequence_gap, projection_revision_invalid, and session.transcript.page/not_found. It does not accept connection_closed. So #failAttempt on a RuntimeHostSubscriptionError('connection_closed') is terminalFailure.
A reseed page read can fail with that mask because the subscription is already dead, while the pump's real reason (slow_consumer, and so on) has not been pulled from the iterator yet. The code waits one setImmediate and then, if the attempt is still current, fails it with the mask.
One event-loop turn is not "the pump has reported". If the iterator has not yielded the closed frame yet, this head turns a recoverable close into a terminal teardown — the inversion the comment says it is trying to avoid.
The test named for this race (a reseed fetch masked by the subscription close does not out-race the pump report) stayed green when I skipped the yield (one run). It does not pin the yield.
Minimal fix: do not #failAttempt the mask from reseed. Let the pump own subscription death, or wait for the attempt to change, instead of one tick plus a terminal fail.
P3 — this pull request — installReseededReplica moves state.replica before steps that can still throw
apps/desktop/src/main/runtime-host-session-observer.ts:697-712
The comment says the pointer moves after the only throwing steps. After state.replica = replica the callback still runs noteDurableTranscriptMessages, consumer reset, and the budget pass. If one of those throws, the owner closes the new replica and rethrows, but attempt.replica is still the evicted one. Observer state and the owner then disagree. I did not force that throw in a test.
Verification bounds
- Ran: replica tests, owner tests, latch mutation (red), skipped-yield mutation (test still green).
- Did not run: observer suite, full desktop
test:dist, e2e, a real main-process CPU repro. - CLI test edits only implement the new
transcriptWatermarkgetter on fakes.
Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
简体中文
只绑 4c8164d15。不 approve,不合并(Astro-Han,仍是 draft)。
问题是真的:死订阅上 catch-up 失败还会再排队,会在主进程里空转。latch + 泵在离开 iterator 之前 dispatch,层放对了。
P2:reseed 读到 connection_closed 会走 terminal(这个 reason 不在可恢复集合里)。setImmediate 一次不是和泵汇合。跳过这段 yield 后,对应测试仍然绿。最小修法:reseed 不要用这个掩码去 #failAttempt,订阅死亡交给泵。
P3:installReseededReplica 在 noteDurable / reset / budget 之前就改了 state.replica,后面若抛,observer 和 owner 的指针会分叉。
| error instanceof RuntimeHostSubscriptionError && | ||
| error.reason === 'connection_closed' | ||
| ) { | ||
| await new Promise((resolve) => setImmediate(resolve)); |
There was a problem hiding this comment.
P2 on this head. connection_closed is not in isRecoverableSubscriptionFailure, so #failAttempt here is terminalFailure.
A reseed page read can fail with this mask before the pump has pulled the real closed reason from the iterator. One setImmediate is not a join with that pump. Skipping the yield left a reseed fetch masked by the subscription close does not out-race the pump report green (one run).
Do not #failAttempt the mask from reseed. Let the pump own subscription death, or wait until this attempt is no longer current.
zhiiw
left a comment
There was a problem hiding this comment.
Independent review. Conclusions bind to 4c8164d15dacf425dc20dd697ba0847baa3c45f5 (still Draft; CI test green on this head).
Verified locally (Windows 11, Node 24.18.1 — the version CI pins): all eight affected desktop main suites green, 137/137 — replica, subscription-owner, observer, range-store, bot-session-adapter, client, history-read, tail-restore.
The claimed root cause checks out in the code, and the fix closes it at the right layer. On the base, advance() re-armed in .finally whenever the target was ahead, with no error classification — on a dead subscription that is exactly the pure-microtask retry storm described. On this head, subscription/operation read failures latch (#failure), the re-arm lives only in the success branch and is additionally gated by #isLive(), so even an unclassified permanent read error can no longer storm: it retries at frame cadence and propagates through acceptFrame into #failAttempt. Retry authority really is singular now.
The claims I verified against source rather than trusting:
- Watermark authority:
ClientSessionSubscriptionupdates#latestTranscriptThroughSequencebefore#offer(frame)hands the frame out (session-subscription.ts), and both the client (correlation_changedon reads past the announced watermark) and the pager (readSessionTranscriptPagerejects beyond the subscription's known high water) keep the replica's reads inside the announced range. - Reseed read semantics:
direction: 'older'+throughSequence: watermark+ null cursor is the same tail-page shape as the bootstrap insession-transcript-pager.ts, and the post-installadvance()covers rows committed mid-fetch. A watermark-moved-mid-fetch case is pinned by the new replica tests. close()implies!resident:close()clears#resident, so the residency-first ordering intrimDurable/discardmakes the budget pass a no-op on a recovery-closed replica — the comment's claim is true. The field/getter split (bytes stay reclaimable while reads reject) is coherent.- Pump failure dispatch: the per-frame
try/catchcalls#failAttemptbefore leaving the iterator, so teardown no longer queues behind the subscription's close handshake — the storm's second half. - Reseed swap atomicity: the observer's install runs synchronously inside the owner's staleness check; on install throw the new replica is closed and state stays on the evicted one; a lost swap waits out the in-flight recovery instead of handing back a corpse; concurrent reseeds serialize to one swap. Each has a pinning test in the new owner suite.
- Zero residue: no
advance(arg)callers, norefresh(), nopendingFrames/MAX_PENDING_*outside tests. The deleted observer tests (refresh candidate fails during commit,active recovery supersedes cold refresh) pinned the deletedrefresh()path; their equivalents exist for reseed (supersede, install-throw, recovery-window races).
Considered and withdrawn: the reseed reads a single bounded page — correct for a bounded tail read model (hasOlder carries the rest); unclassified-but-permanent read errors retry per-frame rather than latching — bounded and observable, acceptable; watermark === null falls back to the bootstrap — matches subscription shape.
Not verified: full e2e (the author also did not run it; the change is main-process-only), and I did not re-run the CLI suites whose fakes gained the watermark field (test-only changes, CI covers them).
No findings. Approving as one independent seat; this is not a merge.
Automated review notice: This comment was posted by an automated review agent operated by zhiiw. It is not an independent human review and does not replace one.
jackwener
left a comment
There was a problem hiding this comment.
P2 — a transient transcript read failure can permanently disable the replica after a successful catch-up
The server can return session.transcript.page / persistence_failed while the subscription remains live; the protocol details are documented in this review thread. I exercised the resulting client behavior on this commit: one such error causes exactly one page read, makes advance() reject with that same error, and leaves resident === false.
The distinct failure window is the fire-and-forget catch-up at desktop-transcript-replica.ts:354-375. If a watermark advances while the preceding read succeeds, that code calls void this.advance().catch(() => undefined). A one-off storage error in that automatic read reaches #catchUp at :404-419, where every RuntimeHostOperationError is latched. The rejection is discarded, so the replica remains permanently unavailable instead of retrying once the storage error clears. A later transcript advance then observes the latched failure.
Please limit the permanent latch to errors that prove this subscription is no longer usable, rather than every operation error. Add a regression test for: successful catch-up, watermark advance, one persistence_failed error from the automatic catch-up, then a successful subsequent read. The existing tests cover not_found, but not this transient failure path.
Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
|
Follow-up to review 5246649685, still on exact head Stronger evidence that nothing pins the yieldMy original note said the named race test stayed green after skipping the kabi-opus ran the ablation as a distribution on macOS/Node, still on this SHA, and confirmed the mutant actually landed (
So it is not that one test missed the yield. Nothing in the desktop suite fails when that line is deleted. I did not run this distribution myself. Scope of the P2The P2 has two halves. They are not the same change.
What the ablation does and does not proveThe ablation proves the suite does not pin the yield. It does not prove that one tick is insufficient in production. Those are different claims. Adding a test that only encodes the current tick would close the first gap and leave the second untouched.
简体中文仍绑 P2 的两半要分开:可恢复集合与 merge-base 相同,不是本 PR 引入;本 PR 引入的是 reseed 拿 |
jackwener
left a comment
There was a problem hiding this comment.
Automated review of exact commit 4c8164d15dacf425dc20dd697ba0847baa3c45f5: 1×P2 current regression; not ready to merge.
[P2][current regression] Wait for the installed replica's post-commit catch-up before resolving reseed
RuntimeHostSessionSubscriptionOwner.reseedTranscriptReplica() installs the replacement replica, starts its final advance() detached, and immediately returns that replica (apps/desktop/src/main/runtime-host-session-subscription-owner.ts:196-208). The observer resumes from that promise and rejects if the detached read has already latched a subscription failure (apps/desktop/src/main/runtime-host-session-observer.ts:304-311).
I reproduced this through the production observer/owner/cache path:
- Cache pressure evicts an active Session's transcript replica.
- Opening the transcript starts a same-subscription reseed.
- A durable row lands between the reseed fetch and commit.
- The post-commit catch-up receives
session.transcript.page/not_foundbecause the old Host transcript context has gone away. - Owner recovery successfully opens and installs the replacement subscription.
- The initiating
openTranscript()still rejects withDesktop transcript replica is unavailable.
The recovery logs had already reached subscription recovered before the caller failed. Normal range controllers automatically reopen, but report the first failure before doing so, which AppShell turns into a visible error/toast. One-shot callers such as session-message-settlement.ts:76-94 await transcripts.open() directly and have no such recovery controller, so revision and Side Chat settlement can fail until the user retries. No durable data is lost.
The smallest fix is to make the final catch-up part of reseed settlement: await replica.advance(); for RuntimeHostSubscriptionError / RuntimeHostOperationError, route through #failAttempt, await replacement readiness, and return undefined; preserve generic read errors for the caller. I applied only that mutation and the same probe passed.
The checked-in routes a post-commit catch-up failure through attempt recovery test proves eventual owner recovery, but it does not exercise the initiating observer caller and currently accepts the soon-to-be-dead replica as a successful reseed result. Please extend or replace it with an observer-level assertion that the initiating open settles on the replacement generation.
Required conclusions:
- The current solution is not optimal because the owner resolves reseed before the last read it owns has settled.
- Production code to delete: none identified; this needs a small settlement correction, not another abstraction.
- Low-value test to replace: strengthen the post-commit catch-up test as described above. No other deletion identified.
- No deeper refactor is required; keep one subscription owner and make its reseed promise authoritative through catch-up or replacement readiness.
- This exact revision is not ready to merge because of the P2.
- Residual gap: I did not run a packaged Electron reconnect journey. Clean exact-head Desktop main build and 112 focused replica/owner/observer/range-store tests passed; the exact-head hosted
testcheck is also successful. No security, data-loss, licensing, or release-boundary effect was identified. User-visible transcript availability is affected and still requires independent human review.
Automated review notice: This comment was posted by an automated review agent operated by jackwener. It is not an independent human review and does not replace one.
…ficate The fourth adversarial round found two real defects in the previous fixes: - The setImmediate deferral only covered the case where the pump's report was already queued; a pump blocked inside acceptFrame's round-trip still lost to the masked connection_closed. The subscription already knows its own death reason — expose terminalError/closedReason and let #failAttempt prefer that certificate, which heals the reseed, acceptFrame, and post-commit paths deterministically and makes the deferral unnecessary. - The latch caught every RuntimeHostOperationError, so a transient blip on the swallowed post-settle re-arm became a sticky terminal on the next frame — a regression over main's silent retry. Narrow the latch to failures that prove the subscription or its transcript context is dead. Also assertTranscriptReadable now throws the recorded terminal error when one exists, so every transcript reader sees the real failure. Tests: the masked-close classification test was vacuous (the pump's chain is strictly shorter, so it could never exercise the race) — it is now driven by the certificate, plus a terminal-removal variant. Add coverage for loser-branch terminal propagation, transient operation failures staying unlatched, acknowledgeTranscriptTail on a latched replica, and a latched replica reseeding through recovery end to end. Generated-by: Devin
…ial review - Record the Host close reason before the queue offer so a full client queue cannot discard session_removed/access_revoked as slow_consumer. - Prefer closedReason unconditionally over terminalError: the Host's own death statement is recorded first, so it always outranks the synthesized connection_closed mask a dying transport stores later. - Normalize subscription.open/not_found at the terminal boundary to the removed-session path so a recovery losing to a deletion still emits sessions:changed deleted instead of a generic observation error. - Exempt TranscriptCacheCapacityError from attempt failure: the rolled- back charge is not a subscription failure, and tearing a healthy subscription down under memory pressure frees nothing. #failAttempt now reports whether it consumed the failure so the pump keeps consuming absorbed errors instead of exiting unconditionally. - Keep throwing steps before the pointer move in activate() and release prep bytes before clearing the adoption flag, matching the install contract. - Add the required certificate members to the CLI test fakes (the new interface members broke the packages/cli build) and cover terminalError/closedReason on the real subscription client. Generated-by: Devin
The classification at #failAttempt was "unclassified error = terminal", so every new failure shape reaching the channel needed an exemption or it tore a healthy subscription down — the capacity RangeError, transient Host operation errors through the pump, and the next undiscovered shape. The invariant is now stated at the boundary: an attempt dies only on positive evidence of subscription death — a recoverable subscription failure or a subscription-scoped error — and unclassified errors are absorbed, but only for read-only callers whose work is safe to retry on the next frame (a catch-up's watermark is still ahead). Non-transcript frame failures may be committed mutations that will not be replayed, so they still die loudly. #failAttempt reports whether it consumed the failure so the pump keeps consuming absorbed errors. - Drop TranscriptCacheCapacityError: the whitelist makes a dedicated exemption type unnecessary. - Make terminalError/closedReason optional on the public RuntimeHostSessionSubscription — they are owner introspection, not contract, and fakes that never produce a certificate need not declare them. Reverts the four CLI test-fake additions. - Keep the certificate recording order (reason before the queue offer), closedReason's unconditional precedence, the open/not_found deleted normalization, and the activate/adopt ordering fixes. Generated-by: Devin
Reverts adoptResidentAccounting to the flag-then-release order: the release can only throw on a prior accounting corruption, which a retry cannot repair, so guarding it was dead weight. Drops the generic-Error no-latch test — its protection is strictly subsumed by the transient-operation-error case, which sits on the instanceof boundary and additionally fails if the latch stops checking the operation code. Generated-by: Devin
Summary
A Maka desktop instance spent ~1h at high CPU in the main process. The cause was a transcript catch-up rejection loop:
DesktopTranscriptReplica.advance()re-armed its read in.finallywhenever the target sequence was still ahead, with no error classification, backoff, or bound. When the replica became orphaned — teardown never reached it because the pump was stuck awaiting the subscription iterator'sreturn()(the close handshake) before it could dispatch the failure — the same dead-subscription read retried as a pure-microtask storm.Retry authority moves to the subscription owner, the only layer with the information to classify failures:
RuntimeHostSubscriptionError— including the replica's own correlation violation — or asession.transcript.pagenot_foundfrom a catch-up read is stored; lateradvance()and all synchronous reads reject with that same error (ahead of the generic closed error). Only a successful catch-up re-arms once to cover frames that arrived mid-read. Transient operation failures stay retryable — the one caller that swallows a rejection is the post-settle re-arm, and latching there would turn a retryable blip into a sticky terminal on the next frame — and renderer delivery failures never touch the replica.advance()takes no argument. It readshandle.transcriptWatermark, which the client updates before handing a frame to the consumer — the single authoritative watermark replaces the replica's mirroredtargetThrough. The#enqueue/#operationTailserial queue collapses into a single in-flight guard.#failAttemptinside the loop, instead of escaping throughfor awaitand queueing behind the iterator's close handshake.DesktopTranscriptReplica.reseed()reads the current tail at the live watermark on the same handle, installs it through the same path asprepare(), and letsadvance()cover rows committed during the fetch.refresh(), the retiring phase,pendingFrames, and theMAX_PENDING_*constants are deleted.installReseededReplicadep inside the owner's staleness check, sostate.replicamoves before the evicted replica closes — the same atomicityactivate()has. The failed attempt detaches before the first teardown await, anddiscard()/trimDurable()check residency before liveness so a closed replica is a no-op during recovery windows.session.transcript.pagenot_foundreaches#failAttemptexactly like a pump-frame failure instead of stranding the session on a dead handle. A fetch that races the subscription's death only sees its dead-state mask (connection_closed), which could invert a recoverable close into a terminal teardown; the subscription records its real reason (terminalErrorfor afail(),closedReasonfor asubscription.closedframe — recorded before the queue offer, so a full client queue cannot discard it) and#failAttemptresolves that certificate on every masked entry path — reseed, pumpacceptFrame, and post-commit catch-up alike. A recorded Host close is always the earlier death statement, so it outranks even a laterterminalErrormask. A reseed that loses the staleness check waits out the in-flight recovery, so callers pick up the installed replica instead of erroring on the closed one.resident. The latch made the replica a dead read model while everyresidentgate still saw it as usable — acks threw the latched error into unrelated IPC, and an open/read could not reseed it.residentnow reflects liveness, so a failed replica takes the same reseed/recovery path as an evicted one; its durable bytes stay reclaimable through the internal field.message_admissionevents stayed stale until a full recovery rebuilt it. The install now notes the reseeded tail the way a publish would (idempotent — already-mapped messages emit nothing), and the install-time budget pass can no longer discard the replica it just installed.#failAttemptused to classify "unrecognized error" as terminal, so every new failure shape reaching it — a cache-capacityRangeErrorduring a decode, a transient Host operation error through the pump — tore a healthy subscription down. The invariant is now stated at the boundary: an attempt dies only on positive evidence of subscription death — a recoverable subscription failure or a subscription-scoped error. Unclassified errors are absorbed, but only for read-only callers (catch-up, reseed reads) whose work retries on the next frame; a non-transcript frame's failure may be a committed mutation that will not be replayed, so it still dies loudly.#failAttemptreports whether it consumed the failure so the pump keeps consuming absorbed errors instead of exiting unconditionally.subscription.opennot_foundis a removed session. A recovery that loses the race to a deletion learned it as a generic operation error and broadcasthost_observation_errorinstead of the existingsessions:changed deletedsignal; the terminal boundary now normalizes it toSessionRemovedSubscriptionError.installReseededReplicathe pointer moves right after the only throwing steps — a throw still leaves the state on the evicted replica for the owner to close — while consumer resets and the budget pass see the installed replica;activate()keeps the same order (throwing steps, then the pointer). The post-commit catch-up runs as an owner-side attempt read whose rejection takes the same#failAttemptpath as a pump-frame failure, instead of leaving a latched-dead replica on a quiet stream.onChangeno longer caches snapshots for replicas that were never installed, and evicted replicas no longer track a watermark nothing reads.Net effect: three retry/recovery authorities become one — the owner — and the replica is a read model that dies on its first subscription-scoped failure instead of outliving its subscription.
No protocol or persisted-state changes: the replica/subscription lifecycle is internal to the desktop main process. Behavior change is limited to failure semantics — a dead subscription's replica now fails fast and recovers through the owner instead of retrying reads forever.
Verification
node --teston affected suites — 144/144 desktop transcript/observer/owner + 23/23 runtime-host subscription client + 180/180 CLI session tests passRuntimeHostSubscriptionErrorandRuntimeHostOperationErrorvariants) and reports!resident; generic read errors stay retryable; evicted replicas resolve frames quietly without reading; catch-up re-reads a watermark that moved mid-fetchreturn()is still blockedopens === 2); concurrent reseeds serialize into exactly one swap; a reseed committing inside recovery's teardown window loses the swap and stays pending until recovery installs the replacement — and propagates the terminal error when that recovery fails; aconnection_closed-masked fetch classifies by the subscription's recorded reason (recoverableslow_consumerheals,session_removedstill reports terminal removal); a throwing install closes the orphan and leaves the evicted replica untouched; a post-commit catch-up failure recovers the subscriptiontranscriptGlobalCacheMaxBytesdep makes eviction reachable without 64MB); the budget pass meets a recovery-closed replica as a no-op; a snapshot published before install never reaches the transcript cache; rows durable during the eviction gap reach the surviving projector's durable map at install (message_admissionemitted); acknowledging the tail of a latched replica is a quiet no-op; a replica latched by the swallowed re-arm reseeds through recovery for the next reader; a catch-up decode rejected by the cache-capacity gate leaves the subscription alive and the next frame still deliversClientSessionSubscription): the close reason is recorded before a full queue can reject the frame; a transcript read afterfail()surfaces the terminal error, not theconnection_closedmaskruntimeHostSessionFixturedefaults empty transcript/events/streams and hosts the shared queue/snapshot/page helpers three files each defined locallynpm run format,npm run lint,build:main/maka-agent/@maka/runtime-hosttypechecks cleanAI use
Select exactly one:
Tool(s) and scope: Devin — implementation, tests, and iterative adversarial review of the fix.
Checklist
Does this PR entail a change in behavior?