cc TUI wedges completely mid-request: event loop spins kevent but dispatches NOTHING — timers frozen, input dead, socket ESTABLISHED, tokio worker starved

[proggeramlug]

Live-captured from a wedged real session (macOS aarch64, main 8d27f7b964 build, ~17 min into authenticated use). The most severe TUI defect observed: total dispatch stop while every component looks alive.

The user-visible sequence

Long conversation → large paste → question submitted → response spinner (Transmuting… (43s)) → counter freezes at 43s → all input dead (typing, ESC, Ctrl-C — nothing echoes or interrupts; raw mode, so Ctrl-C is an in-app byte, not a signal) → stays wedged indefinitely (>10 min). Memory flat at 680 MB throughout (no allocation → not processing).

An earlier, milder sighting in the same session (#9587): multi-second freeze during busy work that then batch-drained on its own. This one never drained. Likely the same defect at different severities; keypress explicitly did NOT unstick either (that hypothesis is dead).

The forensics (5s sample, all preserved)

• Main thread (4158/4158 samples): cycling kevent (3988 in the syscall) + clock_gettime bursts — the loop parks, wakes, checks deadlines, re-parks. It is not stuck; it dispatches nothing.
• stdin reader thread (4158/4158): blocked in read() — alive and waiting, so keys reach the process.
• perry-signal-wake thread: alive.
• tokio-rt-worker: present in only 924/4158 samples — the async runtime's worker was absent for ~78% of the window. The API socket is ESTABLISHED (request sent, response owed) but nothing pumps it.
• kqueues show count=0; fds normal; one live TLS connection to the API.

So: timer wheel says nothing is due (a setInterval spinner at 1s cadence frozen ⇒ the due-check answers wrongly or the wheel lost the timer), stdin notifications don't reach dispatch, and tokio's events don't either. Producers and the loop header have become disconnected — the loop's liveness sees enough to keep parking-and-waking but its dispatch arms never fire.

Suspects, in order

1. fix: the event loop looks before it sleeps (#9441); async fs writes are abandoned on process.exit (#9442) [lands #9492] #9540's look-before-sleep restructure (event_loop.body_check / shared emit_event_loop_liveness) — the loop emission changed recently; a path where the re-check consults different state than the dispatch arms would produce exactly "wakes but never dispatches". The stdin-only programs exit before entering the event loop (40-60% of runs) #9416 two-copies trampoline (stdlib-vs-runtime predicate split) is the precedent for one side consulting a disconnected copy.
2. Timer wheel state after heavy async churn — the freeze began mid-request after a big paste; a corrupted earliest-deadline (huge value) silences all timers while clock_gettime checks keep happening.
3. Tokio runtime starvation/death — if the worker exited or deadlocked, network completion never posts; but that alone doesn't explain frozen timers and dead input, so it may be downstream of 1.

Evidence bundle

/Users/amlug/projects/perry/wedge-9590/: wedge_sample.txt (full thread stacks, unsymbolized — binary stripped), lsof.txt, ps.txt, sbs_ram.log (memory timeline incl. the wedge window). Load address 0x1047f8000; hot frame offsets in the sample (kevent caller at +0xbcb8408, etc.) — a PERRY_KEEP_SYMBOLS=1 rebuild of the same SHA is needed to name them (note: the symbols build links a separate archive, so verify offsets against the symbolized binary's own sample, don't map across).

Repro direction

Not yet reduced. The trigger window: authenticated session, large pasted text, then a question whose response streams. A PTY driver replaying that shape against a symbolized build, sampled the moment the spinner stalls, is the path. The milder self-draining variant may reproduce more easily and share the root.

Severity: highest of the open TUI issues — a hard hang losing the session (work in the input box unrecoverable; transcript survives up to the wedge).

[proggeramlug]

The investigation is done, and this issue's model — my model — is falsified on both pillars. Correcting the record, then the real state.

The stripped field binary was symbolized without a rebuild (function bodies matched against its own archives — 129,071 functions indexed; tools in /Users/amlug/projects/perry/wedge-9590/: macho.py, objidx.py, symbolize.py, plus wedge_sample.SYMBOLIZED.txt). The dispatch census across all 4,158 main-thread samples:

1. "Dispatches nothing" is wrong. The loop was turning at ~40 Hz and every arm fired: microtasks drained (86), interval timers ran JS (55), callback timers ran JS and pumped stdin (14), readline/http/child-process/worker reactors all pumped, all seven liveness predicates evaluated. The park is the designed tokio::time::timeout(budget, notified()) at event_pump.rs:516/246; the clock_gettime bursts are tokio's time driver, not a due-check. fix: the event loop looks before it sleeps (#9441); async fs writes are abandoned on process.exit (#9442) [lands #9492] #9540 / event_loop.body_check is exonerated.
2. "API socket ESTABLISHED with a response owed" is wrong for capture time. The preserved lsof.txt contains zero TCP entries. (A live pre-capture look had shown one — it had closed by the bundle snapshot; lesson: evidence must be atomic with the capture.)
3. The tokio-worker "starvation" was a red herring: that thread is a blocking-pool thread; the scheduler is new_current_thread, driven on the main thread inside run_one_tick — the reactor was polled throughout.

What the wedge actually was: one thread blocked in wait4 inside cp_exec_async — a child process that never exited — alongside twelve leaked cp_spawn_timeout threads (child_process/reactor.rs:474 sleeps the full timeout unconditionally, never observing exit). cc was waiting, correctly idle, on a local child. Why the screen (spinner interval included) appeared frozen while JS timers demonstrably fired is the remaining puzzle — one candidate worth checking next time is terminal ownership (a child sharing the tty / foreground-process-group effects on cc's writes and reads).

Still open — capture recipe for the next occurrence (the user is the sensor): at wedge time, from another terminal: ps -ef filtered to the cc PID's descendants, then sample <child-pid> 5, plus sample <cc-pid> 5 again. Those two artifacts close this.

Fixed on the way (PR #9590): the exact producer/consumer disconnection this issue hypothesized existed — in the input path: the readline reader thread never notified the pump (the only cross-thread producer missing the protocol), keypress latency 603–963 ms when no timer masks it, node 1.3 ms, after 0–2.6 ms; sabotage-verified integration test. Not the wedge (cc's fs.watch 25 ms poll masked it to ~60 ms) — but user-feel-relevant everywhere else.

Also from the census, filed separately: the fs.watch recursive 25 ms full-tree re-walk (~70× node; a 362k-file cwd extrapolates to 1.2 s/poll against a 25 ms schedule — a wedge-shaped loaded gun, though provably not this shot); the cp_spawn_timeout thread leak; bare-ESC latency wanting a real escapeCodeTimeout; and an rl.close() stdin-delivery divergence.