Summary
For a synchronous wasip2 guest (wasi:cli run world, wasmtime run -S inherit-network) doing nonblocking socket I/O in an event loop, output-stream.check-write intermittently reports "no budget" for every open socket simultaneously, for hundreds of milliseconds up to multiple seconds, while the kernel reports the sockets writable the whole time (poll(2) with zero timeout returns POLLOUT). All sockets then unblock at the same instant. From the guest's side this looks like its entire socket set freezing; in our production system (a streaming app serving one busy SSE writer plus several keep-alive connections) stalls reached 14 s.
Mechanism (instrumented)
The CLI executes the whole guest inside one outer Runtime::block_on, so every sync host shim entered through wasmtime_wasi::runtime::in_tokio shares that single task's tokio coop budget (128). tokio::io::unix Registration::poll_ready — the path under p2 check-write / poll_write_ready — consults coop::poll_proceed before consulting readiness. When the budget is exhausted, every socket's readiness poll returns Pending regardless of kernel state. The sync shims poll with noop wakers, so the normal contract ("you'll be woken when the budget resets") is void — nothing re-polls until an unrelated host call happens to yield, which is why the whole set freezes and releases together.
We proved this by re-polling under tokio::task::unconstrained at each divergence: 1,004 of 1,006 write-side divergences flipped to Ready with the budget removed. (The remaining handful, plus a smaller read-side class, are ordinary edge-cache lag: try_read/try_write short-circuit on cached readiness bits, and an edge consumed at the wrong moment never re-fires for a socket that stays ready.) In one 5-minute window under load we counted 2,098 "kernel says POLLOUT, cache says Pending" events. A 10 ms heartbeat task on the same runtime logged zero scheduling gaps >100 ms through every stall — the workers and driver were healthy; the budget gate was the blocker.
try_read/try_write themselves have no coop gate, which is why reads kept limping along while check-write starved — a confusing asymmetry when debugging from the guest side.
Reproduction shape
Sync p2 guest with an event loop that, each turn, does nonblocking reads/writes across several sockets (one continuously-written stream + a few keep-alive request sockets), under CPU contention (we pin the whole process to one core with taskset; on 32 idle cores it stays clean). Within minutes, check-write returns 0 for all sockets in bursts of 0.2–1.2 s locally (much longer on smaller cgroup slices in production). wasmtime @ ac0772970b9ad2cd53866d95db69e26311fe3b75; also reproduces with our threads patches removed (a second guest thread merely multiplies the frequency ~7-12x).
Fixes we validated
- Level-check fallback (what we ship): on the sync send/recv/readiness paths, when the cached path says "not ready", ask the kernel (direct nonblocking send/recv or zero-timeout
poll(2)) before believing it; register the waker before answering Ready so async callers keep their contract. This eliminated the stalls entirely under a 30-minute soak (12,594/12,594 probes clean, worst 16 ms; previously 59 stalls >500 ms per 10 min) and also absorbs the edge-cache-lag class. Patch (against the commit above, plain-git apply): https://github.com/EnclaveHost/enclave/blob/main/wasm/wasmtime-socket-level-check.patch
tokio::task::unconstrained around the sync shims' polls fixes the coop class alone (but not the edge-lag class).
Happy to adapt the patch into a PR if the approach is acceptable — or if you'd prefer the unconstrained route for the sync shims, that's a smaller change that covers ~99.8% of the observed events.
Summary
For a synchronous wasip2 guest (
wasi:clirun world,wasmtime run -S inherit-network) doing nonblocking socket I/O in an event loop,output-stream.check-writeintermittently reports "no budget" for every open socket simultaneously, for hundreds of milliseconds up to multiple seconds, while the kernel reports the sockets writable the whole time (poll(2)with zero timeout returns POLLOUT). All sockets then unblock at the same instant. From the guest's side this looks like its entire socket set freezing; in our production system (a streaming app serving one busy SSE writer plus several keep-alive connections) stalls reached 14 s.Mechanism (instrumented)
The CLI executes the whole guest inside one outer
Runtime::block_on, so every sync host shim entered throughwasmtime_wasi::runtime::in_tokioshares that single task's tokio coop budget (128).tokio::io::unixRegistration::poll_ready— the path under p2check-write/poll_write_ready— consultscoop::poll_proceedbefore consulting readiness. When the budget is exhausted, every socket's readiness poll returnsPendingregardless of kernel state. The sync shims poll with noop wakers, so the normal contract ("you'll be woken when the budget resets") is void — nothing re-polls until an unrelated host call happens to yield, which is why the whole set freezes and releases together.We proved this by re-polling under
tokio::task::unconstrainedat each divergence: 1,004 of 1,006 write-side divergences flipped to Ready with the budget removed. (The remaining handful, plus a smaller read-side class, are ordinary edge-cache lag:try_read/try_writeshort-circuit on cached readiness bits, and an edge consumed at the wrong moment never re-fires for a socket that stays ready.) In one 5-minute window under load we counted 2,098 "kernel says POLLOUT, cache says Pending" events. A 10 ms heartbeat task on the same runtime logged zero scheduling gaps >100 ms through every stall — the workers and driver were healthy; the budget gate was the blocker.try_read/try_writethemselves have no coop gate, which is why reads kept limping along whilecheck-writestarved — a confusing asymmetry when debugging from the guest side.Reproduction shape
Sync p2 guest with an event loop that, each turn, does nonblocking reads/writes across several sockets (one continuously-written stream + a few keep-alive request sockets), under CPU contention (we pin the whole process to one core with
taskset; on 32 idle cores it stays clean). Within minutes,check-writereturns 0 for all sockets in bursts of 0.2–1.2 s locally (much longer on smaller cgroup slices in production). wasmtime @ac0772970b9ad2cd53866d95db69e26311fe3b75; also reproduces with our threads patches removed (a second guest thread merely multiplies the frequency ~7-12x).Fixes we validated
poll(2)) before believing it; register the waker before answering Ready so async callers keep their contract. This eliminated the stalls entirely under a 30-minute soak (12,594/12,594 probes clean, worst 16 ms; previously 59 stalls >500 ms per 10 min) and also absorbs the edge-cache-lag class. Patch (against the commit above, plain-git apply): https://github.com/EnclaveHost/enclave/blob/main/wasm/wasmtime-socket-level-check.patchtokio::task::unconstrainedaround the sync shims' polls fixes the coop class alone (but not the edge-lag class).Happy to adapt the patch into a PR if the approach is acceptable — or if you'd prefer the
unconstrainedroute for the sync shims, that's a smaller change that covers ~99.8% of the observed events.