Skip to content

fix(fsmount): don't let a signal on the caller abort the RPC behind a FUSE request - #13

Merged
vercel-eddie merged 1 commit into
mainfrom
eddie/fsmount-interrupt-eintr
Sep 4, 2026
Merged

fix(fsmount): don't let a signal on the caller abort the RPC behind a FUSE request#13
vercel-eddie merged 1 commit into
mainfrom
eddie/fsmount-interrupt-eintr

Conversation

@vercel-eddie

Copy link
Copy Markdown
Collaborator

Why

Any service that reads a mounted volume at import while the Datadog profiler is on dies under bridge dev with EIO: i/o error, open '/hive-api-config/hive-api-config.json'. api-sandboxes reproduced it on every start; api-devbox only escapes because it happens not to read a mounted file at import.

The chain, each link verified against the library source:

  1. The kernel sends FUSE_INTERRUPT for any signal landing on the calling process mid-request — not just fatal ones. dd-trace's profiler delivers SIGPROF via tgkill every ~10 ms.
  2. go-fuse closes the request's cancel channel on FUSE_INTERRUPT (fuse/protocol-server.go:111,126) and hands our handlers a ctx built on it (fs/bridge.go:360).
  3. Our Stat/ReadFile RPC to the pod takes tens of ms — longer than the profiler's period — so it is cancelled essentially every time.
  4. grpc-go reports the aborted call as a codes.Canceled status, not a bare context.Canceled (status/status.go:159), so errnoFromGRPC's errors.Is(err, context.Canceled) → EINTR branch is dead code for gRPC errors and the switch falls to default: EIO.

What

  • rpcContext() — every RPC (Getattr, Lookup, Readdir, Read, Readlink) runs on context.WithTimeout(context.WithoutCancel(ctx), 60s). The interrupt no longer aborts the round-trip; a non-fatal signal just gets delivered one round-trip later. Fatal signals are unaffected — the kernel abandons the request on its side (wait_event_killable) without waiting for us. The 60 s ceiling exists because nothing else can now end an RPC to a dead pod.
  • codes.Canceled → EINTR in errnoFromGRPC — the mapping the code already intended, made reachable. Still matters for the remaining cancel paths.

Why not just EINTR

I tried that first. It is the POSIX-correct answer and callers do retry it — which is the problem: with the signal period shorter than the RPC, every retry is interrupted again and the open never completes. The repro livelocked instead of failing. Detaching is the only variant that lets the request finish.

Validation

Table test TestErrnoFromGRPC pins the exact bug: on main it fails only on canceled status.

Everything below was run inside a live devcontainer against a real venus bridge pod, swapping the mounted ~/.bridge/bin/bridge-linux between builds:

main (17fb108) EINTR only this PR
Python os.open ×40 on a FUSE file, SIGALRM @ 1 kHz 40/40 EIO livelock 40/40 OK
same loop on /etc/hostname (control) 40/40 OK 40/40 OK 40/40 OK
Node readFileSync(…, 'utf-8') under a signal storm, 3 s crashes 0 failures
api-sandboxes, DD_PROFILING_ENABLED=true dies at import starts and serves

The last row under strace, which is the original failing trace with one character changed:

SIGPROF {si_code=SI_TKILL ...}
SIGPROF {si_code=SI_TKILL ...}
SIGPROF {si_code=SI_TKILL ...}
openat(AT_FDCWD, "/hive-api-config/hive-api-config.json", O_RDONLY|O_CLOEXEC) = 20      ← was: = -1 EIO

300 SIGPROF deliveries in 20 s; 1 FUSE open; 0 failed.

Worth knowing

The 60 s rpcTimeout is a judgment call: long enough for a readChunkSize (1 MiB) chunk over a slow tunnel, short enough that a dead pod returns an error to the caller instead of hanging it. Happy to tune.

🤖 Generated with Claude Code

… FUSE request

go-fuse cancels a handler's ctx when the kernel sends FUSE_INTERRUPT, and the
kernel sends it for any signal landing on the calling process mid-request, not
just fatal ones. A profiler's SIGPROF arrives every ~10ms, faster than a
round-trip to the pod completes, so every Stat/ReadFile behind an open() of a
mounted volume was cancelled. grpc-go reports that as a Canceled status, which
errnoFromGRPC had no case for, so the caller saw EIO on its first read of a
mounted configmap and died at import. api-sandboxes under `bridge dev` with
DD_PROFILING_ENABLED=true reproduced this on every start.

Mapping Canceled to EINTR is correct but not enough: callers retry EINTR, get
interrupted again before the RPC completes, and livelock. So detach the RPC
from the request's interrupt (context.WithoutCancel) with a 60s ceiling in its
place. Fatal signals are unaffected; the kernel abandons the request on its
side without waiting for us.

Verified in a devcontainer against a live bridge: 40 opens under a 1kHz SIGALRM
storm went from 40/40 EIO to 40/40 OK, and api-sandboxes with the profiler on
starts and serves (300 SIGPROF deliveries in 20s, its FUSE open returns an fd).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel-eddie
vercel-eddie marked this pull request as ready for review September 4, 2026 19:53
@vercel-eddie
vercel-eddie merged commit 4e43b48 into main Sep 4, 2026
6 checks passed
@vercel-eddie
vercel-eddie deleted the eddie/fsmount-interrupt-eintr branch September 4, 2026 19:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant