fix(fsmount): don't let a signal on the caller abort the RPC behind a FUSE request - #13
Merged
Merged
Conversation
… FUSE request go-fuse cancels a handler's ctx when the kernel sends FUSE_INTERRUPT, and the kernel sends it for any signal landing on the calling process mid-request, not just fatal ones. A profiler's SIGPROF arrives every ~10ms, faster than a round-trip to the pod completes, so every Stat/ReadFile behind an open() of a mounted volume was cancelled. grpc-go reports that as a Canceled status, which errnoFromGRPC had no case for, so the caller saw EIO on its first read of a mounted configmap and died at import. api-sandboxes under `bridge dev` with DD_PROFILING_ENABLED=true reproduced this on every start. Mapping Canceled to EINTR is correct but not enough: callers retry EINTR, get interrupted again before the RPC completes, and livelock. So detach the RPC from the request's interrupt (context.WithoutCancel) with a 60s ceiling in its place. Fatal signals are unaffected; the kernel abandons the request on its side without waiting for us. Verified in a devcontainer against a live bridge: 40 opens under a 1kHz SIGALRM storm went from 40/40 EIO to 40/40 OK, and api-sandboxes with the profiler on starts and serves (300 SIGPROF deliveries in 20s, its FUSE open returns an fd). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Any service that reads a mounted volume at import while the Datadog profiler is on dies under
bridge devwithEIO: i/o error, open '/hive-api-config/hive-api-config.json'. api-sandboxes reproduced it on every start; api-devbox only escapes because it happens not to read a mounted file at import.The chain, each link verified against the library source:
FUSE_INTERRUPTfor any signal landing on the calling process mid-request — not just fatal ones. dd-trace's profiler deliversSIGPROFviatgkillevery ~10 ms.FUSE_INTERRUPT(fuse/protocol-server.go:111,126) and hands our handlers a ctx built on it (fs/bridge.go:360).Stat/ReadFileRPC to the pod takes tens of ms — longer than the profiler's period — so it is cancelled essentially every time.codes.Canceledstatus, not a barecontext.Canceled(status/status.go:159), soerrnoFromGRPC'serrors.Is(err, context.Canceled) → EINTRbranch is dead code for gRPC errors and the switch falls todefault: EIO.What
rpcContext()— every RPC (Getattr,Lookup,Readdir,Read,Readlink) runs oncontext.WithTimeout(context.WithoutCancel(ctx), 60s). The interrupt no longer aborts the round-trip; a non-fatal signal just gets delivered one round-trip later. Fatal signals are unaffected — the kernel abandons the request on its side (wait_event_killable) without waiting for us. The 60 s ceiling exists because nothing else can now end an RPC to a dead pod.codes.Canceled → EINTRinerrnoFromGRPC— the mapping the code already intended, made reachable. Still matters for the remaining cancel paths.Why not just EINTR
I tried that first. It is the POSIX-correct answer and callers do retry it — which is the problem: with the signal period shorter than the RPC, every retry is interrupted again and the open never completes. The repro livelocked instead of failing. Detaching is the only variant that lets the request finish.
Validation
Table test
TestErrnoFromGRPCpins the exact bug: onmainit fails only oncanceled status.Everything below was run inside a live devcontainer against a real venus bridge pod, swapping the mounted
~/.bridge/bin/bridge-linuxbetween builds:main(17fb108)os.open×40 on a FUSE file,SIGALRM@ 1 kHz/etc/hostname(control)readFileSync(…, 'utf-8')under a signal storm, 3 sDD_PROFILING_ENABLED=trueThe last row under strace, which is the original failing trace with one character changed:
300 SIGPROF deliveries in 20 s; 1 FUSE open; 0 failed.
Worth knowing
The 60 s
rpcTimeoutis a judgment call: long enough for areadChunkSize(1 MiB) chunk over a slow tunnel, short enough that a dead pod returns an error to the caller instead of hanging it. Happy to tune.🤖 Generated with Claude Code