Skip to content

fix(nbd): downgrade broken-pipe socket errors from ERROR to WARN#3296

Open
AdaAibaby wants to merge 2 commits into
e2b-dev:mainfrom
AdaAibaby:fix/nbd-broken-pipe-log-level
Open

fix(nbd): downgrade broken-pipe socket errors from ERROR to WARN#3296
AdaAibaby wants to merge 2 commits into
e2b-dev:mainfrom
AdaAibaby:fix/nbd-broken-pipe-log-level

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Problem

When a sandbox VM terminates, the NBD dispatch loop has in-flight goroutines that try to write responses back to the kernel via the Unix socket. Since the socket is already closed, these writes fail with EPIPE ("broken pipe"). The first failure is sent to the fatal channel; all subsequent concurrent goroutines hit the default: branch and log at ERROR:

ERROR nbd error cmd write  {error: write unix @- write: broken pipe, nbd_op: write, ...}
ERROR nbd error cmd write  {error: write unix @- write: broken pipe, nbd_op: write, ...}
...

This generates a burst of ERROR-level log noise on every normal sandbox shutdown, making it harder to spot real failures.

Fix

Add isSocketClosed() to detect syscall.EPIPE and io.ErrClosedPipe and emit WARN instead of ERROR in the default: branches of cmdRead, cmdWrite, and cmdWriteZeroes. Unexpected non-EPIPE errors remain at ERROR.

This is the write-side counterpart of the read-side ErrObjectNotExist downgrade already in dispatch.go.

Test plan

  • Deploy to staging and confirm EPIPE response errors appear as WARN in logs during sandbox shutdown
  • Confirm unexpected write errors (non-EPIPE) still appear as ERROR

When a sandbox shuts down, in-flight NBD response writes to the
kernel Unix socket fail with EPIPE. These are logged at ERROR in
the cmdRead/cmdWrite/cmdWriteZeroes default branches (reached when
the fatal channel is already occupied), generating spurious error
bursts on every normal sandbox termination.

Add isSocketClosed() to detect EPIPE/ErrClosedPipe and emit WARN
instead. Unexpected socket errors (non-EPIPE) remain at ERROR.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a helper function isSocketClosed to detect when an NBD Unix socket is closed by a peer, and updates the cmdRead, cmdWrite, and cmdWriteZeroes dispatch functions to log these expected socket closures as warnings rather than errors. The review feedback suggests a refactoring to dynamically assign the logging function to a variable, which avoids duplicating the extensive list of logging fields across the if/else branches in each of these methods.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread packages/orchestrator/pkg/sandbox/nbd/dispatch.go
Comment thread packages/orchestrator/pkg/sandbox/nbd/dispatch.go
Comment thread packages/orchestrator/pkg/sandbox/nbd/dispatch.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants