Premature PTY setup in urunc create swallows nsenter failure logs

[safiya2610]

Summary

While reviewing the container lifecycle in cmd/urunc/create.go, I noticed a structural TODO regarding the terminal/PTY setup. Currently, urunc create forcibly attaches a PTY to the reexec process before the CGO nsenter initialization has completed.
Because urunc imports github.com/opencontainers/runc/libcontainer/nsenter, the C code for namespace entry executes prior to the Go runtime in the reexec process. If nsenter fails (e.g., due to missing namespaces or permission errors), it writes its errors to standard output/error and exits.
However, because urunc create uses pty.Start(reexecCommand) when a terminal is requested, all standard output and error from the C-level nsenter are immediately swallowed by the PTY. If the container manager (like containerd) hasn't fully attached to the console socket yet, these critical failure logs vanish into the void, making it nearly impossible for developers to debug early namespace initialization failures.

Affected Code

File: cmd/urunc/create.go (inside createUnikontainer)

	// setup terminal if required and start reexec process
	// TODO: This part of code needs better rhandling. It is not the
	// job of the urunc create to setup the terminal for reexec.
	// The main concern is the nsenter execution before the reexec.
	// If anything goes wrong and we mess up with nsenter debugging
	// is extremely hard.
	if unikontainer.Spec.Process.Terminal {
		ptm, err := pty.Start(reexecCommand)
                // ...

Impact
Poor Debuggability: Failures during early C-level namespace initialization (nsenter) are swallowed, leaving users blind to why their unikernel container failed to start.
Architectural Separation of Concerns: Setting up the execution environment's terminal should be the responsibility of the reexec process (which ultimately sets up and runs the container monitor), rather than the parent urunc create command.
Proposed Solution
Refactor the lifecycle so that terminal setup is deferred until after the reexec process has been successfully spawned and the CGO nsenter block has succeeded.

urunc create should start the reexec process with standard pipes (so nsenter failures are cleanly captured and reported to the parent).
Once nsenter succeeds and hands control over to the Go runtime (reexecUnikontainer), the reexec process itself should handle creating the PTY and establishing the socket connection to send the master file descriptor back to the shim.
urunc create simply waits for the reexec process to signal that the environment (including the terminal) is fully and safely established.

[Nachiket-Roy]

Hey @cmainas this and #890 would go away with #782 right?

[cmainas]

Hello @safiya2610 ,

have you replicated the issue you describe?

Can you back up with some code pointers the statement "If the container manager (like containerd) hasn't fully attached to the console socket yet, these critical failure logs vanish into the void, making it nearly impossible for developers to debug early namespace initialization failures."

[safiya2610]

Hello @safiya2610 ,

have you replicated the issue you describe?

Can you back up with some code pointers the statement "If the container manager (like containerd) hasn't fully attached to the console socket yet, these critical failure logs vanish into the void, making it nearly impossible for developers to debug early namespace initialization failures."

Hello @cmainas,

Thank you for looking into this.

To answer your question regarding the failure logs vanishing: the root cause is a combination of how pty.Start(reexecCommand) handles standard streams and how the parent process handles early initialization errors.

Here is a detailed breakdown with code pointers:

The PTY Setup: At cmd/urunc/create.go:176, urunc create uses pty.Start(reexecCommand) to spawn the reexec process. This immediately redirects both Stdout and Stderr of the reexec process to the slave end of the newly created PTY.

The C-Level Failure: Because urunc relies on runc/libcontainer/nsenter, the C-level namespace initialization executes during the CGO init() phase. If nsenter encounters a fatal error (e.g., a missing namespace or permission issue), it writes the error message to standard error (via fprintf(stderr, ...)) and exits. Since standard error is bound to the PTY, the error message goes directly into the PTY buffer.

The Silent Failure: Meanwhile, urunc create waits for nsenter to succeed by calling handleNsenterRet at cmd/urunc/create.go:235. When nsenter exits with an error, handleNsenterRet fails, causing urunc create to abort and exit with a non-zero status.

Why the logs vanish: If urunc create fails, container managers (like containerd) capture and log urunc create's own standard error. However, because the container creation aborted, the container manager generally closes the PTY master file descriptor without ever reading from it. Thus, the actual nsenter logs sitting in the PTY buffer vanish into the void.

To prove this behavior without needing a complex container manager setup, I created a minimal Go simulator that mimics urunc create:

package main
import (
	"fmt"
	"os"
	"os/exec"
	"github.com/creack/pty"
)
func main() {
	// A mock command mimicking nsenter that fails and writes a C-level error to STDERR
	cmd := exec.Command("sh", "-c", "echo 'C-LEVEL FATAL: nsenter failed to join namespace' >&2; exit 1")
	
	ptm, _ := pty.Start(cmd)
	defer ptm.Close()
	if err := cmd.Wait(); err != nil {
		// Parent failing early without draining the PTY buffer
		fmt.Fprintf(os.Stderr, "urunc create failed: nsenter unsuccessful exit: 1\n")
		os.Exit(1)
	}
}

Output of the simulator:
urunc create failed: nsenter unsuccessful exit: 1
As you can see, the actual error (C-LEVEL FATAL...) is completely swallowed.

SO: To fix this, we should defer the PTY setup to the reexec process itself. The parent urunc create should start the reexec process with standard pipes (as it currently does for the non-terminal case at cmd/urunc/create.go:206-208). This ensures that early nsenter failures surface through urunc create's standard error, which container managers reliably capture and log.

[cmainas]

Hello @safiya2610 ,

one clarification. I was referring to contianerd code pointers not urunc.

[safiya2610]

Hello @safiya2610 ,

one clarification. I was referring to contianerd code pointers not urunc.

The issue occurs in containerd's runc v2 shim (cmd/containerd-shim-runc-v2/process/init.go).

The Setup (Deferring Socket Closure): When a terminal is requested, the shim creates a temporary console socket to receive the PTY master. Crucially, it registers a defer socket.Close() right away. containerd/cmd/containerd-shim-runc-v2/process/init.go#L118-L122

The Early Failure (Bypassing Master Retrieval): The shim then calls the runtime (urunc create). If early namespace initialization (nsenter) fails, the runtime exits with an error. The shim receives this error and returns immediately, completely bypassing the rest of the function. containerd/cmd/containerd-shim-runc-v2/process/init.go#L149-L151

The Vanished Logs: Because the shim returned early in Step 2, execution never reaches the socket.ReceiveMaster() call further down in the function. containerd/cmd/containerd-shim-runc-v2/process/init.go#L159-L161 Instead, the function exits and triggers the defer socket.Close().

Conclusion: Because socket.ReceiveMaster() is never called, containerd never attaches to the PTY master to drain the buffer. The socket is destroyed, and any nsenter failure logs sitting inside that PTY buffer vanish permanently.

Since this is standard and correct error handling on containerd's side, the fix has to be on the urunc side: if we defer the PTY setup to the reexec process (after nsenter succeeds), early failures will just emit to standard error normally, which containerd reliably captures and logs.