fix(sandbox): serialize Docker recreation handoff

[apurvvkumaria]

Summary

Registry-only rebuild now removes one exactly labeled OpenShell Docker orphan before replacement creation and stops on ambiguous ownership. Restart-safe startup-command recreation now waits for the OpenShell create client to finish its ownership handoff before Docker cutover, preventing the replacement race that left sandboxes unready.

Related Issue

Fixes #8720

Changes

• Add one narrowly scoped Docker orphan-cleanup helper that queries exact OpenShell ownership labels across all container states, removes only a single immutable container ID, and confirms absence before stale rebuild continues.
• Fail stale rebuild closed when Docker discovery, removal, confirmation, or ownership cardinality is ambiguous; legacy registry rows without a recorded driver retain best-effort non-Docker compatibility.
• Defer non-GPU restart-safe container recreation until the Ready-triggered OpenShell create client has terminated, with a bounded abort path when ownership handoff does not finish.
• Add targeted tests for exact orphan cleanup, fail-closed cleanup failures, legacy missing-driver compatibility, the create-process termination barrier, and its bounded timeout.
• Keep the production implementation to 96 changed lines (93 additions and 3 deletions); the full PR is 287 changed lines including 191 test lines.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification: this internal ordering and ownership fix restores the rebuild behavior already documented in docs/manage-sandboxes/recover-rebuild-sandboxes.mdx; it adds no command, flag, configuration, schema, policy, or user procedure.
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: self-review confirmed deletion is restricted to one immutable ID returned by exact OpenShell managed-by and sandbox-name labels, ambiguous or failed discovery/removal/confirmation stops before rebuild, and restart-safe cutover cannot begin while the create client still owns the transition. No credential or policy boundary changes.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: no-docs-needed
• Evidence: No documentation files changed; docs/manage-sandboxes/recover-rebuild-sandboxes.mdx already documents registry-only recreation and fail-closed ownership behavior.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: the original 8 focused CLI files passed 97/97 tests and test/rebuild-stale-recovery.test.ts passed 5/5 integration tests; after advisor-requested additions, the affected 3-file CLI slice passed 44/44 and the rebased orphan suite passed 13/13. CLI build and typecheck passed.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not applicable to this bounded two-branch lifecycle fix; repository checks, formatting, test-title, test-size, source-shape, and Vitest-project gates passed.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only)
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

The Docker-backed live E2E scenarios require a reachable Docker daemon and are left to normal required CI because Docker is unavailable on the local host.

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c7c39eab-863e-4288-80c4-33f369015882

📥 Commits

Reviewing files that changed from the base of the PR and between 2cfe2df and 64b5153.

📒 Files selected for processing (2)

• src/lib/actions/sandbox/rebuild-gateway-drift.test.ts
• src/lib/onboard/openshell-docker-sandbox-containers.ts

---

📝 Walkthrough

Walkthrough

The PR adds restart-safe Docker sandbox creation handoff and stale Docker orphan cleanup. Ready handling can wait for client termination. Registry-only rebuilds remove exactly one labeled orphan before recreation and fail closed for ambiguous or unsuccessful cleanup.

Changes

Sandbox ownership recovery

Layer / File(s)	Summary
Restart-safe creation handoff src/lib/sandbox/create-stream.ts, src/lib/onboard/sandbox-create-step.ts, src/lib/onboard/sandbox-create-step.test.ts	Creation waits for the OpenShell client to exit after Ready when required. It sends SIGTERM, uses a timeout-backed SIGKILL fallback, and resolves successful status 143 as forcedReady.
Stale Docker orphan cleanup src/lib/onboard/openshell-docker-sandbox-containers.ts, src/lib/actions/sandbox/rebuild-flow-helpers.ts, src/lib/actions/sandbox/rebuild-gateway-drift.test.ts	Registry-only recovery removes one exactly labeled Docker orphan before recreation. Ambiguous matches and query, removal, or confirmation failures stop recovery.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OpenShellCreateClient
  participant createStream
  participant runSandboxCreateStep
  OpenShellCreateClient->>createStream: report Ready
  createStream->>OpenShellCreateClient: send SIGTERM
  OpenShellCreateClient-->>createStream: close with status 143
  createStream-->>runSandboxCreateStep: resolve forcedReady

Loading

sequenceDiagram
  participant RebuildRecovery
  participant removeStaleRebuildDockerOrphan
  participant Docker
  RebuildRecovery->>removeStaleRebuildDockerOrphan: remove sandbox orphan
  removeStaleRebuildDockerOrphan->>Docker: query labeled containers
  Docker-->>removeStaleRebuildDockerOrphan: return matches
  removeStaleRebuildDockerOrphan->>Docker: force-remove one match
  Docker-->>removeStaleRebuildDockerOrphan: confirm removal
  removeStaleRebuildDockerOrphan-->>RebuildRecovery: complete cleanup

Loading

Possibly related PRs

• NVIDIA/NemoClaw#8517: Modifies related Docker resource filtering and removal behavior.
• NVIDIA/NemoClaw#8583: Modifies related sandbox recreation and Docker lifecycle handling.
• NVIDIA/NemoClaw#8715: Refactors stale-recovery tests for the same rebuild flow.

Suggested labels: area: sandbox, bug-fix

Suggested reviewers: cv, ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The implementation meets [#8720] by removing exact Docker orphans, failing closed on cleanup errors, and serializing restart-safe handoff with targeted tests.
Out of Scope Changes check	✅ Passed	All changes support [#8720], remain narrowly scoped, and add no unrelated refactors or production-code expansion beyond the issue constraints.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: serializing Docker recreation to fix ownership handoff issues.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-8720-docker-handoff

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 24a8d82 in the codex/fix-8720-docke... branch remains at 96%, unchanged from commit b414e22 in the main branch.

---

_{Updated August 10, 2026 10:01 UTC}

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 1 suggestion
• Model comparison: normalized findings differ; normalized terminology decisions differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 more warning, 1 more suggestion.

5 terminology differences from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

• Docker orphan at src/lib/onboard/openshell-docker-sandbox-containers.ts:106: primary classified it as justified; the second opinion classified it as define.
• restart-safe at src/lib/onboard/sandbox-create-step.test.ts:229: selected only by the second-opinion lane as justified.
• create ownership handoff at src/lib/sandbox/create-stream.ts:437: selected only by the second-opinion lane as define.
• registry-only rebuild at src/lib/actions/sandbox/rebuild-gateway-drift.test.ts:210: selected only by the second-opinion lane as define.
• Ready ownership handoff at src/lib/sandbox/create-stream.test.ts:174: selected only by the second-opinion lane as define.

3 additional E2E selections from the second opinion

Advisory only. The primary lane did not select these E2E jobs or targets.

• sessions-agents-cli: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• channels-stop-start: The completed second-opinion lane identified E2E coverage that the primary lane omitted.
• openshell-gateway-upgrade: The completed second-opinion lane identified E2E coverage that the primary lane omitted.

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• justified — ownership handoff at src/lib/sandbox/create-stream.ts:437: Retain the term. The changed message and tests identify the required client-exit boundary.
• justified — restart-safe recreation at src/lib/onboard/sandbox-create-step.test.ts:229: Retain the modifier in the test title because it names the ordering condition under test.
• justified — Docker orphan at src/lib/onboard/openshell-docker-sandbox-containers.ts:106: Retain the term. The helper enforces the single-container and exact-label constraints.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: managed-image-multiarch-startup, managed-image-protected-runtime, onboard-repair, onboard-resume, rebuild-openclaw, state-backup-restore, ubuntu-repo-docker-post-reboot-recovery, cloud-onboard

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.