Skip to content

fix(sandbox): repair legacy gateway upgrade recovery - #8735

Merged
apurvvkumaria merged 1 commit into
mainfrom
codex/fix-e2e-gateway-upgrades
Aug 10, 2026
Merged

fix(sandbox): repair legacy gateway upgrade recovery#8735
apurvvkumaria merged 1 commit into
mainfrom
codex/fix-e2e-gateway-upgrades

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

This change repairs two legacy OpenShell gateway upgrade failures from the live E2E workflow. The v0.0.36 fixture now probes its historical Docker network, and installer-driven recovery reuses the validated pre-upgrade backup for stale Ready or Running sandboxes instead of attempting a second backup from the replaced runtime.

Changes

  • Route the v0.0.36 host firewall fixture to openshell-cluster-nemoclaw. That cluster-era gateway creates a gateway-named Docker network, while newer fixtures still require the shared openshell-docker default. The OpenShell gateway upgrade boundary test protects this compatibility path.
  • Treat stale Ready or Running sandboxes on the selected gateway as prepared-recovery candidates when the installer has completed its strict backup. A direct live rebuild is insufficient after gateway replacement because the legacy sandbox state can no longer be inspected for another backup. Recovery tests verify prepared-manifest reuse, ordinary rebuild behavior outside installer restore intent, and fail-closed handling when no validated backup exists.
  • Document the prepared-backup behavior for stale Ready and Running sandboxes in the shared sandbox update guide.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop reviewed the data-safety boundary. Installer restore intent follows a strict backup, selected-gateway binding remains required, recovery manifests remain validated, and a missing prepared backup stops without rebuilding.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: docs-updated
  • Evidence: docs/manage-sandboxes/update-sandboxes.mdx
  • Agent: Codex Desktop

DGX Station Hardware Evidence

  • Tested on DGX Station
  • Tested commit:
  • Station profile/scenario:
  • Result:
  • Supporting evidence:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: gateway workflow boundary 10/10, upgrade recovery 40/40, and prepared rebuild recovery 11/11 passed with targeted Vitest commands. npm run typecheck:cli, npm run check:installer-hash, and semantic E2E checks also passed.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: Not run because the change is confined to one upgrade recovery path and one live fixture boundary; targeted tests and normal hooks passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — the docs build passed with Fern's two existing hidden warnings.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved sandbox recovery during gateway upgrades by reusing validated pre-upgrade backups.
    • Prevented duplicate backups and rebuild attempts for stale or already-assessed sandboxes.
    • Recovery now fails safely when required backup validation is missing or identities do not match.
    • Preserved compatibility with legacy gateway networking behavior during upgrades.
  • Tests

    • Added coverage for backup validation, stale sandbox recovery, and legacy network selection.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria self-assigned this Aug 10, 2026
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The installer now reuses validated pre-upgrade backups for stale or non-Ready sandbox recovery. Recovery tests cover stale, missing-backup, and disabled-restore cases. Gateway upgrade fixtures now select legacy Docker networks by gateway reference.

Changes

Sandbox recovery

Layer / File(s) Summary
Recovery candidate classification
src/lib/actions/upgrade-sandboxes.ts
Recovery now includes stale Ready or Running sandboxes and excludes current live sandboxes.
Prepared recovery and rebuild orchestration
src/lib/actions/upgrade-sandboxes.ts, src/lib/actions/upgrade-sandboxes-recovery.test.ts, docs/manage-sandboxes/update-sandboxes.mdx, scripts/install.sh
Recovery reuses validated pre-upgrade backups, avoids duplicate rebuild processing, updates status messages, and documents the recovery requirements. Tests cover successful, disabled, and missing-backup cases.

Legacy gateway network selection

Layer / File(s) Summary
Legacy Docker network fixture selection
test/e2e/live/openshell-gateway-upgrade-helpers.ts, test/e2e/live/openshell-gateway-upgrade.test.ts, test/e2e/support/openshell-gateway-upgrade-workflow-boundary.test.ts
The helper maps supported legacy references to Docker network behavior. Upgrade fixtures use the mapping, and boundary tests cover supported and unsupported references.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Installer
  participant RecoveryLogic
  participant BackupStore
  participant ReplacementGateway
  participant Sandbox
  Installer->>RecoveryLogic: assess stale or non-Ready sandbox
  RecoveryLogic->>BackupStore: load validated pre-upgrade backup
  BackupStore-->>RecoveryLogic: return matching backup
  RecoveryLogic->>ReplacementGateway: recreate sandbox with recovery manifest
  ReplacementGateway-->>Sandbox: report Ready or Running state
  RecoveryLogic-->>Installer: report recovery result
Loading

Possibly related PRs

Suggested labels: area: sandbox, area: e2e, bug-fix

Suggested reviewers: cv, ericksoa

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: repairing sandbox recovery during legacy gateway upgrades.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-e2e-gateway-upgrades

Comment @coderabbitai help to get the list of available commands.

@apurvvkumaria
apurvvkumaria enabled auto-merge (squash) August 10, 2026 11:47
@github-code-quality

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit af2dcfb in the codex/fix-e2e-gatewa... branch remains at 96%, unchanged from commit 521ea0f in the main branch.

@github-actions

Copy link
Copy Markdown
Contributor

@apurvvkumaria
apurvvkumaria merged commit acf810d into main Aug 10, 2026
71 of 72 checks passed
@apurvvkumaria
apurvvkumaria deleted the codex/fix-e2e-gateway-upgrades branch August 10, 2026 11:48

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/manage-sandboxes/update-sandboxes.mdx`:
- Around line 81-85: Update the recovery requirements in the paragraph beginning
“During installer-driven recovery” to state that registry evidence of
NemoClaw-managed image ownership is required unless the exact-name
pre-fingerprint OpenClaw or Hermes confirmation exception applies. Align the
wording with the recovery behavior described in the surrounding lines so this
supported fallback remains explicitly available.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fe22a6b9-a389-4623-8ef8-d49b9c3f0f6e

📥 Commits

Reviewing files that changed from the base of the PR and between 006c3dd and af2dcfb.

📒 Files selected for processing (7)
  • docs/manage-sandboxes/update-sandboxes.mdx
  • scripts/install.sh
  • src/lib/actions/upgrade-sandboxes-recovery.test.ts
  • src/lib/actions/upgrade-sandboxes.ts
  • test/e2e/live/openshell-gateway-upgrade-helpers.ts
  • test/e2e/live/openshell-gateway-upgrade.test.ts
  • test/e2e/support/openshell-gateway-upgrade-workflow-boundary.test.ts

Comment on lines +81 to +85
During installer-driven recovery, each stale or non-Ready sandbox requires a validated latest backup.
The backup's sandbox and agent identities must match the registry.
The registry must also contain positive evidence that NemoClaw managed the sandbox image.
If the replacement gateway reports a stale sandbox as Ready or Running, the installer reuses the validated pre-upgrade backup.
It does not attempt another backup from the replaced legacy runtime.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

State the managed-image confirmation exception.

Line 83 says that positive registry evidence is always required. The recovery code also permits an exact-name pre-fingerprint OpenClaw or Hermes confirmation when that registry evidence is absent. This conflicts with lines 87-92 and can make the supported recovery path appear unavailable.

Proposed fix
- The registry must also contain positive evidence that NemoClaw managed the sandbox image.
+ The registry must contain positive evidence that NemoClaw managed the sandbox image, unless a listed pre-fingerprint OpenClaw or Hermes entry has explicit managed-image confirmation.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
During installer-driven recovery, each stale or non-Ready sandbox requires a validated latest backup.
The backup's sandbox and agent identities must match the registry.
The registry must also contain positive evidence that NemoClaw managed the sandbox image.
If the replacement gateway reports a stale sandbox as Ready or Running, the installer reuses the validated pre-upgrade backup.
It does not attempt another backup from the replaced legacy runtime.
During installer-driven recovery, each stale or non-Ready sandbox requires a validated latest backup.
The backup's sandbox and agent identities must match the registry.
The registry must contain positive evidence that NemoClaw managed the sandbox image, unless a listed pre-fingerprint OpenClaw or Hermes entry has explicit managed-image confirmation.
If the replacement gateway reports a stale sandbox as Ready or Running, the installer reuses the validated pre-upgrade backup.
It does not attempt another backup from the replaced legacy runtime.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/manage-sandboxes/update-sandboxes.mdx` around lines 81 - 85, Update the
recovery requirements in the paragraph beginning “During installer-driven
recovery” to state that registry evidence of NemoClaw-managed image ownership is
required unless the exact-name pre-fingerprint OpenClaw or Hermes confirmation
exception applies. Align the wording with the recovery behavior described in the
surrounding lines so this supported fallback remains explicitly available.

@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized terminology decisions differ; normalized E2E selections match; severity counts match.

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

1 semantic terminology decision

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • justified — installer-driven recovery at docs/manage-sandboxes/update-sandboxes.mdx:81: Retain this modifier where the installer restore-intent condition distinguishes the behavior.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: cloud-onboard, managed-image-multiarch-startup, openshell-gateway-upgrade, rebuild-openclaw, state-backup-restore

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

cv pushed a commit that referenced this pull request Aug 10, 2026
<!-- markdownlint-disable MD041 -->
## Summary

This docs-only follow-up to #8735 clarifies the managed-image evidence
required during installer-driven legacy gateway recovery. OpenClaw and
Hermes retain the exact-name pre-fingerprint confirmation exception,
while Deep Agents still requires positive registry evidence.

## Changes

- Render the pre-fingerprint confirmation exception only in the OpenClaw
and Hermes guide variants.
- State that explicit managed-image confirmation is the only exception
to the registry evidence requirement.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: Docs-only clarification of
behavior already tested and merged in #8735.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Documentation Writer Review

- [x] Documentation writer subagent reviewed the completed changes
- Result: `docs-updated`
- Evidence: `docs/manage-sandboxes/update-sandboxes.mdx`
- Agent: Codex Desktop
<!-- docs-review-head-sha: a3a5654 -->
<!-- docs-review-agents-blob-sha: c4923a3 -->

## DGX Station Hardware Evidence

- [ ] Tested on DGX Station
- Tested commit:
- Station profile/scenario:
- Result:
- Supporting evidence:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run validate:pr` passed after refreshing `origin/main` when hooks
were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result or justification: `npm
run docs` and `npx prek run --files
docs/manage-sandboxes/update-sandboxes.mdx` passed; generated variants
preserve the agent-specific requirements.
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — command/result: Not applicable to this
one-page docs-only follow-up.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — passed
with Fern's two existing hidden warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@wscurran wscurran added bug-fix PR fixes a bug or regression area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery labels Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: sandbox OpenShell sandbox lifecycle, runtime, config, or recovery bug-fix PR fixes a bug or regression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants