fix(hermes): preserve MCP reload and Discord handshake

[apurvvkumaria]

Summary

Hermes rebuilds now restart the gateway before managed MCP restoration and verify the MCP-reloaded process before releasing the cron gate. The Hermes Discord E2E proof now uses an HTTP absolute-form target so aiohttp can complete its native WebSocket Upgrade through OpenShell.

Changes

• Split Hermes post-restore gateway restart from final verification so managed MCP restoration runs between them.
• Preserve fail-closed MCP reconciliation, secret-boundary checks, and stable cron process identity validation before dispatch resumes.
• Change only the existing fake Discord gateway test client's target from ws:// to http://; the fake server and Upgrade, credential rewrite, IDENTIFY, READY, and heartbeat assertions remain unchanged.
• Add focused rebuild-ordering, restart-free verification, and Discord proxy URL regression tests.
• Document the corrected Hermes rebuild lifecycle.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with doc updates
• Doc only (prose changes, no code sample modifications)
• Doc only (includes code sample changes)

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Docs updated for user-facing behavior changes
• Docs not applicable — justification:
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Codex Desktop performed a focused read-only review of gateway restart, MCP reload, reconciliation refusal, and cron identity ordering. The review found no actionable correctness or security issue, and focused tests cover the fail-closed paths.
• Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

• Documentation writer subagent reviewed the completed changes
• Result: docs-updated
• Evidence: docs/manage-sandboxes/recover-rebuild-sandboxes.mdx; npm run docs completed with 0 errors and 2 existing Fern warnings.
• Agent: Codex Desktop

DGX Station Hardware Evidence

• Tested on DGX Station
• Tested commit:
• Station profile/scenario:
• Result:
• Supporting evidence:

Verification

• PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project cli src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts passed 47 tests; the focused e2e-support Discord test passed 1 test.
• Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — full npm test was not run. The rebuild-focused sweep passed 684 tests with 3 skips; one parallel 5-second timeout passed 38 tests when rerun alone.
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — completed with 0 errors and 2 existing Fern warnings.
• Doc pages follow the style guide (doc changes only)
• New doc pages include SPDX header and frontmatter (new pages only)

---

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

• Bug Fixes

  • Improved sandbox recovery by restarting the gateway before restoring managed MCP settings.
  • Added final health and configuration checks without unnecessarily replacing a verified gateway process.
  • Improved handling and reporting of gateway restart failures during recovery.

• Documentation

  • Updated sandbox recovery guidance to reflect the revised gateway and MCP restoration sequence.

• Tests

  • Expanded coverage for gateway identity changes, restart failures, MCP reconciliation, and recovery ordering.
  • Added end-to-end coverage for Discord gateway connections through HTTP proxy upgrades.

[coderabbitai]

📝 Walkthrough

Walkthrough

Hermes rebuild recovery now restarts the gateway before MCP restoration and verifies it afterward without another replacement. Hermes Discord E2E tests now connect through an HTTP proxy gateway URL.

Changes

Hermes recovery sequencing

Layer / File(s)	Summary
Split gateway restart and verification APIs src/lib/actions/sandbox/rebuild-hermes-post-restore.ts, src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts	Gateway restart state and separate verification functions support standard and cron-gated restoration flows. Tests cover stable and changed gateway identities and MCP reconciliation refusal.
Post-restore sequencing and validation src/lib/actions/sandbox/rebuild-post-restore-phase.ts, src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts, src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts, docs/manage-sandboxes/recover-rebuild-sandboxes.mdx	Post-restore handling restarts Hermes before MCP repair, performs final verification afterward, and covers restart failures, ordering, and recovery results. Documentation describes the updated sequence.

Hermes Discord proxy URL

Layer / File(s)	Summary
HTTP proxy gateway URL integration test/e2e/live/hermes-discord-proxy.ts, test/e2e/live/hermes-discord.test.ts, test/e2e/support/hermes-discord-proxy-request.test.ts	The E2E gateway proof uses an http://host:port/gateway proxy URL. Tests validate the generated protocol, host, port, and path.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RebuildPhase
  participant HermesGateway
  participant MCPRestoration
  participant GatewayVerification
  RebuildPhase->>HermesGateway: Restart before MCP restoration
  HermesGateway-->>RebuildPhase: Return restart state
  RebuildPhase->>MCPRestoration: Restore managed MCP configuration
  MCPRestoration-->>RebuildPhase: Perform acknowledged reload
  RebuildPhase->>GatewayVerification: Verify using restart state
  GatewayVerification-->>RebuildPhase: Return final gateway state

Loading

Possibly related PRs

• NVIDIA/NemoClaw#8472: Both modify Hermes rebuild post-restore gateway restart and verification.
• NVIDIA/NemoClaw#8581: Both modify the Hermes rebuild post-restore test harness and coverage.
• NVIDIA/NemoClaw#8659: Both address Hermes post-rebuild MCP verification through different APIs and test utilities.

Suggested labels: integration: hermes, area: sandbox, bug-fix

Suggested reviewers: ericksoa, cv

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main changes: preserving MCP reload behavior and correcting the Discord handshake.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-hermes-e2e

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-8661.docs.buildwithfern.com/nemoclaw

[github-code-quality]

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit 82869f2 in the codex/fix-hermes-e2e branch remains at 96%, unchanged from commit 794c754 in the main branch.

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/actions/sandbox/rebuild-hermes-post-restore.ts (1)

135-158: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the combined lifecycle wrappers and their tests.

The wrappers have no non-test callers. The rebuild action already uses the split APIs, which preserve MCP restoration between restart and verification.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/rebuild-hermes-post-restore.ts` around lines 135 -
158, Remove the unused ensureHermesGatewayAfterStateRestore and
ensureHermesGatewayAfterStateRestoreForCronGate wrapper functions, along with
their associated tests. Keep restartHermesGatewayAfterStateRestore and the
separate verification APIs unchanged so the rebuild action continues preserving
MCP restoration between restart and verification.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/sandbox/rebuild-hermes-post-restore.ts`:
- Around line 135-158: Remove the unused ensureHermesGatewayAfterStateRestore
and ensureHermesGatewayAfterStateRestoreForCronGate wrapper functions, along
with their associated tests. Keep restartHermesGatewayAfterStateRestore and the
separate verification APIs unchanged so the rebuild action continues preserving
MCP restoration between restart and verification.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a935e575-3746-462a-ab01-5b3a57a20316

📥 Commits

Reviewing files that changed from the base of the PR and between 794c754 and 82869f2.

📒 Files selected for processing (8)

• docs/manage-sandboxes/recover-rebuild-sandboxes.mdx
• src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts
• src/lib/actions/sandbox/rebuild-hermes-post-restore.ts
• src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts
• src/lib/actions/sandbox/rebuild-post-restore-phase.ts
• test/e2e/live/hermes-discord-proxy.ts
• test/e2e/live/hermes-discord.test.ts
• test/e2e/support/hermes-discord-proxy-request.test.ts

[github-actions]

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

• GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
• Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 0 warnings · 0 suggestions

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

• define — acknowledged gateway reload at docs/manage-sandboxes/recover-rebuild-sandboxes.mdx:226: Define this term at first use or replace it with a phrase that states the reported completion contract.
• justified — HTTP absolute-form at test/e2e/support/hermes-discord-proxy-request.test.ts:9: Keep the modifier because it identifies the proxy request form that the regression test requires.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: cloud-inference, cloud-onboard, managed-image-multiarch-startup, managed-image-protected-runtime, security-posture, hermes-discord, onboard-repair, onboard-resume, rebuild-hermes, rebuild-openclaw, state-backup-restore

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.