Skip to content

test(e2e): add interactive onboard policy-preset step-ordering test (#6042) - #8618

Open
wakqasahmed wants to merge 6 commits into
NVIDIA:mainfrom
wakqasahmed:test/onboard-policy-preset-sequencing-6042
Open

test(e2e): add interactive onboard policy-preset step-ordering test (#6042)#8618
wakqasahmed wants to merge 6 commits into
NVIDIA:mainfrom
wakqasahmed:test/onboard-policy-preset-sequencing-6042

Conversation

@wakqasahmed

@wakqasahmed wakqasahmed commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

Three independent investigations on #6042 (comments from @yimoj, @dfernandez365-rgb) could not reproduce the reported skip of the interactive onboard wizard's Policy Presets TUI step, and a structural audit confirmed the onboard state machine has no transition path from any earlier state directly to complete — every path passes through policies. What was missing was a checked-in test that actually drives the real interactive wizard through a PTY to prove it (piped stdin does not reproduce this wizard's raw-mode selectors, so the existing non-interactive FSM tests don't cover this).

This PR adds that test. No production onboarding behavior changes — it is test-only, matching the defensible contribution boundary @dfernandez365-rgb scoped in their 2026-07-20 comment on the issue.

Changes

  • test/e2e/live/onboard-interactive-pty.ts — new PTY driver that answers scripted prompts (by trigger substring, firing independently of order) against a real pseudo-terminal, using the suite's single audited async child-process boundary (spawnObservedChild).
  • test/e2e/live/onboard-policy-preset-sequencing.test.ts — drives nemoclaw onboard --fresh through the "Other OpenAI-compatible endpoint" journey (hermetic, no NVIDIA credential needed) against a local fake OpenAI-compatible server, and asserts the ordered step markers [1/8] through [8/8] Policy presets appear strictly in order, with completion only reachable after Policy presets.
  • tools/e2e/check-semantic-phases.mts — registers the new driver's progress-capability callsite in the reviewed allowlist (required for the audited async-boundary check).

Verification

  • npm run test:e2e-phases:check — passes (123 tests across 80 files)
  • npx tsc -p tsconfig.cli.json — clean
  • Manually drove the real onboard CLI through a raw PTY multiple times locally (Docker + OpenShell 0.0.99) with the exact rule set encoded in this test — confirmed the full transcript reaches [8/8] Policy presets (including its two raw-mode selectors: Policy tier, then individual preset toggles) and the wizard completes.

Related Issue

Relates to #6042

Type of Change

  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

(Test-only change; no user-facing behavior, command, or documentation changed.)

Quality Gates

  • Tests added or updated for changed behavior
  • Docs not applicable — justification: test-only change; no command, output, configuration, workflow, schema, or supported behavior changes.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: Test-only change adding live E2E coverage; no command, output, configuration, workflow, schema, or supported behavior changes.

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Targeted behavior tests pass for the current change set — npm run test:e2e-phases:check, npx tsc -p tsconfig.cli.json

Signed-off-by: Waqas Ahmed wakqasahmed@protonmail.com

Summary by CodeRabbit

  • Bug Fixes

    • Improved interactive onboarding reliability by detecting hangs, timeouts, unexpected exits, and missing prompts.
    • Ensured onboarding steps, including policy presets, proceed in the correct order without aborting.
  • Tests

    • Added end-to-end coverage for interactive onboarding workflows.
    • Improved automated prompt handling, transcript capture, timeout enforcement, progress monitoring, and process cleanup for more reliable validation.

@copy-pr-bot

copy-pr-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Changes

Interactive onboarding validation

Layer / File(s) Summary
PTY command driver
test/e2e/live/onboard-interactive-pty.ts, tools/e2e/check-semantic-phases.mts
Adds interactive command contracts and a PTY-backed driver. The driver sends prompt responses, captures transcripts and triggers, reports exit status, handles timeouts, and registers observed-child progress through options.progress.
Policy preset sequencing regression test
test/e2e/live/onboard-policy-preset-sequencing.test.ts
Adds an end-to-end onboarding test with a mock OpenAI-compatible server, PTY prompt responses, ordered step validation, completion assertions, transcript capture, prerequisite handling, and cleanup.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OnboardingE2ETest
  participant driveInteractiveCommand
  participant OnboardingWizard
  participant MockOpenAICompatibleServer
  OnboardingE2ETest->>MockOpenAICompatibleServer: start authenticated mock endpoint
  OnboardingE2ETest->>driveInteractiveCommand: provide command and prompt rules
  driveInteractiveCommand->>OnboardingWizard: launch through PTY
  OnboardingWizard->>MockOpenAICompatibleServer: request inference
  MockOpenAICompatibleServer-->>OnboardingWizard: return mock response
  OnboardingWizard-->>driveInteractiveCommand: emit prompts and step markers
  driveInteractiveCommand-->>OnboardingE2ETest: return transcript and exit state
  OnboardingE2ETest->>OnboardingE2ETest: verify ordered steps and successful completion
Loading

Possibly related PRs

Suggested labels: area: sandbox

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the added end-to-end test for interactive onboarding policy-preset step ordering.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (4)
test/e2e/live/onboard-policy-preset-sequencing.test.ts (2)

104-106: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Use process.execPath instead of the node name.

The command resolves node through PATH in buildAvailabilityProbeEnv(). If the probe environment trims PATH, or if the runner exposes a different Node version, the wizard runs under an unexpected interpreter or fails to start. The sibling fixture test/e2e/fixtures/fake-openai-compatible.ts spawns process.execPath for this reason.

♻️ Proposed fix
     cmd: [
-      "node",
+      process.execPath,
       CLI_ENTRYPOINT,
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/onboard-policy-preset-sequencing.test.ts` around lines 104 -
106, Update the command array in the onboarding policy preset sequencing test to
use process.execPath instead of the literal "node" entry, while preserving
CLI_ENTRYPOINT and the remaining arguments.

159-168: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Assert the fired triggers, and check the timeout first.

The test never uses result.firedTriggers. The issue claims the wizard skips the Policy presets TUI step. A header marker proves only that the header printed. A fired "Policy tier" trigger proves the interactive selector appeared and accepted input.

Also check timedOut and exitCode before the marker assertions. On a timeout the ordered-marker loop at line 150 fails first and hides the real cause.

♻️ Proposed addition
   progress.phase("confirm Policy presets is reached before completion");
+  expect(
+    result.firedTriggers,
+    `Policy tier selector was never presented:\n${result.output}`,
+  ).toContain("Policy tier");
   const policyIndex = result.output.indexOf("[8/8] Policy presets");
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/onboard-policy-preset-sequencing.test.ts` around lines 159 -
168, Update the onboarding sequencing test assertions to check result.timedOut
and result.exitCode before validating output markers, so timeout or process
failures are reported first. Then use result.firedTriggers to assert the "Policy
tier" trigger fired, in addition to retaining the existing Policy presets marker
and abort checks.
test/e2e/live/onboard-interactive-pty.ts (2)

141-166: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Stderr line buffering can drop or split FIRED records.

The handler at line 144 splits each chunk on \n. A chunk boundary can fall inside a FIRED\t<trigger> line. The partial line then fails the regular expression, and the record is lost.

Keep a residual buffer between chunks.

♻️ Proposed fix: buffer partial stderr lines
+    let stderrRest = "";
     child.stderr?.on("data", (chunk: Buffer) => {
-      for (const line of chunk.toString("utf-8").split("\n")) {
+      const lines = (stderrRest + chunk.toString("utf-8")).split("\n");
+      stderrRest = lines.pop() ?? "";
+      for (const line of lines) {
         const fired = line.match(/^FIRED\t(.*)$/);
         if (fired) firedTriggers.push(fired[1]);
       }
     });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/onboard-interactive-pty.ts` around lines 141 - 166, Update the
stderr handling around the child process listeners to retain a residual buffer
across chunks, append each decoded chunk, and process only complete
newline-delimited lines for FIRED records. Preserve any trailing partial line
for the next chunk and process it appropriately when the stream closes.

111-125: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Pass the rules payload outside argv.

The payload contains rule responses. In test/e2e/live/onboard-policy-preset-sequencing.test.ts those responses include an API key (line 128). Command arguments are readable by other local processes through ps and /proc. The value is ephemeral in this test, but the driver is a reusable contract.

Pass the payload through an environment variable or the child's stdin instead.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/onboard-interactive-pty.ts` around lines 111 - 125, Update the
call to spawnObservedChild in the surrounding PTY execution flow so the JSON
rules payload is no longer included in the child process argv. Pass payload
through a child environment variable or stdin, and update PTY_DRIVER_SCRIPT to
read it from that channel while preserving existing command, rule, timeout, and
driver behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/e2e/live/onboard-interactive-pty.ts`:
- Around line 133-136: Update the timeout handling around the detached driver
spawn and timer callback: ensure the Python driver is spawned as a detached
process-group leader, then signal the process group rather than only calling
child.kill so the PTY CLI child is terminated too. Preserve the documented
whole-process-tree cleanup behavior and existing timeout state handling.
- Around line 69-101: Update the PTY driver loop around the child read and wait
handling so EOF or OSError from os.read does not leave a normally exited child
classified as a timeout. Track whether the deadline actually expired, then
perform a final os.waitpid(pid, 0) after the loop breaks from EOF/read failure
and derive exit_code when the child has exited; only send SIGKILL and return 124
when the deadline was genuinely reached.

In `@test/e2e/live/onboard-policy-preset-sequencing.test.ts`:
- Around line 124-129: In the provider-selection flow of the live onboarding
test, add an explicit guard that expects the “Other OpenAI-compatible endpoint”
prompt immediately after selecting provider 4, so reordered options fail fast.
Update the API-key response trigger from “API key:” to “Other OpenAI-compatible
endpoint API key:” and retain the existing base URL and model responses.

---

Nitpick comments:
In `@test/e2e/live/onboard-interactive-pty.ts`:
- Around line 141-166: Update the stderr handling around the child process
listeners to retain a residual buffer across chunks, append each decoded chunk,
and process only complete newline-delimited lines for FIRED records. Preserve
any trailing partial line for the next chunk and process it appropriately when
the stream closes.
- Around line 111-125: Update the call to spawnObservedChild in the surrounding
PTY execution flow so the JSON rules payload is no longer included in the child
process argv. Pass payload through a child environment variable or stdin, and
update PTY_DRIVER_SCRIPT to read it from that channel while preserving existing
command, rule, timeout, and driver behavior.

In `@test/e2e/live/onboard-policy-preset-sequencing.test.ts`:
- Around line 104-106: Update the command array in the onboarding policy preset
sequencing test to use process.execPath instead of the literal "node" entry,
while preserving CLI_ENTRYPOINT and the remaining arguments.
- Around line 159-168: Update the onboarding sequencing test assertions to check
result.timedOut and result.exitCode before validating output markers, so timeout
or process failures are reported first. Then use result.firedTriggers to assert
the "Policy tier" trigger fired, in addition to retaining the existing Policy
presets marker and abort checks.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 34d2bc6d-f833-4c1d-bb0f-53622aff0a15

📥 Commits

Reviewing files that changed from the base of the PR and between a76e297 and 32fd0ea.

📒 Files selected for processing (3)
  • test/e2e/live/onboard-interactive-pty.ts
  • test/e2e/live/onboard-policy-preset-sequencing.test.ts
  • tools/e2e/check-semantic-phases.mts

Comment thread test/e2e/live/onboard-interactive-pty.ts
Comment thread test/e2e/live/onboard-interactive-pty.ts
Comment on lines +124 to +129
// "Other OpenAI-compatible endpoint" — position depends on
// src/lib/onboard/providers.ts's provider list for the openclaw agent.
{ trigger: "Select your inference provider:", response: "4\n" },
{ trigger: "OpenAI-compatible base URL", response: `${fake.baseUrl}\n` },
{ trigger: "API key:", response: `${apiKey}\n` },
{ trigger: "endpoint model", response: `${MODEL}\n` },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm the provider list order for the openclaw agent and the exact prompt strings.
fd -t f 'providers.ts' -p src/lib/onboard --exec cat -n {}
rg -n 'Select your inference provider|OpenAI-compatible base URL|API key:' src --type=ts -C2

Repository: NVIDIA/NemoClaw

Length of output: 32639


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- test outline ---'
ast-grep outline test/e2e/live/onboard-policy-preset-sequencing.test.ts
printf '%s\n' '--- test focused range ---'
cat -n test/e2e/live/onboard-policy-preset-sequencing.test.ts | sed -n '70,155p'
printf '%s\n' '--- provider selection implementation ---'
cat -n src/lib/onboard/provider-selection-prompt.ts | sed -n '1,130p'
printf '%s\n' '--- provider selection callers/options ---'
rg -n 'provider-selection-prompt|REMOTE_PROVIDER_CONFIG|Select your inference provider|Other OpenAI-compatible endpoint' src/lib/onboard test/e2e/live --type ts -C3
printf '%s\n' '--- E2E trigger and transcript APIs ---'
rg -n 'trigger:|transcript|Transcript|class .*Driver|35.?minute|timeout' test/e2e src --glob '*.{ts,js}' -C2

Repository: NVIDIA/NemoClaw

Length of output: 50372


🏁 Script executed:

#!/bin/bash
set -e
log=/tmp/coderabbit-shell-logs/shell-output-Zzob9i
printf '%s\n' '--- saved output markers ---'
rg -n -- '-- test outline|-- test focused range|-- provider selection implementation|-- provider selection callers/options|-- E2E trigger and transcript APIs' "$log" || true
printf '%s\n' '--- test source ---'
cat -n test/e2e/live/onboard-policy-preset-sequencing.test.ts | sed -n '1,170p'
printf '%s\n' '--- provider prompt source ---'
cat -n src/lib/onboard/provider-selection-prompt.ts | sed -n '1,120p'
printf '%s\n' '--- focused provider-selection references ---'
rg -n 'providerSelection|provider-selection-prompt|REMOTE_PROVIDER_CONFIG|Select your inference provider|Other OpenAI-compatible endpoint' src/lib/onboard test/e2e/live --glob '*.{ts,js}' -C2
printf '%s\n' '--- focused E2E support files ---'
fd -t f . test/e2e | rg 'driver|shell|probe|fixture|progress|live' | head -80

Repository: NVIDIA/NemoClaw

Length of output: 38798


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- provider menu ---'
cat -n src/lib/onboard/provider-menu.ts | sed -n '1,180p'
printf '%s\n' '--- menu construction and agent options ---'
rg -n 'buildInferenceProviderMenu|agentProviderOptions|openclaw|ProviderMenuChoice' src/lib/onboard --glob '*.{ts,js}' -C4
printf '%s\n' '--- interactive driver ---'
cat -n test/e2e/live/onboard-interactive-pty.ts | sed -n '1,280p'
printf '%s\n' '--- custom provider prompt path ---'
rg -n 'API key|base URL|endpoint model|COMPATIBLE_API_KEY|custom' src/lib/onboard/inference-providers src/lib/onboard/setup-inference.ts --glob '*.{ts,js}' -C4

Repository: NVIDIA/NemoClaw

Length of output: 50372


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- provider menu ---'
cat -n src/lib/onboard/provider-menu.ts | sed -n '1,130p'
printf '%s\n' '--- interactive driver ---'
wc -l test/e2e/live/onboard-interactive-pty.ts
cat -n test/e2e/live/onboard-interactive-pty.ts | sed -n '1,260p'
printf '%s\n' '--- custom-provider prompt matches ---'
rg -n 'API key|base URL|endpoint model|COMPATIBLE_API_KEY' src/lib/onboard/inference-providers src/lib/onboard/setup-inference.ts -C2
printf '%s\n' '--- menu construction calls ---'
rg -n 'buildInferenceProviderMenu|agentProviderOptions' src/lib/onboard --glob '!*.test.ts' -C3

Repository: NVIDIA/NemoClaw

Length of output: 16674


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- all API-key prompt strings ---'
rg -n -i 'API key|apiKey|api_key' src/lib/onboard --glob '!*.test.ts' -C2
printf '%s\n' '--- setup-nim-selection ---'
cat -n src/lib/onboard/setup-nim-selection.ts | sed -n '1,180p'
printf '%s\n' '--- setup-nim-flow provider selection range ---'
cat -n src/lib/onboard/setup-nim-flow.ts | sed -n '490,660p'
printf '%s\n' '--- agent provider option definitions ---'
rg -n 'getAgentInferenceProviderOptions|agentInferenceProviderOptions|inferenceProviderOptions' src --glob '!*.test.ts' -C3

Repository: NVIDIA/NemoClaw

Length of output: 50372


🏁 Script executed:

#!/bin/bash
set -e
printf '%s\n' '--- credential prompt helpers ---'
rg -n 'ensureNamedCredential|replaceNamedCredential|promptNamedCredential|credential.*prompt|prompt\(.*key|API key:' src/lib test/e2e --glob '*.{ts,js}' -C3 | head -240
printf '%s\n' '--- setup inference prompt calls ---'
rg -n 'ensureNamedCredential|replaceNamedCredential|prompt\(' src/lib/onboard/setup-inference.ts src/lib/onboard/inference-providers src/lib/onboard/credential-env.ts src/lib/onboard --glob '!*.test.ts' -C2 | head -240
printf '%s\n' '--- test driver behavioral probe ---'
node - <<'JS'
const rules = [
  { trigger: "Select your inference provider:", response: "4\n" },
  { trigger: "API key:", response: "custom-key\n" },
  { trigger: "Enable web search", response: "1\n" },
];
let text = "";
const fired = [];
for (const chunk of [
  "Select your inference provider:\n",
  "Other OpenAI-compatible endpoint\n",
  "OpenAI-compatible API key: ",
  "Enable web search\n",
  "Brave Search API key: ",
]) {
  text += chunk;
  for (const [i, rule] of rules.entries()) {
    if (!fired[i] && text.includes(rule.trigger)) {
      fired[i] = true;
      console.log(`fired ${rule.trigger}`);
    }
  }
}
JS

Repository: NVIDIA/NemoClaw

Length of output: 36614


Make provider selection fail fast.

"4\n" selects custom only while BASE_REMOTE_PROVIDER_OPTIONS keeps it at position 4. A reordered menu can select another provider and leave later rules waiting for the 35-minute driver timeout. Add a guard for Other OpenAI-compatible endpoint, and replace "API key:" with Other OpenAI-compatible endpoint API key:. A post-timeout transcript assertion alone will not prevent the delay.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/onboard-policy-preset-sequencing.test.ts` around lines 124 -
129, In the provider-selection flow of the live onboarding test, add an explicit
guard that expects the “Other OpenAI-compatible endpoint” prompt immediately
after selecting provider 4, so reordered options fail fast. Update the API-key
response trigger from “API key:” to “Other OpenAI-compatible endpoint API key:”
and retain the existing base URL and model responses.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · medium confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Failed after a partial review · low confidence · 0 blockers · 3 warnings · 1 suggestion

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

3 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • define — real PTY at test/e2e/live/onboard-interactive-pty.ts:7: Define real PTY at first use as a pseudo-terminal allocated for the child process.
  • justified — mock-provider at test/e2e/live/onboard-policy-preset-sequencing.test.ts:25: Keep mock-provider when identifying the credential-free fake-provider test journey.
  • established — interactive mode at tools/e2e/workflow-boundary.mts:2753: Keep interactive mode for onboarding that requires terminal prompt interaction.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite against this exact revision.

Recommended E2E: cloud-inference, cloud-onboard, security-posture

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

…VIDIA#6042)

Three independent investigations (comments on NVIDIA#6042) could not reproduce
the reported skip of the Policy Presets TUI step, and a structural audit
confirmed the onboard state machine has no transition path from any
earlier state directly to complete -- every path passes through
policies. What was missing was a checked-in test that actually drives
the real interactive wizard through a PTY (piped stdin does not
reproduce this wizard's raw-mode selectors) to prove it.

Add such a test: it answers every interactive prompt in the
compatible-endpoint onboarding journey through a real pseudo-terminal
and asserts the ordered step markers ([1/8] through [8/8] Policy
presets) appear in order, with completion only reachable after Policy
presets. This is test-only; no production onboarding behavior changes.

The new PTY driver is routed through the suite's single audited
async child-process boundary (spawnObservedChild), so its
progress-capability callsite is registered in the reviewed allowlist
in tools/e2e/check-semantic-phases.mts.

Signed-off-by: Waqas Ahmed <wakqasahmed@protonmail.com>
@wakqasahmed
wakqasahmed force-pushed the test/onboard-policy-preset-sequencing-6042 branch from 32fd0ea to 49c18e4 Compare August 8, 2026 11:30
@wakqasahmed

wakqasahmed commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Re: PRA-1 (interactive onboarding regression not wired into a checked-in E2E workflow job).

Confirmed the gap. Investigated adding it as a second step in cloud-onboard (the closest existing job with Docker/expect/OpenShell/CLI already provisioned), but every one of the 63 existing jobs in e2e.yaml has exactly one Vitest test-path invocation, and cloud-onboard is one of the jobs tools/e2e/operations-workflow-boundary.mts's corruption-detection test requires to have exactly one — adding a second invocation (as a new step, or combined into the existing step) masks that test's ability to detect a corrupted invocation in the job.

Doing this properly means a dedicated new job (its own bespoke validator function in tools/e2e/workflow-boundary.mts, a test/e2e/mock-parity.json entry, and a tools/e2e/cli-artifact-workflow-boundary.mts registration — matching the ceremony behind e.g. double-onboard), which is new CI surface rather than a fix to this PR's diff. Leaving this open for maintainer call: happy to build the dedicated job in a follow-up if that's the preferred direction.

Correction to an earlier draft of this comment: this test lives under test/e2e/live/, so it's part of the e2e-live Vitest project (opt-in, gated behind NEMOCLAW_RUN_LIVE_E2E=1), not e2e-support. It does not run in the aggregate checks for ordinary PRs today — only a maintainer-dispatched live E2E run or an explicit local NEMOCLAW_RUN_LIVE_E2E=1 npx vitest run --project e2e-live invocation exercises it, which is exactly the gap PRA-1 is flagging.

wakqasahmed and others added 4 commits August 8, 2026 16:34
…heck

codebase-growth-guardrails flagged the manual Docker prerequisite check as
two added `if` statements in the test body. Replace it with the existing
`docker` fixture's `requireDocker()` (already used by e.g.
sandbox-operations.test.ts), which encapsulates the same
throw-in-CI/skip-locally branching in the fixture layer instead of the
test body. Drops the now-unused `resultText` import.

Signed-off-by: Waqas Ahmed <wakqasahmed@protonmail.com>
…ssages

PRA-2 (PR review advisor): the assertion messages for the abort/timeout/
exit-code checks interpolated the raw PTY transcript, which still
contains the generated mock API key -- ArtifactSink's own redaction only
applies to the separate onboard-transcript.txt write, not to text Vitest
prints inline on a failed assertion. Route the same redactionValues
through the shared redactString() helper (ArtifactSink's own redaction
primitive) before interpolating, and point the messages at the artifact
file the way the step-marker assertion already does.

Signed-off-by: Waqas Ahmed <wakqasahmed@protonmail.com>
@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

@cv cv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking findings:

  1. test/e2e/live/onboard-interactive-pty.ts:69-101 treats PTY EOF or Linux EIO after a normal child exit as a timeout because those branches bypass waitpid. A successful onboarding run can therefore return 124 nondeterministically. Reap the child after EOF/read failure and report timeout only after deadline expiry. Add focused regression evidence for a PTY child that exits 0 after producing output.
  2. test/e2e/live/onboard-interactive-pty.ts:118-136 passes every scripted response, including the generated provider API key, in the Python process arguments. The timeout also kills only the Python driver, leaving its PTY onboarding child and host resources active. Transport responses through a channel that is absent from process arguments, and terminate/reap the complete process group on timeout. Add regression evidence that the secret is absent from the spawned argument list and that a timed-out descendant no longer exists.
  3. test/e2e/live/onboard-policy-preset-sequencing.test.ts:54-164 is an opt-in e2e-live test, but no checked-in workflow selects it. The existing cloud-onboard job also sets NEMOCLAW_NON_INTERACTIVE=1. The claimed interactive regression evidence therefore does not run in ordinary CI or a supported E2E job. Select this test in the smallest suitable checked-in E2E job with interactive mode enabled, and add or update the workflow-boundary regression evidence for that invocation.

Three blocking findings from cv's review:

1. The PTY driver's read loop broke on EOF/EIO without reaping the child
   first, so a clean successful run could be misreported as
   DRIVER_TIMEOUT (exit 124) nondeterministically. Block on waitpid at
   that point and record the real exit code instead.

2. The Node-side timeout only killed the Python driver by pid, leaving its
   pty.fork()'d onboard child (and any sandbox operations it had in
   flight) running past the test. Spawn the driver detached (its own
   process group/session) and kill the whole group on timeout so the
   descendant is reaped too.

3. onboard-policy-preset-sequencing.test.ts had no checked-in E2E workflow
   selection. Added a dedicated onboard-policy-preset-sequencing job
   (modeled on double-onboard, the closest existing job shape) that does
   NOT set NEMOCLAW_NON_INTERACTIVE -- unlike cloud-onboard/double-onboard,
   this test needs real interactive mode. Registered its bespoke
   workflow-boundary validator (mirroring validateDoubleOnboardJob),
   added it to the CLI-artifact consumer list and recomputed the pinned
   contract hash, added it to report-to-pr's needs, and pointed its
   mock-parity entry at the new fast coverage below.

Regression evidence for all three, per review request:
- test/e2e/support/onboard-interactive-pty.test.ts (new): a clean exit
  after output is not misreported as a timeout; a generated secret never
  appears in the spawned process arguments; a timed-out driver and its
  forked PTY child are both gone afterward.
- test/e2e/support/onboard-policy-preset-sequencing-workflow-boundary.test.ts
  (new, its own file rather than growing the already near-budget
  e2e-workflow.test.ts): the new job selects the right test file with
  interactive mode enabled today, and reintroducing
  NEMOCLAW_NON_INTERACTIVE on that job is caught. Registered both cases in
  ci/source-shape-test-budget.json's exception list, matching this
  repo's existing workflow-boundary test entries.

Signed-off-by: Waqas Ahmed <wakqasahmed@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants