Skip to content

test(state): remove the wall-clock race from sync legacy-marker recovery - #8948

Merged
prekshivyas merged 4 commits into
NVIDIA:mainfrom
Dreamstick9:test/deterministic-sync-lock-recovery
Aug 13, 2026
Merged

test(state): remove the wall-clock race from sync legacy-marker recovery#8948
prekshivyas merged 4 commits into
NVIDIA:mainfrom
Dreamstick9:test/deterministic-sync-lock-recovery

Conversation

@Dreamstick9

@Dreamstick9 Dreamstick9 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

The CLI test does not strand synchronous recovery behind an expired legacy short-token marker races a real 1-second clock: options() sets timeoutMs: 1000 and the synchronous acquisition path defaults monotonicNow to performance.now(), so legacy-marker recovery doing real filesystem work can exceed the budget on a loaded CI runner. That produced the shard-11 failure Timed out waiting for sandbox mutation lock for 'alpha' on #8900 after its transient-failure retry was exhausted. This change drives the deadline from a stepping clock, so recovery success no longer depends on wall-clock load while a genuine strand regression still exhausts the budget after 1,000 clock reads and fails with the real timeout error.

Changes

  • src/lib/state/mcp-lifecycle-lock-acquisition.test.ts: inject a stepping monotonicNow into the synchronous legacy short-token recovery case, with a comment recording the CI evidence. The asynchronous sibling pins its clock to 0; the synchronous case steps instead because its sleep is Atomics.wait, and a fully pinned clock would let a future strand regression block the vitest worker where a stepping clock keeps that failure bounded at 1,000 clock reads with the real timeout error.
  • test/mcp-lifecycle-lock.test.ts: apply the same stepping-clock idiom (already used elsewhere in this file) to preserves a replacement main lock published during stale recovery, which raced a real 50 ms budget the same way and failed on this PR's own CI shard 4 with observed-stale-token when the deadline expired before stale recovery reached its first rename.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification: this change edits the test itself to remove a load-dependent race; the case still asserts the same recovery behavior, now deterministically.
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: test-only change; no user-facing surface.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification:
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Documentation Writer Review

  • Documentation writer subagent reviewed the completed changes
  • Result: no-docs-needed
  • Evidence: test-only change; no documentation page references this test or the acquisition timeout budget.
  • Agent: Claude Code

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run validate:pr passed after refreshing origin/main when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npx vitest run --project cli src/lib/state/mcp-lifecycle-lock-acquisition.test.ts → 28 passed, three consecutive runs; npx vitest run --project integration test/mcp-lifecycle-lock.test.ts → 44 passed, five consecutive runs plus one run on the merged tree
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: not applicable; one-case test edit with no runtime or repo-wide validation change.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Kushagar Garg dreamstick909@gmail.com

The synchronous legacy short-token recovery case raced the real
1-second acquisition budget: options() sets timeoutMs to 1000 and the
sync path defaults to performance.now(), so marker recovery on a
loaded CI runner can exceed the deadline and fail with "Timed out
waiting for sandbox mutation lock for 'alpha'" (PR NVIDIA#8900, CLI shard
11).

Drive the deadline from a stepping clock instead. Recovery success no
longer depends on wall-clock load, and a genuine strand regression
still exhausts the budget after 1,000 clock reads, so it fails with
the real timeout error instead of blocking the synchronous worker the
way a fully pinned clock would.

Signed-off-by: Kushagar Garg <dreamstick909@gmail.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c7b39320-d426-48f9-8354-8799f64a9c3b

📥 Commits

Reviewing files that changed from the base of the PR and between 4082498 and 66b73df.

📒 Files selected for processing (1)
  • test/mcp-lifecycle-lock.test.ts

📝 Walkthrough

Walkthrough

Lifecycle lock recovery tests now use deterministic, incrementing monotonic clocks. The clocks allow filesystem recovery work and bound simulated timeout progression.

Changes

Lifecycle lock recovery

Layer / File(s) Summary
Deterministic recovery timing
src/lib/state/mcp-lifecycle-lock-acquisition.test.ts, test/mcp-lifecycle-lock.test.ts
Both tests pass incrementing monotonicNow callbacks to lifecycle lock helpers instead of relying on real elapsed time. The tests retain their recovery scenarios and timeout bounds.

Estimated code review effort: 2 (Simple) | ~5 minutes

Mergeability Score: ⚪ Minimal · up to 66b73

This change makes a flaky recovery test deterministic without changing product behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: aasthajh, afourniernv, ahunnargikar-nvidia

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: removing wall-clock races from synchronous legacy-marker recovery tests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — No blocking findings reported

Advisor assessment: No blocking advisor findings reported
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized terminology decisions differ; normalized E2E selections match; severity counts match.
1 terminology difference from the second opinion

Advisory only. These are normalized differences from the primary terminology receipt.

  • pinned clock at src/lib/state/mcp-lifecycle-lock-acquisition.test.ts:146: selected only by the second-opinion lane as justified.

Second-opinion terminology and E2E selections are advisory. Live E2E does not run automatically for pull requests.

2 semantic terminology decisions

Terminology decisions are advisory. They affect the assessment only when a separate finding identifies concrete semantic impact.

  • justified — stepping clock at src/lib/state/mcp-lifecycle-lock-acquisition.test.ts:143: Keep “stepping clock” for this advancing deterministic clock.
  • established — strand at src/lib/state/mcp-lifecycle-lock-acquisition.test.ts:146: Keep “strand”; repository usage already applies it to operations and resources that cannot complete.

E2E guidance

Advisory only. A maintainer can dispatch the default E2E suite for the commit under review.

Recommended E2E: None

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

cv and others added 2 commits August 12, 2026 23:08
…nt observation

CLI shard 4 on this PR's own run failed "preserves a replacement
main lock published during stale recovery": the case runs acquisition
against a real 50 ms budget, and on a loaded runner the deadline can
expire before stale recovery reaches its first rename, so the injected
replacement is never published and the final assertion observes the
planted stale token.

Drive the deadline from the file's existing stepping-clock idiom so
the recovery sequence is load-independent while the expected timeout
still fires once the budget's clock reads are exhausted.

Signed-off-by: Kushagar Garg <dreamstick909@gmail.com>
@prekshivyas
prekshivyas enabled auto-merge (squash) August 13, 2026 07:50
@prekshivyas
prekshivyas merged commit 735e88e into NVIDIA:main Aug 13, 2026
38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants