Skip to content

test: isolate and clean temporary test roots - #1099

Open
hxy91819 wants to merge 2 commits into
mainfrom
codex/disk-full-incident
Open

test: isolate and clean temporary test roots#1099
hxy91819 wants to merge 2 commits into
mainfrom
codex/disk-full-incident

Conversation

@hxy91819

@hxy91819 hxy91819 commented Aug 10, 2026

Copy link
Copy Markdown
Member

Summary

Repeated test runs can leave fixture directories behind in the shared system temporary directory. Some fixtures intentionally contain non-regular files such as FIFOs, which can also cause content-inspecting host cleanup tools to block indefinitely.

This change keeps the FIFO safety coverage while tightening the test lifecycle:

  • track and remove validation fixtures after each test
  • run standard test commands inside a unique per-run temporary root
  • clean the run root after success, assertion failure, and termination signals
  • reap abandoned run roots after a TTL while protecting active concurrent runs with a heartbeat
  • remove roots recursively without opening fixture contents

The tracker follows the established OpenClaw auto-cleanup temp-directory pattern, adapted locally so ClawSweeper does not depend on another repository's test helpers.

Review follow-up

Validated at head a602f2ba67.

  • P2 fixed at the supported test-entry boundary: target-validation is now a canonical run-node-tests.mjs target. Validation fixtures created through the supported narrow-test command inherit the run root, so a SIGKILL, crash, or reboot leaves a clawsweeper-test-run-* root that the existing heartbeat/TTL reaper recognizes. The raw node --test ... target-validation.test.ts command is no longer used as the recommended evidence path.
  • P3 fixed: the Windows CI job no longer passes test/codex-process.test.ts twice; the named target remains the single source of file selection.
  • A new heartbeat-specific unit test was not added because this follow-up does not change heartbeat semantics; the existing lifecycle suite continues to cover stale-root reaping, assertion failure, SIGTERM, and successful cleanup.

Local black-box validation

Linux, Node v24.15.0. Corepack resolved the repository pin to pnpm 11.10.0.

Runner and lifecycle behavior

node --test test/run-node-tests.test.ts test/test-temp-root-runner.test.ts

Result: 10 passed, 0 failed. This covers named-target selection plus successful FIFO cleanup, assertion-failure cleanup, stale-root TTL cleanup, and SIGTERM cleanup through the real CLI runner.

Incident-equivalent FIFO fixture through the canonical target

node scripts/run-node-tests.mjs target-validation --test-concurrency=1 -- \
  --test-name-pattern "workspace preflight rejects non-regular package metadata without blocking"

Result: 1 passed, 0 failed; FIFO behavior completed in 19.75 ms. Matching counts were unchanged:

clawsweeper-test-run-*   0 -> 0
clawsweeper-validation-* 0 -> 0

Repeated full target validation through the canonical target

Executed twice:

node scripts/run-node-tests.mjs target-validation --test-concurrency=1 -- \
  --test-reporter=tap

Results:

  • run 1: 198 passed, 3 skipped, 0 failed; 133.8 s test duration / 134 s wall clock
  • run 2: 198 passed, 3 skipped, 0 failed; 131.6 s test duration / 131 s wall clock
  • clawsweeper-test-run-*: 0 -> 0 -> 0
  • clawsweeper-validation-*: 0 -> 0 -> 0
  • no run root or validation fixture remained after either run

Repository checks

  • corepack pnpm run check: static checks, formatting, builds, lint, and changed coverage passed; the final full-coverage run failed only in three test/review-blob-hydration.test.ts partial-clone hydration cases, which returned hydrated: false / blobs: 0 in this environment. The same failures reproduce when that file is run directly outside the new runner; this PR does not modify that test or hydration code. The current-head hosted pnpm check passed: https://github.com/openclaw/clawsweeper/actions/runs/31464912192/job/93695706262.
  • focused autoreview before commit: clean; TruffleHog clean
  • branch autoreview against origin/main after commit: clean; TruffleHog clean

Scope boundary

This change affects only test fixture and test-runner lifecycle. It does not change OpenClaw Bay state, queues, sweep workflows, publication behavior, or dashboard contracts. It does not downgrade pnpm and does not delete host-level pnpm, npm, or Docker caches.

@hxy91819
hxy91819 requested a review from a team as a code owner August 10, 2026 15:27
@clawsweeper

clawsweeper Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 10, 2026
@clawsweeper

clawsweeper Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed August 12, 2026, 7:32 PM ET / 23:32 UTC.

ClawSweeper review

What this changes

This PR runs named Node test targets inside unique temporary roots, cleans fixtures and stale roots safely, and routes selected package and Windows CI commands through that shared runner.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open for maintainer review. This member-authored PR is not eligible for automated closure; current main does not yet contain its temporary-root lifecycle, and no discrete correctness defect was found in the current branch diff.

Priority: P3
Reviewed head: a602f2ba67f9db0b25c604324cacd307a1075ed8
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The implementation is focused, covered by lifecycle tests, and supported by concrete current-head runtime evidence, with only the shared-runner compatibility decision remaining.
Proof confidence 🌊 off-meta tidepool Not applicable: The contributor is a repository member, so external-contributor proof gating does not apply; the PR body nevertheless records current-head real CLI lifecycle runs and hosted CI evidence.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: The contributor is a repository member, so external-contributor proof gating does not apply; the PR body nevertheless records current-head real CLI lifecycle runs and hosted CI evidence.
Evidence reviewed 5 items Current main lacks the requested lifecycle: Current main launches the child test process with only cwd and inherited stdio; it does not create an isolated temporary root or cleanup lifecycle.
Branch implementation: The PR creates a per-run root, forwards TMPDIR/TEMP/TMP to the child, heartbeats active roots, and removes the root in finally after child completion or forwarded termination.
Focused lifecycle coverage: The added black-box tests exercise FIFO cleanup, assertion failure, stale-root reaping, and SIGTERM cleanup through the real runner process.
Findings None None.
Security None None.

How this fits together

ClawSweeper’s Node test runner expands named test targets and starts Node’s test process for local commands and CI. The change supplies that process with an isolated temporary root and removes it after completion, failure, or termination.

flowchart LR
  A[Package scripts and CI] --> B[Named test target]
  B --> C[Shared test runner]
  C --> D[Per-run temporary root]
  D --> E[Node test process]
  E --> F[Cleanup and stale-root reaper]
Loading

Decision needed

Question Recommendation
Should a maintainer approve this member-authored change to the temporary-directory contract for all named Node test targets? Approve the scoped runner lifecycle: Accept the isolated temporary-root contract, relying on the current-head lifecycle proof and clean hosted Windows and check runs.

Why: The patch changes shared test and CI execution behavior; repository policy requires explicit maintainer judgment rather than automated closure for member-authored work.

Before merge

  • Resolve merge risk (P1) - Because the shared runner now sets the temporary-directory environment for named targets, an unrecognized test dependency on host-level temporary storage could surface after merge; the focused lifecycle proof and successful Windows check reduce this risk.
  • Complete next step (P2) - A maintainer must decide whether to adopt the shared temporary-directory contract; no narrow repair-lane task is indicated.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Code and test delta production +68 net lines; tests +163 net lines Most growth is focused lifecycle coverage around one shared test-runner behavior.
Execution surfaces 1 shared runner, 1 package script, 1 Windows CI command changed These are the entry points that determine post-merge test execution behavior.

Merge-risk options

Maintainer options:

  1. Accept the runner contract (recommended)
    Merge after maintainer acceptance that named test targets should use an isolated temporary root, supported by the submitted lifecycle and Windows CI evidence.
  2. Add targeted compatibility proof
    Request a focused real-run check for any named target known to rely on a host-level temporary path before merging.

Technical review

Best possible solution:

Land the isolated runner lifecycle only after a maintainer accepts the shared-test-environment tradeoff and confirms the current-head CI and submitted lifecycle evidence remain applicable.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a user bug report; the PR body supplies a high-confidence real-run lifecycle verification path for the test harness.

Is this the best way to solve the issue?

Yes. Centralizing temporary-root isolation in the existing cross-platform named-target runner avoids duplicating cleanup behavior across package scripts and CI.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9a257905e50b.

Labels

Label justifications:

  • P3: This is test-harness cleanup and reliability work without a reported production-user regression.
  • merge-risk: 🚨 automation: The patch changes the shared runner and CI command paths used to execute repository tests.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The contributor is a repository member, so external-contributor proof gating does not apply; the PR body nevertheless records current-head real CLI lifecycle runs and hosted CI evidence.

Evidence

What I checked:

  • Current main lacks the requested lifecycle: Current main launches the child test process with only cwd and inherited stdio; it does not create an isolated temporary root or cleanup lifecycle. (scripts/run-node-tests.mjs:145, 9a257905e50b)
  • Branch implementation: The PR creates a per-run root, forwards TMPDIR/TEMP/TMP to the child, heartbeats active roots, and removes the root in finally after child completion or forwarded termination. (scripts/run-node-tests.mjs:157, a602f2ba67f9)
  • Focused lifecycle coverage: The added black-box tests exercise FIFO cleanup, assertion failure, stale-root reaping, and SIGTERM cleanup through the real runner process. (test/test-temp-root-runner.test.ts:11, a602f2ba67f9)
  • Branch provenance: The current head adds the canonical target-validation runner entry after the initial temporary-root implementation. (scripts/run-node-tests.mjs:27, a602f2ba67f9)
  • Main/release status: The PR head is not contained by the checked-out main branch or any local release tag, so the change is not already implemented or shipped. (a602f2ba67f9)

Likely related people:

  • steipete: The reachable current-main history attributes the existing shared runner path to a Peter Steinberger commit; the local partial history makes deeper ownership attribution uncertain. (role: recent area contributor; confidence: low; commits: 621b297eadd5; files: scripts/run-node-tests.mjs)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (7 earlier review cycles)
  • reviewed 2026-08-10T15:35:28.822Z sha 4b99634 :: found issues before merge. :: [P1] Extend the established runner instead of adding a second one | [P2] Re-raise termination after cleaning the run root
  • reviewed 2026-08-10T15:50:37.034Z sha 1aa53a6 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-11T00:28:19.431Z sha 1aa53a6 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-11T06:29:52.041Z sha a602f2b :: found issues before merge. :: [P2] Refresh proof for the latest runner invocation
  • reviewed 2026-08-11T06:36:47.348Z sha a602f2b :: needs maintainer review before merge. :: none
  • reviewed 2026-08-11T12:11:07.285Z sha a602f2b :: needs maintainer review before merge. :: none
  • reviewed 2026-08-12T06:24:31.715Z sha a602f2b :: needs maintainer review before merge. :: none

@hxy91819
hxy91819 force-pushed the codex/disk-full-incident branch from 4b99634 to 1aa53a6 Compare August 10, 2026 15:46
@hxy91819
hxy91819 changed the base branch from feature/automerge-stable-red-proof to main August 10, 2026 15:46
@clawsweeper clawsweeper Bot added rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 10, 2026
@hxy91819

Copy link
Copy Markdown
Member Author

Local black-box validation completed against commit 1aa53a6a09.

Environment:

  • Linux, Node v24.15.0
  • Corepack resolved the repository pin to pnpm 11.10.0

Evidence:

  1. Lifecycle and runner behavior

    Command:

    node --test test/run-node-tests.test.ts test/test-temp-root-runner.test.ts
    

    Result: 10 passed, 0 failed. This exercises successful FIFO cleanup, assertion-failure cleanup, stale-root TTL cleanup, and SIGTERM cleanup through the real CLI runner. Matching run-root and validation-directory counts were unchanged before and after.

  2. Incident-equivalent FIFO fixture

    Command:

    node --test --test-name-pattern "workspace preflight rejects non-regular package metadata without blocking" test/repair/target-validation.test.ts
    

    Result: 1 passed, 0 failed. The FIFO preflight itself completed in 28.8 ms, below its 1-second behavior bound, and left no matching validation directory behind.

  3. Repeated full target-validation runs

    Command, executed twice:

    node --test --test-reporter=tap test/repair/target-validation.test.ts
    

    Results:

    • run 1: 198 passed, 3 skipped, 0 failed; 132.9 s
    • run 2: 198 passed, 3 skipped, 0 failed; 130.6 s
    • the total count of matching top-level ClawSweeper temporary directories was identical before and after each run
    • no clawsweeper-validation-* directory remained after either run

No pre-existing historical temporary directories were deleted during this validation.

The PR CI pnpm check also passed: https://github.com/openclaw/clawsweeper/actions/runs/31405434604/job/93510465585

steipete added a commit that referenced this pull request Aug 10, 2026
…ck finalization (#1109)

Three repair lanes rolled up PR status checks independently, and the
finalizer's copy had a live bug: it processed every check entry instead of
deduplicating to the newest run per check identity, so an older failed run of
a re-run check remained a blocker after the newer run succeeded. The
finalizer's classifications feed the published repair queue via
repair-publish-results, making stale failures veto finalization.

All three lanes now share src/repair/status-check-rollup.ts (latest-run
selection, ignored-name parsing, casing and timestamp precedence implemented
once); lane policy stays in the callers.

Proof (docs/proof/status-check-rollup): a red-then-green regression test, and
a real-payload production-boundary trace -- the identical real 26-entry check
payload from PR #1099 classified by both implementations: pre-change code
reports two stale CANCELLED blockers, this head reports zero, and the real
report-only finalizer confirms needs_checks=0. Verified in a Docker-backed
Crabbox local-container with committed provenance.

Automation-risk disposition: the newest run of a check identity is its
current state (the rule the router and post-flight lanes already applied);
blast radius is finalizer classifications in the published repair queue;
rollback is a revert restoring per-entry processing.
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Aug 11, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 11, 2026
@clawsweeper clawsweeper Bot removed the proof: sufficient Contributor real behavior proof is sufficient. label Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P3 Low-risk cleanup, docs, polish, ergonomics, or speculative feature. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant