Skip to content

fix: validate on-disk existence and integrity of local-check artifacts - #26

Open
anvithsg2004 wants to merge 3 commits into
cloudflare:mainfrom
anvithsg2004:fix/coverage-artifact-validation
Open

anvithsg2004 wants to merge 3 commits into
cloudflare:mainfrom
anvithsg2004:fix/coverage-artifact-validation

Conversation

@anvithsg2004

@anvithsg2004 anvithsg2004 commented Sep 17, 2026

Copy link
Copy Markdown

Fixes #21

Summary

Previously, validate-coverage-ledger.cjs only validated the lexical format and ownership prefix (agents/<agent-id>/artifacts/...) of local_checks[].artifact paths. It did not verify whether referenced artifacts actually existed on disk in the retained output directory.

As a result, units marked as covered, candidate, or blocked could pass validation while pointing to phantom or non-existent local check evidence.

This PR extends validate-coverage-ledger.cjs to enforce strict on-disk validation of all local-check artifacts, verifying their existence, file type, symlink status, and directory containment, while adding a comprehensive regression test suite.


How We Solved It

  1. On-Disk Artifact Validation (validateArtifactFile):

    • Existence Check: Uses fs.lstatSync (without following symlinks) to verify the artifact file exists. If missing (ENOENT / ENOTDIR), it produces:
      $[...].artifact: local check artifact does not exist
    • Symlink Rejection: Rejects symlink artifacts using stat.isSymbolicLink():
      $[...].artifact: local check artifact must not be a symlink
    • Regular File Enforcement: Ensures the artifact is not a directory, FIFO, socket, or device using !stat.isFile():
      $[...].artifact: local check artifact must be a regular file
    • Boundary & Realpath Containment: Resolves fs.realpathSync of the artifact against the real path of the agent's artifacts directory (baseDir/agents/<agent_id>/artifacts/). If directory-level symlinks or path traversal attempt to escape this subtree, it produces:
      $[...].artifact: local check artifact must reside inside agent artifacts directory
  2. Pipeline Option Plumbing & Historical Checks:

    • Extended validateChecks(value, location, errors, options = {}) to trigger filesystem checks whenever options.baseDir is provided and lexical checks pass.
    • Passed options into validateAttempts so that archived history in unit.attempts[].local_checks[] also enforces artifact integrity.
    • Updated collectUnitErrors and validateDocument(ledger, options = {}) to thread options throughout the document traversal.
  3. CLI Directory Resolution & Backward Compatibility:

    • In CLI mode (run()), baseDir automatically defaults to the ledger file's parent directory (path.dirname(path.resolve(file))), which naturally aligns with the <output-dir>/coverage-ledger.json layout used throughout the audit workflow.
    • Added CLI arguments --output-dir <path> (and --base-dir <path>) supporting both space-separated and --output-dir=<path> syntax.
    • Added --schema-only flag to allow purely structural validation when desired.
    • Kept programmatic validateDocument(ledger) calls without baseDir schema-only, preserving backward compatibility for existing in-memory tests.
  4. Comprehensive Test Suite (validate-coverage-ledger.test.cjs):
    Added extensive positive, negative, and edge-case tests:

    • Missing artifacts report exact diagnostic paths ($[0].local_checks[0].artifact: local check artifact does not exist).
    • Valid promoted regular files pass cleanly with 0 errors.
    • Directories passed as artifacts are rejected (must be a regular file).
    • Symlinks and directory-escaping symlinks are rejected.
    • Historical checks in attempts[] are validated on disk.
    • 0-byte empty files and deeply nested files inside the artifacts directory are verified and accepted.
    • Unicode (NFC-normalized) filenames are handled correctly.
    • Multi-unit ledgers and multi-check units accurately report errors with exact index precision ($[N].local_checks[M]).
    • CLI integration tests verify exit codes (1 on missing artifact, 0 on valid artifact, --output-dir custom resolution, and --schema-only).

Verification

All existing and newly added tests pass cleanly:

node --test skills/security-audit/validate-coverage-ledger.test.cjs

@anvithsg2004 anvithsg2004 changed the title Validate local check artifact integrity fix: validate on-disk existence and integrity of local-check artifacts Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Coverage validator accepts nonexistent local-check artifacts

1 participant