Conversation
validateChecks only checked that a local check's artifact path was lexically owned by the checking agent (agents/<id>/artifacts/...); it never confirmed the file actually exists. A covered/blocked/candidate unit could pass validation while pointing at local evidence that was never produced. The CLI now resolves each local-check artifact against the ledger's own directory (the documented <output-dir>/coverage-ledger.json convention) and rejects a missing artifact, a non-regular file, or a symlink, and rejects an artifact whose real path escapes the owning agent's artifacts directory. validateDocument/validateChecks keep today's shape-only behavior when called without a base directory, so existing unit tests that check schema rules in isolation are unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #21
validateChecks in skills/security-audit/validate-coverage-ledger.cjs only checked that a
local check's artifact path was lexically owned by the checking agent (isOwnedArtifactPath:
matches agents/<agent_id>/artifacts/... and is a safe relative path). It never confirmed the
file the path names actually exists. A covered, blocked, or candidate coverage unit could
pass validation while its local check evidence was never produced, exactly the repro in the
issue: a syntactically valid artifact path with no file behind it passes with "PASS: 1
coverage units valid".
Added validateArtifactExists, called for every local method check once its path already
passes the existing ownership check. It resolves the artifact against the ledger's own
directory (the /coverage-ledger.json convention already documented in
RECONNAISSANCE.md and VALIDATION-AND-REPORTING.md, same directory the README's
~/security-audit-skill//run- default describes), and rejects when:
artifact must be a regular file")
directory ("local check artifact resolves outside the owning agent's artifacts
directory"). This catches a symlinked ancestor directory smuggling the artifact outside
its owner's tree even when the artifact itself isn't a symlink.
Diagnostic format matches the issue's own suggestion:
$[0].local_checks[0].artifact: local check artifact does not exist.
validateDocument, collectUnitErrors, validateAttempts, and validateChecks all take a new
optional baseDir parameter that only the CLI (run(file), via
path.dirname(path.resolve(file))) supplies. Direct callers of validateDocument that don't
have a real ledger directory, like the existing unit tests exercising isolated schema rules,
keep today's shape only behavior. That's deliberate, not an oversight: hundreds of existing
tests build ledgers in memory without ever writing artifact files to disk, and forcing every
one of them to also manage real fixtures would turn a focused validator correctness fix into
an unrelated test suite rewrite. The CLI path, the only one that runs for real per
RECONNAISSANCE.md's "run after seeding, after every parent update, and before Phase 6", now
always gets the real check.
Tested:
via spawnSync, matching the file's existing CLI test convention: missing artifact,
directory instead of file, symlink artifact, and the accepting case with a real file in
place. Confirmed all three rejecting cases pass against unmodified code (the bug
reproduces, PASS: 1 coverage units valid when it should fail) before the fix, and fail
correctly after.
pre-existing unit test in the file, keeps its current shape only behavior. That's the
compatibility guarantee the design above depends on.
cross check that findings and coverage ledger validator share aligned helpers.