fix: validate on-disk existence and integrity of local-check artifacts - #26
Open
anvithsg2004 wants to merge 3 commits into
Open
anvithsg2004 wants to merge 3 commits into
anvithsg2004 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #21
Summary
Previously,
validate-coverage-ledger.cjsonly validated the lexical format and ownership prefix (agents/<agent-id>/artifacts/...) oflocal_checks[].artifactpaths. It did not verify whether referenced artifacts actually existed on disk in the retained output directory.As a result, units marked as
covered,candidate, orblockedcould pass validation while pointing to phantom or non-existent local check evidence.This PR extends
validate-coverage-ledger.cjsto enforce strict on-disk validation of all local-check artifacts, verifying their existence, file type, symlink status, and directory containment, while adding a comprehensive regression test suite.How We Solved It
On-Disk Artifact Validation (
validateArtifactFile):fs.lstatSync(without following symlinks) to verify the artifact file exists. If missing (ENOENT/ENOTDIR), it produces:$[...].artifact: local check artifact does not existstat.isSymbolicLink():$[...].artifact: local check artifact must not be a symlink!stat.isFile():$[...].artifact: local check artifact must be a regular filefs.realpathSyncof the artifact against the real path of the agent's artifacts directory (baseDir/agents/<agent_id>/artifacts/). If directory-level symlinks or path traversal attempt to escape this subtree, it produces:$[...].artifact: local check artifact must reside inside agent artifacts directoryPipeline Option Plumbing & Historical Checks:
validateChecks(value, location, errors, options = {})to trigger filesystem checks wheneveroptions.baseDiris provided and lexical checks pass.optionsintovalidateAttemptsso that archived history inunit.attempts[].local_checks[]also enforces artifact integrity.collectUnitErrorsandvalidateDocument(ledger, options = {})to thread options throughout the document traversal.CLI Directory Resolution & Backward Compatibility:
run()),baseDirautomatically defaults to the ledger file's parent directory (path.dirname(path.resolve(file))), which naturally aligns with the<output-dir>/coverage-ledger.jsonlayout used throughout the audit workflow.--output-dir <path>(and--base-dir <path>) supporting both space-separated and--output-dir=<path>syntax.--schema-onlyflag to allow purely structural validation when desired.validateDocument(ledger)calls withoutbaseDirschema-only, preserving backward compatibility for existing in-memory tests.Comprehensive Test Suite (
validate-coverage-ledger.test.cjs):Added extensive positive, negative, and edge-case tests:
$[0].local_checks[0].artifact: local check artifact does not exist).must be a regular file).attempts[]are validated on disk.$[N].local_checks[M]).1on missing artifact,0on valid artifact,--output-dircustom resolution, and--schema-only).Verification
All existing and newly added tests pass cleanly: