Skip to content

fix(verifier): surface self-contradictory finish instead of dropping the vuln - #221

Merged
gadievron merged 1 commit into
masterfrom
fix/famreport2-agree-contradiction
Aug 10, 2026
Merged

fix(verifier): surface self-contradictory finish instead of dropping the vuln#221
gadievron merged 1 commit into
masterfrom
fix/famreport2-agree-contradiction

Conversation

@gadievron

Copy link
Copy Markdown
Collaborator

What

Surface a self-contradictory Stage-2 verifier finish instead of silently dropping the finding it carries.

Why

The verifier's finish tool schema declares agree and correct_finding as independent required fields with no cross-field constraint (utilities/finding_verifier.py:153-199), so a model can return agree=True (claims to agree with the Stage-1 verdict) while correct_finding names a different verdict. _parse_finish_result took agree verbatim, and the two write-back consumers (_verify_one:740, experiment.py:590) only copy correct_finding into result["finding"] on the agree==False branch. So an upgrade contradiction — Stage-1 safe, correct_finding=vulnerable — left result["finding"]="safe", and the reporter's disclosure filter (core/reporter.py:283, which selects on finding in ("vulnerable","bypassable")) dropped the vulnerable finding from the report.

The drop is reachable via the experiment.py --verify flow, which verifies every Stage-1 result (experiment.py:551-552). The production scanner path pre-filters to vulnerable/bypassable before verification (core/verifier.py:101-105), so on that path a downgrade contradiction conservatively keeps the vulnerable finding surfaced.

How

  • utilities/finding_verifier.py::_parse_finish_result: on a contradiction (agree==True and correct_finding != Stage-1), take the more-severe of the two verdicts, set agree=False and incomplete=True, then return that. The reporter renders it as unverified (needs manual review) rather than a droppable agreed.
  • _more_severe helper ranks by the existing FINDING_VERDICT_ORDER (core/verdict_taxonomy.py:32), so a vulnerable reading on either side is kept and a downgrade cannot silence a real Stage-1 verdict.
  • One edit at the shared chokepoint covers both consumers: _verify_one and experiment.py both reach _parse_finish_result via verify_result. Mirrors the file's existing degenerate-path fail-safe (which also returns agree=False + incomplete=True).

Observed: a contradictory finish now increments the needs_review bucket (core/verifier.py:280) and renders stage2_verdict="unverified" (core/reporter.py:390) instead of vanishing.

Tests

tests/test_famreport2_agree_contradiction.py (8 tests): both contradiction directions, both consumers (via a mirror of the write-back branch + the verbatim reporter filter), a non-vuln-vs-non-vuln edge, and 3 regression guards for normal agree/disagree/missing-agree.

# RED on the pre-fix commit (fad4b01), proof test applied:
$ pytest tests/test_famreport2_agree_contradiction.py -q
4 failed, 4 passed          # the 4 contradiction tests fail: VerificationResult(agree=True, incomplete=False)

# GREEN on this commit:
$ pytest tests/test_famreport2_agree_contradiction.py -q
8 passed in 0.03s

# full suite on this branch:
$ pytest tests/ -q
2540 passed, 28 skipped     # 2 pre-existing tests/parsers/zig/ failures are unrelated (fail with this fix reverted)

Compatibility

Behavior change on the production path only for a self-contradictory finish: a finding previously labeled agreed/confirmed despite the model contradicting itself now renders unverified (needs review). It stays surfaced either way. No API, schema, or signature changes. No changes to core/parsers/reachability.

Author notes

  • Q1 — which lines implement the fix? utilities/finding_verifier.py _parse_finish_result: the if agree and correct_finding != original_finding: block (sets correct_finding = _more_severe(...), agree=False, incomplete=True) plus the _more_severe helper and the FINDING_VERDICT_ORDER import.
  • Q2 — one concrete input handled wrong before, right now? finish(agree=True, correct_finding="vulnerable") over a Stage-1 safe finding: before, result["finding"] stayed "safe" and the reporter dropped it; now result["finding"] becomes "vulnerable" with incomplete=True, so it appears as an unverified finding for review.
  • Q3 — likely reviewer pushback + answer? "Why not copy correct_finding on agree==True too?" Because that would override agree with correct_finding on every normal turn; the contradiction path is the only place the two disagree, and there the safe move is to surface for review, not to silently pick a side.

…the vuln

A verifier finish with agree=True but correct_finding diverging from the
Stage-1 verdict (the finish schema declares the two fields independently, no
cross-field constraint) was read as a clean agreement. The write-back consumers
(_verify_one, experiment.py) only propagate correct_finding on the disagree
branch, so an upgrade contradiction (stage1=safe, correct_finding=vulnerable)
left result["finding"]="safe" and the vuln was dropped by the reporter's
disclosure filter (core/reporter.py:283 selects finding in vulnerable/bypassable).

Detect the contradiction at the shared chokepoint _parse_finish_result: take the
more-severe verdict (FINDING_VERDICT_ORDER), set agree=False and incomplete=True,
so it renders as "unverified" (needs manual review) rather than a droppable
"agreed". One edit covers both consumers, which both reach _parse_finish_result
via verify_result. Mirrors the file's existing R4-7 degenerate-path fail-safe.

The false-negative is reachable via the experiment.py --verify flow (verifies all
Stage-1 results); the production scanner (core/verifier.py) pre-filters to
vulnerable/bypassable, so on that path a downgrade contradiction conservatively
keeps the vulnerable finding surfaced.

Tests: tests/test_famreport2_agree_contradiction.py (8; RED 4-fail pre-fix,
covering both contradiction directions + regression guards).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gadievron
gadievron merged commit 3718696 into master Aug 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant