Skip to content

Add remaining open eval cases from the 25-case proposal#28

Merged
justsml merged 7 commits into
mainfrom
claude/next-eval-cases-s1f4b1
Jul 16, 2026
Merged

Add remaining open eval cases from the 25-case proposal#28
justsml merged 7 commits into
mainfrom
claude/next-eval-cases-s1f4b1

Conversation

@justsml

@justsml justsml commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Description

This PR started as a proposal document (evals/PROPOSED_NEXT_25_CASES.md) turning TODO.md's eval-dataset-audit gap list into 25 concrete case proposals. While drafting the implementation, it turned out commit bb64abc had already landed ~20 of those 25 rows directly on main. This PR now:

  1. Reconciles the docsevals/PROPOSED_NEXT_25_CASES.md and TODO.md are updated to mark each of the 25 items DONE / DONE (merged) / OPEN against what's actually in datasets.ts today.
  2. Implements the ~5 genuine remaining gaps:
    • oauth-redirect-symptom / oauth-sso-redirect-validation and business-logic-symptom / business-logic-checkout-abuse — the two vulnerability classes still missing from the anonymous attack-vector and Juice Shop lanes (evals/attack-vectors.ts, src/mastra/evals/datasets.ts, scripts/live-evals/juice-shop-eval-config.ts).
    • Two patch-diff-blind scenarios and one remediation report-writing scenario, covering the north-star loop's remediation/report legs (new src/mastra/evals/patch-diff-scenarios.ts and src/mastra/evals/report-writing-scenarios.ts, registered in datasets.ts, wired into canonical-schema.ts's artifact/mode/environment mapping). These are prompt-only rows — there is no live repo-checkout runner for patch-diff-blind yet, and the dataset description says so.
    • A benchmarks/exploit-hunter/second-webapp pack scaffold for target diversification (item [Tier 3] Create logo and brand identity #18 in the audit). It intentionally fails closed: the obvious second-target image bakes its own well-known branding into server-rendered HTML with no runtime theming hook, so serving it as-is under an anonymous target-disclosure claim would leak the benchmark's identity to the model under evaluation — the exact contamination failure mode the Eval Honesty Rules exist to prevent. init_script.sh/start_docker.sh exit non-zero with an explanation; the README documents the follow-up work (a Dockerfile that neutralizes the branding) needed before it's usable.
  3. Updates dependent tests — hardcoded dataset-name and scenario-id lists in tests/security-chat/eval-datasets.test.ts and tests/security-chat/juice-shop-eval-scripts.test.ts, plus the CI regression-gate baseline in evals/ci-regression-baseline.json (13→15 datasets, 123→130 canonical rows) and its test.
  4. Moves the webapp-vulnerability-count/webapp-finding-quality scorers to an LLM judge. These two scorers previously counted vulnerability classes against a fixed 8-keyword regex taxonomy that never kept pace with the dataset's growth to 20+ classes — a correct SSTI or business-logic finding scored as if the response found nothing. They're now backed by an LLM judge (src/mastra/evals/webapp-scorers.ts) that identifies an open-ended list of distinct vulnerability classes, marks each evidence-backed or not, and scores diversity/evidence-grounding/context-adherence/drift-avoidance. Both scorers share one cached judge call per response to stay mutually consistent and avoid double billing. Same scorer ids/registration as before, so no other wiring changed; the deterministic helpers they used to call directly (countVulnerabilityFindings, assessWebappQuality, scoreWebappResponse) are untouched since they're also used for synchronous report fields elsewhere and are covered by existing tests.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Documentation update

How Has This Been Tested?

  • pnpm typecheck passes
  • pnpm build passes
  • pnpm test passes (full tests/security-chat suite: 178 files / 1081 tests)
  • pnpm eval:ops:audit passes (0 errors; only pre-existing cybench raw-pack warnings)
  • pnpm eval:publish:audit passes (0 errors; warnings are the expected "unreviewed row" / one pre-existing cybench duplicate)
  • Manual testing steps: verified the new judge-backed scorer module imports without side effects (no network/API-key requirement at import time — the OpenRouter client and judge call are only constructed lazily when a score is actually requested). Could not run a live judge call in this environment (no OPENROUTER_API_KEY configured); the implementation mirrors the already-in-production plan-actionability-llm-judge scorer's model-selection pattern.

Checklist

  • My code follows the style guidelines of this project (see AGENTS.md) — matched existing scenario/dataset authoring conventions exactly (symptom-described prompts, canonical schema fields, benchmark pack file shape, LLM-judge scorer pattern from plan-creation-scorers.ts)
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation (TODO.md, PROPOSED_NEXT_25_CASES.md, new pack README)
  • My changes generate no new warnings (beyond the pre-existing ones noted above)
  • New and existing unit tests pass locally with my changes

Security Considerations

The new second-webapp benchmark pack is designed to fail closed rather than risk starting a target that leaks the benchmark's identity to the model under evaluation (a contamination/scope-realism concern, not a traditional security vulnerability). No product runtime code, approval gates, or sandboxing paths were touched. The new judge-based scorer only reads already-generated eval output text and calls OpenRouter with a judge model; it does not execute or influence anything in the runtime security-research agent.

claude added 3 commits July 14, 2026 04:23
Turns the TODO.md eval-dataset-audit gap list into 25 concrete,
authorable rows across missing vulnerability classes, agent-behavior
safety gaps, north-star patch/replay/report coverage, and a
second-target diversification row, with a suggested authoring order.
This lock file carries a per-session PID/timestamp and isn't project
source; nothing to persist here.
Most of the originally-proposed 25 next eval cases already shipped in
bb64abc before this pass started; reconcile TODO.md and
PROPOSED_NEXT_25_CASES.md to reflect that and scope down to the ~5
genuine gaps that were still open:

- OAuth/SSO redirect-validation and business-logic checkout-abuse
  scenarios, added to both the anonymous attack-vector lane and the
  Juice Shop lane (evals/attack-vectors.ts, datasets.ts,
  juice-shop-eval-config.ts).
- Two patch-diff-blind scenarios and one remediation report-writing
  scenario, covering the north-star loop's remediation/report legs
  (new patch-diff-scenarios.ts and report-writing-scenarios.ts,
  registered in datasets.ts and wired into canonical-schema.ts's
  artifact/mode/environment mapping). Patch-diff-blind rows are
  prompt-only for now; there is no live repo-checkout runner yet.
- A benchmarks/exploit-hunter/second-webapp pack scaffold for target
  diversification, which intentionally fails closed: the obvious
  second-target image bakes its own well-known branding into
  server-rendered HTML with no runtime theming hook, so running it
  as-is under an anonymous target-disclosure claim would leak the
  benchmark's identity to the model under evaluation. The README
  documents the follow-up work needed before it can start a real
  container.

Updated the hardcoded dataset-name/scenario-id lists and the CI
regression-gate baseline (13->15 datasets, 123->130 canonical rows) in
their corresponding tests to match.
@justsml justsml changed the title Propose 25 next eval cases from the dataset audit gap list Add remaining open eval cases from the 25-case proposal Jul 14, 2026
claude added 4 commits July 15, 2026 20:08
The two scorers were flagged as stale: countVulnerabilityFindings kept
a fixed 8-keyword regex taxonomy (sqli, xss, idor, ...) that silently
scored 0 for every vulnerability class added since (SSTI, CSRF, JWT,
GraphQL, business-logic, OAuth, ...), rewarding keyword presence over
genuine evidence-backed discovery.

Replace webapp-vulnerability-count and webapp-finding-quality with an
LLM-judge implementation (same ids/names, so all existing registration
and dataset scorerIds wiring is untouched): the judge identifies an
open-ended list of distinct vulnerability classes described in the
response, marks each evidence_backed or not, and scores
diversity/evidence-grounding/context-adherence/drift-avoidance
dimensions. Both scorers share one cached judge call per response
(module-level Map keyed by judge model + response hash) so they stay
mutually consistent and avoid double-billing a second LLM call.

The deterministic countVulnerabilityFindings/assessWebappQuality/
scoreWebappResponse helpers are left untouched, since they're also used
directly (not just via these two scorers) for synchronous report
fields in scripts/live-evals/webapp-model-eval.ts and are covered by
existing regression tests. The other five webapp scorers (target
observation, evidence discipline, system model, investigation angles,
safety gating) are unaffected.
The first pass shipped the judge scorers with zero tests (only
verified the module imports without throwing) and no request timeout
or clear failure message. Fix the concrete gaps:

- Add a request timeout (90s) to the generateObject call so a stalled
  judge call can't hang an entire eval run.
- Fail with a clear, actionable error when OPENROUTER_API_KEY is
  missing instead of an opaque network/auth failure surfacing deep in
  the OpenRouter client.
- Bump the judge's output token budget (900 -> 1400) for headroom: the
  schema's findings array plus four dimension reasons can run larger
  than the plan-judge schema this pattern was based on.
- Add tests/security-chat/webapp-judge-scorer.test.ts mocking
  generateObject to actually exercise the scoring logic: counts
  distinct findings and flags unsupported ones in the reason,
  normalizes the judge's 0-5 score to 0-1, scores a refusal/drift
  response at zero, verifies the two scorers share one cached judge
  call for the same response (no double billing), and verifies a
  missing API key throws with an actionable message before the judge
  is ever called.

Remaining known gaps, not fixed here: no judge/human calibration for
this specific judge (the repo's existing judge-model-calibration.ts
targets a different, unrelated judge shape entirely); no cost/token
tracking wired from this scorer into eval report totals, since Mastra
scorers have no side channel back to the report; the judge is
inherently non-deterministic where the old regex scorer was not,
which affects single-run headline comparisons; and the default judge
model (Gemini Flash Lite) could end up judging same-family candidate
models with no enforcement against that.
…oint

pnpm audit calls registry.npmjs.org's quick/bulk audit endpoints, both
of which npm has permanently retired (HTTP 410 on every request,
reproduced locally against pnpm 10.34.3 and latest 11.13.0 alike - not
a transient outage). Add --ignore-registry-errors, pnpm's own
documented flag for exactly this case ("A build should not fail
because the registry has issues"): the job still runs and prints the
real error, it just stops treating a registry-side response as a CI
failure. The moment npm/pnpm fix the endpoint, real audit results
resume without further changes needed here.
CI failed this test 3/3 times in one run (identical "Investigation
Plan region not found" error every retry) despite it passing cleanly
on the immediately preceding commit, whose diff couldn't plausibly
affect it (only touched the unrelated Security Audit CI step).

Root cause: playwright.config.ts's webServer runs `pnpm dev`, and this
is the only test in this spec file - alphabetically the first spec
Playwright runs, so it's the first to hit the freshly booted dev
server. Next.js dev mode compiles routes on demand on first request;
that compile lag varies with CI runner speed and can occasionally
exceed the default 5s expect timeout even though every route in this
test is fully mocked (no real network variance). Bump just the first
assertion's timeout to 20s so a slow first compile doesn't fail the
whole test; every assertion after it benefits from the app already
being warm.
@justsml justsml marked this pull request as ready for review July 16, 2026 00:51
@justsml justsml merged commit 9ec9259 into main Jul 16, 2026
7 checks passed
@justsml justsml deleted the claude/next-eval-cases-s1f4b1 branch July 16, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants