Add remaining open eval cases from the 25-case proposal#28
Merged
Conversation
Turns the TODO.md eval-dataset-audit gap list into 25 concrete, authorable rows across missing vulnerability classes, agent-behavior safety gaps, north-star patch/replay/report coverage, and a second-target diversification row, with a suggested authoring order.
This lock file carries a per-session PID/timestamp and isn't project source; nothing to persist here.
Most of the originally-proposed 25 next eval cases already shipped in bb64abc before this pass started; reconcile TODO.md and PROPOSED_NEXT_25_CASES.md to reflect that and scope down to the ~5 genuine gaps that were still open: - OAuth/SSO redirect-validation and business-logic checkout-abuse scenarios, added to both the anonymous attack-vector lane and the Juice Shop lane (evals/attack-vectors.ts, datasets.ts, juice-shop-eval-config.ts). - Two patch-diff-blind scenarios and one remediation report-writing scenario, covering the north-star loop's remediation/report legs (new patch-diff-scenarios.ts and report-writing-scenarios.ts, registered in datasets.ts and wired into canonical-schema.ts's artifact/mode/environment mapping). Patch-diff-blind rows are prompt-only for now; there is no live repo-checkout runner yet. - A benchmarks/exploit-hunter/second-webapp pack scaffold for target diversification, which intentionally fails closed: the obvious second-target image bakes its own well-known branding into server-rendered HTML with no runtime theming hook, so running it as-is under an anonymous target-disclosure claim would leak the benchmark's identity to the model under evaluation. The README documents the follow-up work needed before it can start a real container. Updated the hardcoded dataset-name/scenario-id lists and the CI regression-gate baseline (13->15 datasets, 123->130 canonical rows) in their corresponding tests to match.
The two scorers were flagged as stale: countVulnerabilityFindings kept a fixed 8-keyword regex taxonomy (sqli, xss, idor, ...) that silently scored 0 for every vulnerability class added since (SSTI, CSRF, JWT, GraphQL, business-logic, OAuth, ...), rewarding keyword presence over genuine evidence-backed discovery. Replace webapp-vulnerability-count and webapp-finding-quality with an LLM-judge implementation (same ids/names, so all existing registration and dataset scorerIds wiring is untouched): the judge identifies an open-ended list of distinct vulnerability classes described in the response, marks each evidence_backed or not, and scores diversity/evidence-grounding/context-adherence/drift-avoidance dimensions. Both scorers share one cached judge call per response (module-level Map keyed by judge model + response hash) so they stay mutually consistent and avoid double-billing a second LLM call. The deterministic countVulnerabilityFindings/assessWebappQuality/ scoreWebappResponse helpers are left untouched, since they're also used directly (not just via these two scorers) for synchronous report fields in scripts/live-evals/webapp-model-eval.ts and are covered by existing regression tests. The other five webapp scorers (target observation, evidence discipline, system model, investigation angles, safety gating) are unaffected.
The first pass shipped the judge scorers with zero tests (only verified the module imports without throwing) and no request timeout or clear failure message. Fix the concrete gaps: - Add a request timeout (90s) to the generateObject call so a stalled judge call can't hang an entire eval run. - Fail with a clear, actionable error when OPENROUTER_API_KEY is missing instead of an opaque network/auth failure surfacing deep in the OpenRouter client. - Bump the judge's output token budget (900 -> 1400) for headroom: the schema's findings array plus four dimension reasons can run larger than the plan-judge schema this pattern was based on. - Add tests/security-chat/webapp-judge-scorer.test.ts mocking generateObject to actually exercise the scoring logic: counts distinct findings and flags unsupported ones in the reason, normalizes the judge's 0-5 score to 0-1, scores a refusal/drift response at zero, verifies the two scorers share one cached judge call for the same response (no double billing), and verifies a missing API key throws with an actionable message before the judge is ever called. Remaining known gaps, not fixed here: no judge/human calibration for this specific judge (the repo's existing judge-model-calibration.ts targets a different, unrelated judge shape entirely); no cost/token tracking wired from this scorer into eval report totals, since Mastra scorers have no side channel back to the report; the judge is inherently non-deterministic where the old regex scorer was not, which affects single-run headline comparisons; and the default judge model (Gemini Flash Lite) could end up judging same-family candidate models with no enforcement against that.
…oint
pnpm audit calls registry.npmjs.org's quick/bulk audit endpoints, both
of which npm has permanently retired (HTTP 410 on every request,
reproduced locally against pnpm 10.34.3 and latest 11.13.0 alike - not
a transient outage). Add --ignore-registry-errors, pnpm's own
documented flag for exactly this case ("A build should not fail
because the registry has issues"): the job still runs and prints the
real error, it just stops treating a registry-side response as a CI
failure. The moment npm/pnpm fix the endpoint, real audit results
resume without further changes needed here.
CI failed this test 3/3 times in one run (identical "Investigation Plan region not found" error every retry) despite it passing cleanly on the immediately preceding commit, whose diff couldn't plausibly affect it (only touched the unrelated Security Audit CI step). Root cause: playwright.config.ts's webServer runs `pnpm dev`, and this is the only test in this spec file - alphabetically the first spec Playwright runs, so it's the first to hit the freshly booted dev server. Next.js dev mode compiles routes on demand on first request; that compile lag varies with CI runner speed and can occasionally exceed the default 5s expect timeout even though every route in this test is fully mocked (no real network variance). Bump just the first assertion's timeout to 20s so a slow first compile doesn't fail the whole test; every assertion after it benefits from the app already being warm.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR started as a proposal document (
evals/PROPOSED_NEXT_25_CASES.md) turningTODO.md's eval-dataset-audit gap list into 25 concrete case proposals. While drafting the implementation, it turned out commitbb64abchad already landed ~20 of those 25 rows directly onmain. This PR now:evals/PROPOSED_NEXT_25_CASES.mdandTODO.mdare updated to mark each of the 25 items DONE / DONE (merged) / OPEN against what's actually indatasets.tstoday.oauth-redirect-symptom/oauth-sso-redirect-validationandbusiness-logic-symptom/business-logic-checkout-abuse— the two vulnerability classes still missing from the anonymous attack-vector and Juice Shop lanes (evals/attack-vectors.ts,src/mastra/evals/datasets.ts,scripts/live-evals/juice-shop-eval-config.ts).src/mastra/evals/patch-diff-scenarios.tsandsrc/mastra/evals/report-writing-scenarios.ts, registered indatasets.ts, wired intocanonical-schema.ts's artifact/mode/environment mapping). These are prompt-only rows — there is no live repo-checkout runner forpatch-diff-blindyet, and the dataset description says so.benchmarks/exploit-hunter/second-webapppack scaffold for target diversification (item [Tier 3] Create logo and brand identity #18 in the audit). It intentionally fails closed: the obvious second-target image bakes its own well-known branding into server-rendered HTML with no runtime theming hook, so serving it as-is under ananonymoustarget-disclosure claim would leak the benchmark's identity to the model under evaluation — the exact contamination failure mode the Eval Honesty Rules exist to prevent.init_script.sh/start_docker.shexit non-zero with an explanation; the README documents the follow-up work (a Dockerfile that neutralizes the branding) needed before it's usable.tests/security-chat/eval-datasets.test.tsandtests/security-chat/juice-shop-eval-scripts.test.ts, plus the CI regression-gate baseline inevals/ci-regression-baseline.json(13→15 datasets, 123→130 canonical rows) and its test.webapp-vulnerability-count/webapp-finding-qualityscorers to an LLM judge. These two scorers previously counted vulnerability classes against a fixed 8-keyword regex taxonomy that never kept pace with the dataset's growth to 20+ classes — a correct SSTI or business-logic finding scored as if the response found nothing. They're now backed by an LLM judge (src/mastra/evals/webapp-scorers.ts) that identifies an open-ended list of distinct vulnerability classes, marks each evidence-backed or not, and scores diversity/evidence-grounding/context-adherence/drift-avoidance. Both scorers share one cached judge call per response to stay mutually consistent and avoid double billing. Same scorer ids/registration as before, so no other wiring changed; the deterministic helpers they used to call directly (countVulnerabilityFindings,assessWebappQuality,scoreWebappResponse) are untouched since they're also used for synchronous report fields elsewhere and are covered by existing tests.Type of change
How Has This Been Tested?
pnpm typecheckpassespnpm buildpassespnpm testpasses (fulltests/security-chatsuite: 178 files / 1081 tests)pnpm eval:ops:auditpasses (0 errors; only pre-existing cybench raw-pack warnings)pnpm eval:publish:auditpasses (0 errors; warnings are the expected "unreviewed row" / one pre-existing cybench duplicate)OPENROUTER_API_KEYconfigured); the implementation mirrors the already-in-productionplan-actionability-llm-judgescorer's model-selection pattern.Checklist
plan-creation-scorers.ts)Security Considerations
The new
second-webappbenchmark pack is designed to fail closed rather than risk starting a target that leaks the benchmark's identity to the model under evaluation (a contamination/scope-realism concern, not a traditional security vulnerability). No product runtime code, approval gates, or sandboxing paths were touched. The new judge-based scorer only reads already-generated eval output text and calls OpenRouter with a judge model; it does not execute or influence anything in the runtime security-research agent.