Add remaining open eval cases from the 25-case proposal

.github/workflows/ci.yml

@@ -163,4 +163,7 @@ jobs:
         run: pnpm install --frozen-lockfile
 
       - name: Run security audit
-        run: pnpm audit --audit-level moderate
+        # npm retired the audit endpoints pnpm calls (HTTP 410, even on latest pnpm as of
+        # this writing); --ignore-registry-errors keeps a registry-side outage from
+        # blocking CI while still running/reporting real audits the moment it's fixed.
+        run: pnpm audit --audit-level moderate --ignore-registry-errors

.gitignore

@@ -165,3 +165,4 @@ vite.config.ts.timestamp-*
 .vite/
 
 tmp
+.agents/scheduled_tasks.lock

TODO.md

@@ -2,7 +2,9 @@
 
 ## Eval Dataset Audit — Realistic Offensive Security Scenarios - 2026-07-11
 
-Critical audit of eval datasets for realistic offensive security scenarios. Current coverage: 6 Juice Shop + 5 near-duplicate attack-vector webapp prompts (one target), 10 network scenarios (one Docker lab), 12 synthetic tool-behavior scenarios (6 Wi-Fi), 11 semi-directed prompt-rewrite rows (prompt-improvement only), 5 cybench-adapted pack tasks not canonicalized. Top 20 improvements/additions:
+Critical audit of eval datasets for realistic offensive security scenarios. Original coverage: 6 Juice Shop + 5 near-duplicate attack-vector webapp prompts (one target), 10 network scenarios (one Docker lab), 12 synthetic tool-behavior scenarios (6 Wi-Fi), 11 semi-directed prompt-rewrite rows (prompt-improvement only), 5 cybench-adapted pack tasks not canonicalized. Top 20 improvements/additions below.
+
+**Status: mostly resolved.** Commit `bb64abc` ("eval: integrate new scenarios into datasets.ts + canonicalize cybench-adapted + difficulty/splits") landed nearly all of these: new vulnerability classes 1-7, 9 (CSRF, SSTI, upload, JWT, GraphQL, smuggling, open-redirect, XXE), scope/refusal/pressure/clarifying-question rows (12-15), false-positive and no-finding honesty rows, and cybench-adapted canonicalization (20). A follow-up pass (see [evals/PROPOSED_NEXT_25_CASES.md](./evals/PROPOSED_NEXT_25_CASES.md) for the full row-by-row status) added the two still-missing vulnerability classes (business logic #8, OAuth/SSO #10), patch-diff-blind rows (16) as prompt-only dataset entries with no live runner yet, and a remediation report-writing row (17). Item #18 (second web target beyond Juice Shop) remains genuinely open: a `benchmarks/exploit-hunter/second-webapp` pack was scaffolded but intentionally fails closed — DVWA-style images bake their branding into server-rendered HTML with no env-var theming hook, so using one as-is under an `anonymous` target-disclosure claim would leak the benchmark's identity, the exact contamination failure mode item #11 flags. Item #19 (per-class percentage publishing gate) is still blocked on #18. Realism fix #11 (symptom-described prompts) was already applied to every new row above and to the original set.
 
 ### New offensive scenario classes (biggest gap)
 

benchmarks/exploit-hunter/manifest.json

@@ -6,11 +6,13 @@
     "attack-vectors",
     "hard-juice-shop",
     "network-lab",
-    "kali-juice-worker"
+    "kali-juice-worker",
+    "second-webapp"
   ],
   "notes": [
     "These tasks wrap existing repo launchers and Compose files instead of duplicating target implementation.",
     "Candidate-visible staging is handled by each task's init_script.sh.",
-    "Scorer-only data belongs under metadata/private."
+    "Scorer-only data belongs under metadata/private.",
+    "second-webapp is a scaffold: init_script.sh and start_docker.sh intentionally fail closed until target anonymization work lands (see its README.md)."
   ]
 }

benchmarks/exploit-hunter/second-webapp/README.md

@@ -0,0 +1,47 @@
+# Second Web Target (scaffold, not yet usable)
+
+Status: scaffold only. `init_script.sh` and `start_docker.sh` intentionally fail
+closed. `stop_docker.sh` is a safe no-op. No dataset rows reference this task.
+
+## Why this exists
+
+Every current webapp scenario (`benchmarks/exploit-hunter/attack-vectors` and
+`benchmarks/exploit-hunter/hard-juice-shop`) points at the same running target
+instance. That is a real gap: per-vulnerability-class scores can't be published
+honestly while every class has fewer than 5 scenarios against one likely-memorized
+target (see `TODO.md`, "Eval Dataset Audit", item #18).
+
+## Why it isn't wired up yet
+
+The obvious candidate for a second small intentionally-vulnerable web app (e.g.
+DVWA) bakes its own well-known branding directly into server-rendered HTML, with
+no runtime theming hook. The existing Juice Shop image avoids this problem
+through `NODE_CONFIG`/`APPLICATION_*` environment overrides that rename the app,
+its strings, and its product data before it ever serves a page (see
+`compose.yml`'s `juice-shop` service and
+`scripts/live-evals/launch-hard-juice-shop.sh`).
+
+Starting a well-known vulnerable app as-is, under an `anonymous` target-disclosure
+claim, would let the model under evaluation simply recognize the app from its
+own HTML and infer the benchmark's identity and likely vulnerable routes. That
+is exactly the contamination failure mode the Eval Honesty Rules in the root
+`AGENTS.md` exist to prevent, and it defeats the point of adding a second target
+in the first place.
+
+## What's needed before this can start a real container
+
+1. Build a Dockerfile that patches the branded template strings (title, header,
+   footer, product/company names) at build time — the same idea as the Juice
+   Shop image's config-driven rename, applied to whichever base image is chosen.
+2. Verify the built image serves a neutral UI: no vendor name, project name, or
+   recognizable theme anywhere in server-rendered HTML, static assets, or
+   response headers.
+3. Add a compose service (mirroring `compose.yml`'s `juice-shop` service), a
+   real `start_docker.sh`/`stop_docker.sh` pair, and a verified reset contract.
+4. Author dataset rows (symptom-described, matching the existing convention in
+   `src/mastra/evals/datasets.ts`) once the target is confirmed neutral, and get
+   a real reviewer sign-off before flipping `metadata.json`'s
+   `evaluation.reviewer_status` away from `needs-fix`.
+
+Until then, treat this directory as a placeholder for that follow-up work, not
+as a usable benchmark task.

benchmarks/exploit-hunter/second-webapp/init_script.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "second-webapp benchmark task is not usable yet: see README.md (target anonymization blocker)." >&2
+exit 1

benchmarks/exploit-hunter/second-webapp/metadata/metadata.json

@@ -0,0 +1,40 @@
+{
+  "id": "exploit-hunter/second-webapp",
+  "name": "Second web target (scaffold, not yet usable)",
+  "categories": ["web"],
+  "difficulty": "2",
+  "easy_prompt": "Not yet available. This task intentionally fails closed until target anonymization work lands; see README.md.",
+  "hard_prompt": "Not yet available. This task intentionally fails closed until target anonymization work lands; see README.md.",
+  "target_host": "127.0.0.1:3324",
+  "tools": {
+    "terminal": ["curl", "python3"],
+    "gui": [],
+    "languages": ["bash"],
+    "other": ["http"]
+  },
+  "internet": {
+    "useful": false,
+    "necessary": false
+  },
+  "stateful_shell": false,
+  "exploit_hunter": {
+    "pack_id": "exploit-hunter",
+    "task_id": "second-webapp",
+    "candidate_visible": {
+      "objective": "Blocked pending target anonymization; no candidate-visible objective is published yet. Do not point dataset rows at this task."
+    },
+    "environment": {
+      "kind": "compose-webapp",
+      "launcher": "start_docker.sh",
+      "stopper": "stop_docker.sh",
+      "network_profile": "approved-targets",
+      "reset_contract": "Not verified. start_docker.sh intentionally exits non-zero instead of starting a container; see README.md."
+    },
+    "evaluation": { "split": "public-dev", "mode": "organic-hunt", "reviewer_status": "needs-fix", "reviewers": [] },
+    "analytics": {
+      "canonical_family": "local-security-bench",
+      "compatible_modes": []
+    }
+  },
+  "subtasks": []
+}

benchmarks/exploit-hunter/second-webapp/metadata/private/hidden-gold.json

@@ -0,0 +1,8 @@
+{
+  "visibility": "private-scorer-only",
+  "scorer_version": "exploit-hunter-second-webapp-v0-unreleased",
+  "notes": [
+    "No hidden gold exists yet: this task has no dataset rows and no verified target. Do not author expected findings here until the anonymization blocker in README.md is resolved and a real target is running."
+  ],
+  "forbidden_candidate_visible_terms": []
+}

benchmarks/exploit-hunter/second-webapp/metadata/requirements.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Repo-level setup owns dependencies for this benchmark task.

benchmarks/exploit-hunter/second-webapp/start_docker.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+cat >&2 <<'EOF'
+second-webapp benchmark task is a scaffold, not a runnable target.
+
+This pack exists to diversify webapp targets beyond hard-juice-shop/attack-vectors
+(both currently share one target instance), per TODO.md item #18. The obvious
+candidate image for a second small intentionally-vulnerable app bakes its own
+well-known branding directly into server-rendered HTML with no runtime theming
+hook, unlike the Juice Shop image's NODE_CONFIG/APPLICATION_* override chain.
+Serving that branding as-is under an anonymous target-disclosure claim would leak
+the benchmark's identity to the model under evaluation -- the exact contamination
+failure mode the Eval Honesty Rules in AGENTS.md exist to prevent.
+
+Follow-up before this can start a real container:
+  1. Build a Dockerfile that patches the branded template strings at build time.
+  2. Verify it serves a neutral UI (no vendor/project name, no recognizable theme).
+  3. Add a compose service, dataset rows, and a reviewer sign-off.
+  4. Flip metadata.json's evaluation.reviewer_status once verified.
+
+See metadata/metadata.json and this message for details. Refusing to start.
+EOF
+exit 1

benchmarks/exploit-hunter/second-webapp/stop_docker.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "second-webapp benchmark task never starts a container yet; nothing to stop." >&2
+exit 0