Feat: benchmark agent token cost with and without QAMap on a fixed task suite

[ivory-code]

Summary

The README is careful not to claim that a calling agent spends zero model tokens, and bench:context measures context reuse, but there was no measured answer to the question adopters ask first: how much does a QA task cost with QAMap in the loop versus without it? This adds pnpm bench:agent, a provider-neutral A/B harness over a fixed public task suite, reporting provider-reported tokens, tool calls, wall-clock, and deterministic task success as median plus range, with first-authoring cost in its own column.

Behavioral Contract

• test/agent-tasks/ holds three tasks that reuse committed test/benchmarks/ fixtures — reproduce a duplicate-request regression, verify surface copy against a specification table, and re-verify a persistence fix after a seeded regression. Each task.json validates against test/agent-tasks/schema.json; success is judged only by deterministic local checks (file exists, command exit code, stdout includes, JSON path equals). The model's prose is never read.
• scripts/agent-bench.mjs materializes each fixture as a temporary Git repository and runs the same system prompt and task prompt through the same model twice: generic (bash, read_file, list_dir, grep) and qamap (the same tools plus qamap_qa, qamap_qa_run, qamap_e2e_draft_dry_run shelling out to the local dist/cli.js). Only the tool list differs.
• Token counts come only from the provider's usage fields, summed over turns; a missing field makes the adapter throw, and a field the provider does not report stays null. Tool calls are the tool-use blocks the provider returned. Provider, model, system prompt sha256, tool schema sha256, QAMap version, and run count are pinned in the report.
• Without QAMAP_BENCH_API_KEY the command prints status: skipped and exits 0, also with --assert, so CI stays green without a key. --dry-run replaces the provider with a scripted stand-in that returns null token counts; its report is deterministic and is never a measurement. Supported providers: Messages API and Chat Completions API through Node's built-in fetch, no new dependency.
• Reports never contain pricing, a fixed saving multiplier, absolute temporary paths, or the key. The interpretation block states that QAMap itself makes no model request and that both arms spend the calling agent's tokens.
• Benchmark runners now share scripts/lib/fixture-repo.mjs; bench:ci output is byte-identical before and after the refactor apart from durations.

Evidence

Closes #211.

• test/agent-benchmark.test.mjs: two dry runs deep-equal; schema name and dry-run status; pinned hashes present; forbidden patterns absent ($<digits>, "times cheaper", "x cheaper", temp paths, key prefixes); no key and no --dry-run yields skipped with exit 0 even under --assert.
• test/agent-benchmark-provider.test.mjs: usage parsers return exact numbers and throw when usage is missing; an injected fetch proves the key reaches only the request header and never the returned objects or error messages.
• test/agent-benchmark-aggregate.test.mjs: odd and even medians, range, first-authoring split, errored runs excluded.
• test/agent-tasks.test.mjs: every task validates, referenced fixtures exist, success criteria use local kinds only.
• Negative control: the dry-run report carries null token fields rather than numbers, so nothing reads as measured.

Checks

• Focused regression test
• pnpm test
• pnpm bench:ci for inference, routing, trace, or output
• pnpm bench:execution for E2E compiler or execution fixtures
• pnpm scan for scanner, security, or repository policy
• pnpm plugin:check and pnpm plugin:smoke for plugin changes
• Documentation links and commands verified

Public OSS Check

• No private repository, source, path, customer data, credential, or internal smoke output is included.
• Shared inference has unrelated positive and negative coverage, or this is not applicable.
• User-facing commands and claims match actual behavior.

Review Notes

pnpm bench:agent --dry-run --assert runs in CI and in release:check; pnpm bench:context, pnpm plugin:check, and pnpm plugin:smoke also pass locally (plugin checks marked N/A, no plugin asset changed). No measured run has been executed yet: the measured path is verified only through the injected fetch, so the first run against a live provider may surface response-shape differences. Success checks are static (files, exit codes, stdout, structured answers) and do not drive a browser; test/agent-tasks/README.md says so. Chat Completions does not report cache-write tokens, so that field stays null rather than zero. Publishing a results table is deliberately left to a run with a pinned model and recorded provider usage.