feat: record token usage and duration for eval runs

[mattrossman]

Records usage and durationMs on each result so we can see what a run costs in tokens and how long the agent took. Usage tracks input, cache read, cache write, and output tokens, inspired by OTel GenAI conventions where the cache buckets are subsets of input. All four harnesses report these fields, and from them we can price a run (the two cache buckets each have their own rate, and input minus both is the fresh input billed at the base rate), see OpenAI or Claude pricing.

// result.json
{
  // ...,
  "usage": [
    {
      "model": "gpt-5.6-sol",
      "inputTokens": 267028,
      "cacheReadInputTokens": 230032,
      "cacheWriteInputTokens": 36957,
      "outputTokens": 1971
    }
  ],
  "durationMs": 61877
}

Usage is a list because runs can potentially use multiple models, e.g. by default Claude Code incurs a small amount of Haiku usage for title renames (thread), but we could also see runs that spawn subagents.

Aside: based on that finding, I took the opportunity to disable background traffic (Claude Code nonessential traffic + session title, codex analytics, opencode auto-update) to prevent unexpected token usage or other run interference. E.g. CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC and CLAUDE_CODE_DISABLE_TERMINAL_TITLE per docs.

The UI details for a run now shows duration and a one-line token total (Preview deep link)

Sanity checked every bucket against raw CLI output for all three sandbox harnesses plus direct OpenAI API calls, and ran one eval per harness end to end. After merge, we can let performance metrics trickle in from nightly regression runs, and optionally trigger a full benchmark refresh when it's convenient.

Closes AI-971

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
evals	Ready	Preview	Sep 7, 2026 11:40pm UTC

[mattrossman]

Dispatching a sample run of one eval per benchmark experiment to demo what the data and UI looks like.

[mattrossman]

Realizing that the current shape deviates slightly from OTel GenAI conventions, gonna align more closely to that. Same data ultimtately just different derivation.

(Edit: pushed that change and kicking off new run)

[Rodriguespn]

Approved - token/duration usage wiring is solid across all four harnesses (inclusive OTel-style buckets, real fixture-tested math).

Small nits: ai-sdk lacks an usage-extraction test the other three have, and the shipped inclusive inputTokens naming diverges from the plan doc's disjoint design without a reconciliation note. IIRC, we're not running any ai-sdk experiment anymore but for consistency sake, should we also add a test for this to the ai-sdk?