Skip to content

feat(codex): surface cached/reasoning token breakdown in Challenge and Consult modes - #2660

Open
nova28 wants to merge 1 commit into
garrytan:mainfrom
nova28:codex-usage-breakdown
Open

feat(codex): surface cached/reasoning token breakdown in Challenge and Consult modes#2660
nova28 wants to merge 1 commit into
garrytan:mainfrom
nova28:codex-usage-breakdown

Conversation

@nova28

@nova28 nova28 commented Aug 21, 2026

Copy link
Copy Markdown

Why (in your own words)

codex exec --json's turn.completed event already reports a five-field usage
breakdown — input_tokens, cached_input_tokens, cache_write_input_tokens,
output_tokens, reasoning_output_tokens — but the Challenge and Consult mode
parsers in /codex only read input_tokens + output_tokens into a single
tokens used: N line and silently drop the other three. Anything downstream that
wants to actually price a codex call (cache hit rate, reasoning spend vs response
spend) has nothing to read. This adds a second, machine-parseable line
(CODEX_USAGE: input=... cached_input=... cache_write=... output=... reasoning_output=...)
printed right alongside the existing one, so the human-readable output is
byte-for-byte unchanged and any downstream tooling can key off the new line.

Live evidence

$ codex exec "say hi in one word" -s read-only --skip-git-repo-check --json < /dev/null
{"type":"thread.started","thread_id":"01a0249b-250c-7e13-a32d-82b9746dd022"}
{"type":"turn.started"}
{"type":"item.completed","item":{"id":"item_0","type":"error","message":"Skill descriptions were shortened..."}}
{"type":"item.completed","item":{"id":"item_1","type":"agent_message","text":"Hi"}}
{"type":"turn.completed","usage":{"input_tokens":21294,"cached_input_tokens":6912,"cache_write_input_tokens":0,"output_tokens":5,"reasoning_output_tokens":0}}

That raw capture piped through the exact parser code now in codex/SKILL.md.tmpl
(Challenge-mode block):

$ cat codex-verify-out.jsonl | python3 -u -c "<the new parser code>"
Hi

tokens used: 21299
CODEX_USAGE: input=21294 cached_input=6912 cache_write=0 output=5 reasoning_output=0

tokens used: 21299 is unchanged from before this PR (21294 + 5). CODEX_USAGE is
the new line, sourced from fields that were already in the JSON object and simply
weren't being read.

Note on obj.model: while investigating this I checked test/helpers/providers/gpt.ts,
which has a defensive if (obj.model) modelUsed = obj.model; on the same
turn.completed event. I could not find a model field anywhere in a real capture
(checked every event type: thread.started, turn.started, item.completed,
turn.completed) on codex-cli 0.146.0. Left that alone — not touching an existing
assumption I can't independently confirm, and out of scope for this PR either way.

Scope

  • Changed: codex/SKILL.md.tmpl — Challenge mode and Consult mode (new-session)
    JSONL parsers both print a CODEX_USAGE: line on turn.completed, in addition to
    the existing tokens used: N line. Consult mode's resumed-session path shares the
    same parser by reference ("same python streaming parser as above" in the template)
    so it picks this up without a separate edit. Documented in the ## Cost Estimation
    section. Regenerated codex/SKILL.md via bun run gen:skill-docs (verified with
    --dry-run afterward — all files report FRESH).
  • Verified live by: a real codex exec --json call (codex-cli 0.146.0, shown
    above), the extracted parser code run standalone against that capture (shown
    above), bun test test/skill-validation.test.ts (331 pass / 0 fail — includes the
    existing expect(content).toContain('tokens used') assertion, unaffected since that
    string is untouched), and the full codex test surface: codex-e2e*.test.ts,
    codex-hardening.test.ts, codex-model-probe.test.ts,
    codex-resume-flag-semantics.test.ts, codex-under-codex-detection.test.ts,
    codex-web-search-flag.test.ts, codex-generation-model.test.ts (76 pass / 20
    skip / 0 fail, 96 tests across 10 files).
  • Did NOT test: Review mode is unaffected by design (native codex review never
    uses --json, so it has no turn.completed event to read this from at all — noted
    explicitly in the doc addition). Did not add a new automated test asserting the
    CODEX_USAGE: line's exact format, since the existing test suite validates rendered
    skill content/structure rather than executing the embedded Python against live
    JSONL — happy to add one if a maintainer wants a specific shape/location for that.

Liveness proof (required)

Checklist

  • Liveness screenshot attached: GSTACK PR typed live into a real surface (not edited onto the image) — pending, see note above
Screenshot 2026-08-21 at 11 43 16 PM - [x] This is not a generated-file-only diff (I edited the source/template and regenerated) - [x] No ETHOS.md edits, and no changes to voice / founder perspective / YC references - [x] New public command / external service / host adapter has an accepted issue linked (or N/A) — N/A, no new command/service/adapter - [ ] Linked issue or reproduction: #

…d Consult modes

codex exec --json's turn.completed event reports five usage fields (input, cached_input,
cache_write_input, output, reasoning_output), but the Challenge and Consult mode parsers
only ever read input_tokens + output_tokens into a single "tokens used: N" line, discarding
the other three. A consumer that wants a real per-call cost breakdown (cache hit rate,
reasoning spend vs response spend) has no way to get one.

Add a second, machine-parseable line printed alongside the existing human-readable one:

  CODEX_USAGE: input=21294 cached_input=6912 cache_write=0 output=5 reasoning_output=0

The existing "tokens used: N" line is untouched byte-for-byte, so nothing that already
parses it needs to change. Verified against a live `codex exec --json` capture (codex-cli
0.146.0) — the usage object's field names were confirmed live rather than assumed from the
gpt.ts benchmark helper, which also checks for an `obj.model` field that does not actually
appear anywhere in the real event stream on this CLI version (left alone here; that's a
separate, unverified claim not worth carrying into this change).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Aug 21, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant