feat(codex): surface cached/reasoning token breakdown in Challenge and Consult modes - #2660
Open
nova28 wants to merge 1 commit into
Open
feat(codex): surface cached/reasoning token breakdown in Challenge and Consult modes#2660nova28 wants to merge 1 commit into
nova28 wants to merge 1 commit into
Conversation
…d Consult modes codex exec --json's turn.completed event reports five usage fields (input, cached_input, cache_write_input, output, reasoning_output), but the Challenge and Consult mode parsers only ever read input_tokens + output_tokens into a single "tokens used: N" line, discarding the other three. A consumer that wants a real per-call cost breakdown (cache hit rate, reasoning spend vs response spend) has no way to get one. Add a second, machine-parseable line printed alongside the existing human-readable one: CODEX_USAGE: input=21294 cached_input=6912 cache_write=0 output=5 reasoning_output=0 The existing "tokens used: N" line is untouched byte-for-byte, so nothing that already parses it needs to change. Verified against a live `codex exec --json` capture (codex-cli 0.146.0) — the usage object's field names were confirmed live rather than assumed from the gpt.ts benchmark helper, which also checks for an `obj.model` field that does not actually appear anywhere in the real event stream on this CLI version (left alone here; that's a separate, unverified claim not worth carrying into this change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why (in your own words)
codex exec --json'sturn.completedevent already reports a five-field usagebreakdown —
input_tokens,cached_input_tokens,cache_write_input_tokens,output_tokens,reasoning_output_tokens— but the Challenge and Consult modeparsers in
/codexonly readinput_tokens + output_tokensinto a singletokens used: Nline and silently drop the other three. Anything downstream thatwants to actually price a codex call (cache hit rate, reasoning spend vs response
spend) has nothing to read. This adds a second, machine-parseable line
(
CODEX_USAGE: input=... cached_input=... cache_write=... output=... reasoning_output=...)printed right alongside the existing one, so the human-readable output is
byte-for-byte unchanged and any downstream tooling can key off the new line.
Live evidence
That raw capture piped through the exact parser code now in
codex/SKILL.md.tmpl(Challenge-mode block):
tokens used: 21299is unchanged from before this PR (21294 + 5).CODEX_USAGEisthe new line, sourced from fields that were already in the JSON object and simply
weren't being read.
Note on
obj.model: while investigating this I checkedtest/helpers/providers/gpt.ts,which has a defensive
if (obj.model) modelUsed = obj.model;on the sameturn.completedevent. I could not find amodelfield anywhere in a real capture(checked every event type:
thread.started,turn.started,item.completed,turn.completed) on codex-cli 0.146.0. Left that alone — not touching an existingassumption I can't independently confirm, and out of scope for this PR either way.
Scope
codex/SKILL.md.tmpl— Challenge mode and Consult mode (new-session)JSONL parsers both print a
CODEX_USAGE:line onturn.completed, in addition tothe existing
tokens used: Nline. Consult mode's resumed-session path shares thesame parser by reference ("same python streaming parser as above" in the template)
so it picks this up without a separate edit. Documented in the
## Cost Estimationsection. Regenerated
codex/SKILL.mdviabun run gen:skill-docs(verified with--dry-runafterward — all files reportFRESH).codex exec --jsoncall (codex-cli 0.146.0, shownabove), the extracted parser code run standalone against that capture (shown
above),
bun test test/skill-validation.test.ts(331 pass / 0 fail — includes theexisting
expect(content).toContain('tokens used')assertion, unaffected since thatstring is untouched), and the full codex test surface:
codex-e2e*.test.ts,codex-hardening.test.ts,codex-model-probe.test.ts,codex-resume-flag-semantics.test.ts,codex-under-codex-detection.test.ts,codex-web-search-flag.test.ts,codex-generation-model.test.ts(76 pass / 20skip / 0 fail, 96 tests across 10 files).
codex reviewneveruses
--json, so it has noturn.completedevent to read this from at all — notedexplicitly in the doc addition). Did not add a new automated test asserting the
CODEX_USAGE:line's exact format, since the existing test suite validates renderedskill content/structure rather than executing the embedded Python against live
JSONL — happy to add one if a maintainer wants a specific shape/location for that.
Liveness proof (required)
Checklist
GSTACK PRtyped live into a real surface (not edited onto the image) — pending, see note above