Bash-heavy workflows are misread: shell reads uncounted, retries overcounted

[laulpogan]

Problem

Two detectors treat the Bash tool inconsistently, and the result is that any workflow which does file lookup through the shell gets scored badly for doing nothing wrong.

READ_TOOL_NAMES ignores Bash entirely, so rg, grep, and cat count as zero reads. countRetries treats any Bash call as a verification step, so edit → grep → edit is recorded as rework.

The same tool is invisible where it should count and decisive where it should not. Both detectors then feed the Optimize tab and the efficiency grade.

1. read-edit-ratio does not count shell reads

src/optimize.ts:2190

export const READ_TOOL_NAMES = new Set(['Read', 'Grep', 'Glob', 'FileReadTool', 'GrepTool', 'GlobTool'])

Bash is absent, so a session that searches with rg reads nothing as far as this detector is concerned.

Measured over 7 days of local Claude transcripts (1,787 session files), counting tool_use blocks:

Source	Calls
Read / Grep / Glob	539
Bash, read-shaped (rg, grep, cat, head, find, git log, git diff, …)	5,277
Edit / Write / NotebookEdit	1,565

Ratio excluding Bash: 0.34:1. Including it: 3.72:1, against a HEALTHY_READ_EDIT_RATIO of 4.

The detector reported "Claude edits more than it reads" at high impact with trend: active. Roughly 90% of the reads were never counted.

This bites hardest on the workflows the advice is aimed at. An agent instructed to prefer rg over the Grep tool — a common convention, and faster — scores worse the more disciplined it is about looking before editing.

2. countRetries treats every Bash call as verification

src/classifier.ts:168

if (BASH_TOOLS.has(call.tool)) {
  lastVerifyStep = i
}
if (EDIT_TOOLS.has(call.tool)) {
  const fileKey = call.file ?? '__no_file__'
  const prevStep = lastEditStep.get(fileKey)
  if (prevStep !== undefined && lastVerifyStep > prevStep && lastVerifyStep < i) {
    retries++
  }

Any Bash call sets lastVerifyStep. So edit X → ls → edit X and edit X → pytest → edit X are scored identically, though only the second is rework. The first is a normal refinement.

Replaying this algorithm over the same 7 days and classifying the intervening command:

Intervening Bash command	Retries	Share
Real verification (pytest, vitest, tsc, eslint, npm run build, cargo test, …)	53	15.7%
Lookup only (rg, ls, cat, git log, git status, …)	284	84.3%

This number propagates. retryTax in src/usage-aggregator.ts:634 multiplies retries by average cost per edit turn, and it drove the headline "retry tax" past 190% of spend on this machine. findLowWorthCandidates (src/optimize.ts:2578) also gates on retries >= WORTH_IT_MIN_RETRIES, so inflated retries pull ordinary sessions into "possibly low-worth expensive sessions".

Suggested approach

src/bash-utils.ts already exports extractBashCommands(rawCommand): string[], which resolves wrappers — including rtk, added in #657. Both fixes can build on it rather than introduce new parsing.

1. Classify each Bash call once into read / verify / mutate / other from its extracted command names.
2. Count read toward READ_TOOL_NAMES's numerator in detectLowReadEditRatio.
3. Set lastVerifyStep only for verify in countRetries.

Step 3 changes the retry number for every existing user, so it is a judgment call rather than a clear bug fix: the question is where the line sits between verification and inspection. git diff after an edit is a reasonable check; ls is not. I would put test, build, typecheck, lint, and run commands in verify, and leave search and inspection in read, but that is the part worth settling before any code is written.

Scope of the evidence

Measured on one machine, Claude transcripts only, 7-day window, using an independent reimplementation of countRetries rather than the shipped code path. The split shows the pattern; it is not a restatement of what codeburn itself would report across all providers.

Environment

• codeburn at 8578199
• macOS 24.6.0, Node 26
• Providers present: Claude, Codex, Antigravity

Happy to send a PR for any subset of this once there is a call on the verify boundary in step 3. Per CONTRIBUTING I am checking on the approach before writing code.

[ozymandiashh]

Both defects reproduce here — I checked the two sites before forming a view rather than after. READ_TOOL_NAMES at src/optimize.ts:2190 has no Bash entry, and countRetries sets lastVerifyStep for any member of BASH_TOOLS. The measurement is the useful part of this report; thank you for doing it properly, including saying it was one machine, one provider, and a reimplementation rather than the shipped path.

The call on step 3

Your split is right, with one disagreement.

• verify — anything that establishes whether the edit works: test runners, build, typecheck, lint, formatters in check mode, and executing the program under edit.
• read — anything that looks at state without asserting correctness: rg, grep, cat, head, tail, find, ls, git log, git status.
• mutate — git commit, git checkout, mv, rm, mkdir, npm install.
• other — unrecognised.

git diff goes in read, not verify. You lean the other way, and the instinct is reasonable — it is a sensible thing to do after an edit. But it inspects your own change; it does not establish that the change works. Treating inspection as verification is precisely the conflation that produces defect 2, and putting git diff in verify re-imports a smaller version of the bug: edit → git diff → edit is refinement, not rework.

Two things worth settling that the issue does not cover

Mixed commands. One Bash call can carry several: rg foo && pytest is both a read and a verification. The precedence rule matters more than which answer you pick, but it needs to be explicit — my leaning is that verify wins for countRetries (a real check happened) while the read still counts toward the ratio, i.e. the two detectors consume the same classification differently rather than sharing one collapsed verdict.

PowerShell. BASH_TOOLS is ['Bash', 'BashTool', 'PowerShellTool'], so countRetries already treats PowerShell calls as verification — but extractBashCommands splits on POSIX separators, and PowerShellTool is the only mention of that shell anywhere in src/. Classifying Get-ChildItem or Invoke-Pester by POSIX rules will misfire silently. I would let unrecognised PowerShell fall to other, so those users keep today's behaviour rather than getting a confidently wrong new one — but it should be a stated decision with a test, not an accident.

Blast radius, for whoever writes it

Retries drop for every existing user, and the number feeds two surfaces beyond the finding itself: retryTax in src/usage-aggregator.ts:807, and the WORTH_IT_MIN_RETRIES = 3 gate in findLowWorthCandidates. So this is a re-baseline, not a bug fix that leaves history alone — it wants a changelog entry saying so, and it is worth checking whether the cached optimize results and the daily rollup need invalidating rather than assuming the TTL covers it.

On the PR

Yes please, and it is yours — you asked first and did the measurement. While deciding the boundary I had a prototype built against main to make sure the answers above were concrete rather than theoretical; happy to hand over the classification table and its tests if that saves you time, or to stay out of the way entirely. Say which.

One request either way: build on extractBashCommands as you suggested, and classify each call once into a shape both detectors read. Two independent classifiers that agree today drift apart later — this repo has been bitten by exactly that more than once recently.