Skip to content

Enforce the review-only contract instead of only asking for it - #125

Merged
atomantic merged 1 commit into
mainfrom
enforce-review-readonly
Jul 23, 2026
Merged

Enforce the review-only contract instead of only asking for it#125
atomantic merged 1 commit into
mainfrom
enforce-review-readonly

Conversation

@atomantic

Copy link
Copy Markdown
Owner

Summary

A --review-with pass is supposed to report findings and let the orchestrator apply them. But every reviewer was launched with full write access — claude --dangerously-skip-permissions, codex --sandbox danger-full-access, agy --dangerously-skip-permissions, grok --permission-mode bypassPermissions — and the "do not touch the working tree" contract existed only as a sentence in $LOCAL_PROMPT.

A reviewer that decides to fix things itself therefore silently rewrites the caller's uncommitted work. The loop doesn't catch it: it computes NEW_COMMITS/UNCOMMITTED, documents that they "should be zero" in review-only mode, and then never checks them.

This happened to me during a PortOS review — a review-only pass reverted uncommitted fixes twice before I noticed, and the process was still writing ~20 minutes after its wrapper reported exit 0.

codex → --sandbox read-only on the review-only path. This makes the contract unbypassable at the OS level. The rationale it replaces — that danger-full-access is required or "codex review produces no usable findings" — is empirically wrong. Verified in a scratch repo:

  • codex --sandbox read-only review --base <ref> reads the diff, tracked-file list, commit graph and base tree, and returns normal severity-tagged findings.
  • A write inside that repo under the same sandbox fails: zsh:1: operation not permitted, file byte-identical, and codex reports the failure rather than continuing silently.

lib/enhance-loop.md already ran codex --sandbox read-only for exactly this reason, so this makes the two loops consistent rather than introducing a new idea. The reviewer-applies path keeps danger-full-access, which it genuinely needs.

claude/agy/grok → a detection-and-restore backstop. These have no read-only mode, so enforcement isn't available. Step 1 now snapshots HEAD, the index, tracked content and untracked files — the same four artifacts enhance-loop.md uses, because a git status --porcelain count alone misses an edit to an already-dirty tracked file ( M line unchanged) and an edit or deletion of a pre-existing untracked file (?? line unchanged). The review-only branch of step 3 compares them, restores wholesale on violation, and warns.

One deliberate divergence from enhance-loop.md: this keeps the findings. A reviewer's product is its findings list, which stays useful even if it also wrongly applied them; an enhancer's product is the text it returns, so a violating enhancer is untrustworthy end-to-end. The orchestrator re-derives every fix itself regardless.

The prompt gains a reason, not more emphasis. It already said "do NOT modify files"; shouting louder adds nothing. It now explains that edits will be detected and reverted, so applying a fix saves no step and destroys the caller's in-progress state.

Also updated the now-stale cross-reference in enhance-loop.md that described danger-full-access as "the review loop's" policy.

Test plan

  • npm test — 173 pass, 0 fail (no behavior code changed; these are prompt/loop docs)
  • Sandbox behavior verified empirically in a throwaway git repo, both directions: read-only review produces correct findings (it caught a seeded null.foo deref and an add that subtracts), and a write attempt under the same policy is refused by the OS.
  • Swept every copy of the codex invocation string per lib/code-review-checklist.md:263 (table cell, model-pinning prose, both rationale bullets, enhance-loop.md cross-reference). commands/do/review.md:379 mentions codex review --base only to describe the review target, not a sandbox policy, so it needs no change. .changelogs/v3.1.2.md is a released record and is left alone.
  • Not exercised end-to-end: I did not run a full /do:pr --review-with cycle against a live repo to watch the restore path fire, since triggering it requires a reviewer that actually violates the contract.

A --review-with pass is supposed to report findings and let the orchestrator
apply them, but every reviewer was launched with full write access
(--dangerously-skip-permissions / --sandbox danger-full-access /
bypassPermissions) and the "do not touch the working tree" contract existed only
as a sentence in $LOCAL_PROMPT. A reviewer that decides to fix things itself
silently rewrites the caller's uncommitted work, and the loop never notices: it
computes NEW_COMMITS/UNCOMMITTED, documents that they "should be zero" in
review-only mode, and then never checks them.

codex: the review-only invocation moves to --sandbox read-only, which makes the
contract unbypassable at the OS level. The claim it replaces — that
danger-full-access is required or "codex review produces no usable findings" —
is empirically wrong: `codex --sandbox read-only review --base <ref>` reads the
diff, tracked-file list, commit graph and base tree and returns normal
severity-tagged findings, while a write inside the repo fails with `operation
not permitted`. enhance-loop.md already ran codex read-only for exactly this
reason, so this makes the two loops consistent rather than introducing a new
idea. The reviewer-applies path keeps danger-full-access, which it needs.

claude/agy/grok have no read-only mode, so they get a backstop instead: step 1
snapshots HEAD, the index, tracked content and untracked files (the same four
artifacts enhance-loop.md uses, since a porcelain count alone misses an edit to
an already-dirty tracked file or to a pre-existing untracked file), and the
review-only branch of step 3 compares them, restores wholesale on violation, and
warns. Unlike enhance-loop it KEEPS the findings — a reviewer's product is its
findings list, which stays useful even when it wrongly applied them too, whereas
an enhancer's product is the text it returns.

The prompt line gains the reason rather than more emphasis: edits will be
reverted, so applying a fix saves nothing and destroys in-progress state.
@atomantic
atomantic merged commit d0634a0 into main Jul 23, 2026
4 checks passed
@atomantic
atomantic deleted the enforce-review-readonly branch July 23, 2026 16:28
@atomantic atomantic mentioned this pull request Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant