Enforce the review-only contract instead of only asking for it - #125
Merged
Conversation
A --review-with pass is supposed to report findings and let the orchestrator apply them, but every reviewer was launched with full write access (--dangerously-skip-permissions / --sandbox danger-full-access / bypassPermissions) and the "do not touch the working tree" contract existed only as a sentence in $LOCAL_PROMPT. A reviewer that decides to fix things itself silently rewrites the caller's uncommitted work, and the loop never notices: it computes NEW_COMMITS/UNCOMMITTED, documents that they "should be zero" in review-only mode, and then never checks them. codex: the review-only invocation moves to --sandbox read-only, which makes the contract unbypassable at the OS level. The claim it replaces — that danger-full-access is required or "codex review produces no usable findings" — is empirically wrong: `codex --sandbox read-only review --base <ref>` reads the diff, tracked-file list, commit graph and base tree and returns normal severity-tagged findings, while a write inside the repo fails with `operation not permitted`. enhance-loop.md already ran codex read-only for exactly this reason, so this makes the two loops consistent rather than introducing a new idea. The reviewer-applies path keeps danger-full-access, which it needs. claude/agy/grok have no read-only mode, so they get a backstop instead: step 1 snapshots HEAD, the index, tracked content and untracked files (the same four artifacts enhance-loop.md uses, since a porcelain count alone misses an edit to an already-dirty tracked file or to a pre-existing untracked file), and the review-only branch of step 3 compares them, restores wholesale on violation, and warns. Unlike enhance-loop it KEEPS the findings — a reviewer's product is its findings list, which stays useful even when it wrongly applied them too, whereas an enhancer's product is the text it returns. The prompt line gains the reason rather than more emphasis: edits will be reverted, so applying a fix saves nothing and destroys in-progress state.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A
--review-withpass is supposed to report findings and let the orchestrator apply them. But every reviewer was launched with full write access —claude --dangerously-skip-permissions,codex --sandbox danger-full-access,agy --dangerously-skip-permissions,grok --permission-mode bypassPermissions— and the "do not touch the working tree" contract existed only as a sentence in$LOCAL_PROMPT.A reviewer that decides to fix things itself therefore silently rewrites the caller's uncommitted work. The loop doesn't catch it: it computes
NEW_COMMITS/UNCOMMITTED, documents that they "should be zero" in review-only mode, and then never checks them.This happened to me during a PortOS review — a review-only pass reverted uncommitted fixes twice before I noticed, and the process was still writing ~20 minutes after its wrapper reported exit 0.
codex →
--sandbox read-onlyon the review-only path. This makes the contract unbypassable at the OS level. The rationale it replaces — thatdanger-full-accessis required or "codex review produces no usable findings" — is empirically wrong. Verified in a scratch repo:codex --sandbox read-only review --base <ref>reads the diff, tracked-file list, commit graph and base tree, and returns normal severity-tagged findings.zsh:1: operation not permitted, file byte-identical, and codex reports the failure rather than continuing silently.lib/enhance-loop.mdalready ran codex--sandbox read-onlyfor exactly this reason, so this makes the two loops consistent rather than introducing a new idea. The reviewer-applies path keepsdanger-full-access, which it genuinely needs.claude/agy/grok → a detection-and-restore backstop. These have no read-only mode, so enforcement isn't available. Step 1 now snapshots HEAD, the index, tracked content and untracked files — the same four artifacts
enhance-loop.mduses, because agit status --porcelaincount alone misses an edit to an already-dirty tracked file (Mline unchanged) and an edit or deletion of a pre-existing untracked file (??line unchanged). The review-only branch of step 3 compares them, restores wholesale on violation, and warns.One deliberate divergence from
enhance-loop.md: this keeps the findings. A reviewer's product is its findings list, which stays useful even if it also wrongly applied them; an enhancer's product is the text it returns, so a violating enhancer is untrustworthy end-to-end. The orchestrator re-derives every fix itself regardless.The prompt gains a reason, not more emphasis. It already said "do NOT modify files"; shouting louder adds nothing. It now explains that edits will be detected and reverted, so applying a fix saves no step and destroys the caller's in-progress state.
Also updated the now-stale cross-reference in
enhance-loop.mdthat describeddanger-full-accessas "the review loop's" policy.Test plan
npm test— 173 pass, 0 fail (no behavior code changed; these are prompt/loop docs)null.fooderef and anaddthat subtracts), and a write attempt under the same policy is refused by the OS.lib/code-review-checklist.md:263(table cell, model-pinning prose, both rationale bullets,enhance-loop.mdcross-reference).commands/do/review.md:379mentionscodex review --baseonly to describe the review target, not a sandbox policy, so it needs no change..changelogs/v3.1.2.mdis a released record and is left alone./do:pr --review-withcycle against a live repo to watch the restore path fire, since triggering it requires a reviewer that actually violates the contract.