v1.70.1.0 fix: ship names the /document-release subagent at every decision point (tripwire + gate E2E) - #2700
Merged
Merged
Conversation
…ion point The v1.54.0.0 carve moved Step 18 (documentation sync) into ship/sections/pr-body.md and the Claude-host skeleton stopped saying "document-release" anywhere in the workflow body — the dispatch became invisible at exactly the moments an agent decides whether to open the section. Restore visibility at three touchpoints, all subagent-framed (never bare-slash-framed, which would invite an inline Skill invocation that bypasses the fresh-context subagent + JSON contract): - manifest trigger (renders into the section-index row AND the STOP pointer): "dispatching the /document-release subagent to sync docs (Step 18) and then creating or updating the PR/MR (Step 19)" - Step 17 handoff line names Step 18's dispatch explicitly - new hoisted doc-sync invariant beside the PR-title invariant: the dispatch itself is never skipped; only a failed subagent is non-blocking Pin it in carve-guards: 'the /document-release subagent' (all three touchpoints) + 'dispatches the /document-release subagent' (invariant) must stay in the skeleton; the carved imperative 'Dispatch /document-release as a subagent' must stay carved. Skeleton cap 91,600 → 92,300 (measured 91,764; trigger renders twice). Goldens regenerated for all three hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five substring/structure asserts across the carved section, the Claude skeleton's three touchpoints, the manifest trigger, and the codex/factory goldens (inlined Step 18 ordered before Step 19). Claude-golden asserts deliberately omitted: host-config.test.ts already enforces golden == generated byte-for-byte. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…agent
New skill-e2e-ship-docsync: a live agent gets the sliced Step 17→19 tail
of the generated ship skeleton in a bare-remote git fixture (Steps 0-16
"done"), under a fake HOME so the STOP pointer and the Step 18 subagent
prompt resolve to planted copies, with a stub document-release skill that
returns the empty-result JSON contract. Hard assert: an Agent/Task
tool-call matching /document-release/i exists in result.toolCalls and
precedes any `gh pr create`. Neutral prompt (no STOP-Read priming, no
document-release mention — the prompt echoes into the transcript, so
asserts read toolCalls only).
Hardening from review: throw-on-marker-drift fixture slice; per-test
GSTACK_HOME + .redact-prepush-prompted marker (routes Step 17's
credential guard to its silent branch — the hermetic GSTACK_HOME pin
defeats a HOME-only override); 480s/540s timeouts (nested subagent adds
wall clock the 300s sibling never carried); 'timeout' accepted in
exitReason only because the dispatch assert is independently hard;
whole-file describeE2ETier('gate') composed with diff selection (keeps
the file out of the periodic shard census, which sits at its ceiling,
and under the hard tier-alignment invariant).
Registered as 'ship-docsync' in E2E_TOUCHFILES + E2E_TIERS (gate) in the
same commit — touchfiles.test.ts rejects either half landing first.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…items The SHIPPED entry still described the deleted Step 8.5 post-PR cat-delegation design from v0.8.4; replace with the current Step 18 subagent design and its test pins. Add the three P3 items deferred from the v1.69 plan review: dispatch receipt enforcement, land-and-deploy→canary dispatch-pin pattern, and the periodic shard-census boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Testing-specialist findings, all mechanical: (1) pin the E2E fixture's git branch (-b main / init.defaultBranch=main) and assert every setup command's exit status so operator git config can't silently corrupt a paid run; (2) tighten the dispatch matcher to Step 18-prompt-specific markers (document-release/SKILL.md | executing the /document-release workflow) so a subagent merely quoting section text can't false-pass the regression assert (verified against recorded burn-in transcripts); (3) replace the subsumed carve-guards anchor with three non-overlapping per-touchpoint anchors (gerund/imperative/3rd-person) so each touchpoint is independently enforced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five informational findings: TODOS shard-census arithmetic corrected (census is 67 with one free ungated slot; the SECOND ungated file trips the floor) and version pointer fixed (v0.18.2.0, not v0.18.1.0); the free tripwire now pins the two dispatch-matcher marker strings so a pr-body prompt reword fails the free suite instead of surfacing as a paid-tier mystery; the E2E matcher gains a section-paste exclusion (scaffold strings disqualify) — verified against all recorded runs; the E2E header documents the tierless test:evals invisibility tradeoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin the E2E matcher's two EXCLUSION markers in the free tripwire (an unpinned 'Parent processing:' reword would silently deaden the section-paste guard while every test stayed green); add an ordering pin (the hoisted doc-sync invariant must sit above the pr-body STOP pointer — presence-only anchors can't catch drift below it); plant a third cwd-relative pr-body copy inside the fixture repo, gitignored so the agent never tries to commit test scaffolding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three factual corrections the Step 18 doc subagent caught in the fresh v1.70.1.0 entry: 5 tripwire tests (not 6), cost floor $0.63 per the cited eval store (not $0.59), and the visibility claim scoped to decision points (the re-run checklist mention survived the carve). Plus the E2E header's stale pending-burn-in note replaced with the observed numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
😎 This pull request was merged. |
E2E Evals: ✅ PASS29/29 tests passed | $2.83 total cost
ubicloud-standard-8 runners (Docker: pre-baked toolchain + deps) | wall clock ≈ slowest suite |
… runners The 50ms-sleep test blew the 20s budget on BOTH bun retry attempts on PR #2700's windows-latest runner (run 32989821401) — sustained AV/runner pressure, not just the documented cold-start. Same flake passed-on-rerun on the prompt-token-load-reduction branch yesterday. Budget only; every assertion still checks exact output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…kip tripwire The evals.yml matrix is hand-enumerated and the Run step never exported EVALS_TIER, so the new whole-file-gated ship-docsync E2E would have self-skipped even with a row — a hollow green one layer deeper than the documented rehomed-monolith incident. Add the e2e-ship-docsync row with a row-level `tier: gate` property, exported as EVALS_TIER by the Run step (empty = unset for every existing row: all readers are `=== '<tier>'` or truthiness). New free tripwire test/evals-workflow-matrix.test.ts ratchets the class: matrix files must exist; gate-hosting files must have a row; whole-file-gated matrix files must carry a matching row tier; and the burn-down lists enforce their own cleanup. It enumerates the PRE-EXISTING holes found while wiring this (8 gate-hosting files with no row; codex/gemini rows running zero tests; the pty-plan-smoke row hollow since its files adopted describeE2ETier) — tracked in TODOS as the CI gate-lane hollow-coverage burn-down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/shiphas dispatched/document-releaseas Step 18 since v0.18.2.0, but the v1.54.0.0 carve moved the step into an on-demand section and the always-loaded Claude-host skeleton stopped naming it at any decision point — the wiring survived, the visibility didn't, and nothing tested the handoff.Fix — Step 18 visibility restored (subagent-framed at three touchpoints)
ship/sections/manifest.jsontrigger (renders into the section index AND the STOP pointer), the Step 17 handoff line, and a new hoisted doc-sync invariant beside the PR-title invariant all name "the /document-release subagent". Subagent-framed wording is deliberate: a bare slash-command mention invites an inline Skill invocation that bypasses the fresh-context subagent + JSON contract.Tests — the handoff can no longer silently regress
test/ship-document-release-dispatch.test.ts(free tripwire, 5 tests): carved Step 18 contract, three skeleton touchpoints, invariant-above-STOP ordering, the E2E matcher's four marker strings (lockstep-pinned), codex/factory goldens' inlined Step 18 → Step 19 ordering.test/skill-e2e-ship-docsync.test.ts(ship-docsync, gate tier): live agent runs the sliced Step 17→19 tail in a hermetic bare-remote fixture; hard assert that an Agent dispatch matching the Step 18 prompt markers appears inresult.toolCallsBEFORE anygh pr create. Fixture pins its git branch against operator config, asserts every setup command, fails loud on step-marker drift, and neutralizes the Step 17 credential-guard question branch (GSTACK_HOME+ marker). Registered inE2E_TOUCHFILES/E2E_TIERSwith a whole-file gate self-gate composed with diff selection (keeps the file out of the periodic shard census, which is one ungated file from its ceiling).Review hardening (14 findings absorbed in-flight)
test:evalsinvisibility tradeoff documented.Docs
Test Coverage
Tests: 554 → 556 (+2 new)
Pre-Landing Review
8 issues (2 critical-tagged test-quality, 6 informational) — 8 auto-fixed, 0 asked, 0 skipped. Sources: checklist pass (0), testing specialist (3), maintainability (0), red team (5). Fix loop converged in 2 cycles; every fix round re-verified with targeted free tests plus a live E2E run. Cross-model note: Codex passes did NOT run (OpenAI account out of API credits) — that coverage is missing on this ship, not clean.
Design Review
No frontend files changed — design review skipped.
Eval Results
ship-docsynclifetime: 9/9 live runs dispatch-clean ($0.63-1.04, 234-319s each), including runs after every matcher/fixture hardening round.Scope Drift
Scope Check: CLEAN
Intent: restore ship→document-release Step 18 visibility and pin it with a unit test + E2E.
Delivered: exactly that, plus the review-driven hardening of those same tests.
Plan Completion
48 plan items: 46 DONE, 1 CHANGED (byte cap 92,300 vs plan's "~92,100" — measured value recorded), 1 completed-at-ship-time (CHANGELOG framing note, applied in the v1.70.1.0 entry). 0 NOT DONE, 0 deferred. Commit-split legality followed exactly ({templates+registry+goldens} → {unit test} → {E2E+registration} → docs).
Verification Results
Skipped: no dev server (CLI/skill repo — the plan's verification section is test commands, all executed: free guard set green on the shipped tree, burn-in 5/5, gate lane 61/0).
Known environmental caveat: this cloud sandbox actively re-clamps /tmp to mode 700 root:root, which intermittently breaks /tmp-fixture tests (bun
access(2)traversal). Every observed failure class reproduces on origin/main worktrees (verified independently three times today with receipts) and touches zero files in this diff. The required secretlessfree-testsCI check on this PR is the authoritative full-suite signal.TODOS
No TODO items completed in this PR.
Documentation
/document-releaseaudited all project docs against the v1.70.1.0 diff (14 files, 8 commits). No doc files required updates: README, ARCHITECTURE, CONTRIBUTING, CLAUDE.md, and AGENTS.md remain accurate; CHANGELOG (v1.70.1.0), VERSION, package.json, and TODOS.md were already updated at ship time. CHANGELOG entry scores 3/3 on the sell test. Coverage map: no new public surface shipped with zero documentation coverage; no architecture diagram drift.Documentation Debt
Found by the doc review (Claude subagent; Codex unavailable). The three CHANGELOG nits it caught were fixed in-branch (bb56dfe); the rest are pre-existing docs outside this diff, deferred:
/document-releasesection): "After /ship creates the PR" framing predates the Step 18 pre-PR dispatch; a one-line reword would match the integrated flow.$3.85/run" / "$4/run max" E2E cost figures predate the new gate E2E (+$0.63-1.04/run); needs a re-measured total.bun run test:evals) is documented only in the test file header.~/.gstack-dev/; current is~/.gstack/projects/<slug>/evals/with~/.gstack-devas legacy fallback.Test plan
ship-docsyncgate E2E: 9/9 live dispatch-clean🤖 Generated with Claude Code
Open workspace in Conductor