diff --git a/README.md b/README.md index e1a9342..a1e1260 100644 --- a/README.md +++ b/README.md @@ -137,7 +137,8 @@ the machine-readable shape is in **Refuse: no safe path exists, so nothing runs.** A genuine table rewrite needs the copy-and-swap backend (a later phase); the dry run exits 2 so CI -can gate on it without parsing JSON. The exit-code gate stops refusals only — +can gate on it without parsing JSON (the full four-code ladder is under +[Exit codes](#exit-codes)). The exit-code gate stops refusals only — a destructive-but-executable change (`DROP COLUMN`) warns and exits 0, so a gate that must stop drops checks `.statements[].destructive` in the `--json` report. Watch it in @@ -184,7 +185,7 @@ ever commits a change — every other command is read-only or fully offline. | `capabilities` | none | Print the embedded support matrix as a compact table, or as the versioned automation contract with `--json` | | `fmt` | none | Canonicalize a schema file — parser only | | `lint` | none | Flag patterns the engine would refuse, rewrite, or gate, from the DDL text alone | -| `suggest` | none | Map risky DDL to the safer native form the engine would run, with typed caveats; advisory, always exits 0 | +| `suggest` | none | Map risky DDL to the safer native form the engine would run, with typed caveats; advisory, exits 0 on any script it can parse | The offline commands have no connection flags at all, so they cannot be pointed at a database by accident. @@ -196,6 +197,49 @@ exact boundary. Why safer sequences run without a wrapping transaction (PostgreSQL forbids it for the online forms) and what each documented partial state means is [docs/execution-model.md](docs/execution-model.md). +## Exit codes + +The process status is the contract a shell gate reads without parsing JSON. +Each code answers one question: did anything commit, and does the engine +vouch for it as online-safe. + +| Exit | Meaning | Anything committed? | Online-safe? | +|---|---|---|---| +| 0 | Executed through an online-safe path; or a dry run, `diff`, or `pull` found nothing to refuse | yes (dry run, `diff`, `pull`: nothing runs) | yes | +| 1 | Failed: a PostgreSQL error after execution started (rolled back), or an operational or usage error before it — bad flags, an unreachable database, a mismatched `--force` acknowledgement | no; a safer sequence that stopped mid-flight keeps its committed prefix, named in the verdict's `executed_sql` | not applicable | +| 2 | Refused: no online-safe path, and the verdict names the typed `reason` (and `class`) automation switches on | no | not applicable | +| 3 | Committed through the accepted-blocking passthrough — the operator explicitly accepted a blocking form, and the engine ran it under bounded budgets without vouching for online safety | yes | no | + +Three rules follow from the table, and one note for embedders: + +- **Gate on non-zero.** `pg-sprite migrate … || exit 1` is fail-closed for + every code above. Allow 3 explicitly only where a maintenance-window + blocking change is intended; exit 0 is never borrowed for it. +- **Refusal is one code, whichever command produced it.** `migrate`, its dry + run, `diff`, and `pull` all exit 2 on a refusal; exit 3 can come only from + `migrate`, because no other command executes DDL. The offline `lint` exits + 1 when a script has error-severity findings (warnings alone exit 0), and + `suggest` exits 0 on any script it can parse — findings never gate; only an + unreadable or unparsable script exits 1. +- **Exit 2 means nothing *committed*, not nothing ran.** An optimistic + attempt that exceeded its statement budget did run — PostgreSQL cancelled + it and transactional DDL rolled it back — and still exits 2, because the + refusal is a routing answer (the change needs a different strategy). +- **Library callers get the same facts typed, not as a status.** An + orchestrator that imports `pkg/executor` branches on the verdict's + `outcome`, `errors.As` to the executor's typed errors, and + `executor.OutcomeCode` — the exit code is the CLI's rendering of those + facts, not a surface the library exposes. The typed contract is in + [docs/execution-model.md](docs/execution-model.md#how-a-failure-is-reported). + +Exit 3 is reserved today: `executor.ExecuteAcceptedBlocking` ships as a +library primitive, and no `migrate` flag reaches it yet, so no CLI +invocation currently produces it. Why the code is non-zero, and why a +statement-budget cancellation on that path is exit 1 rather than 2, is in +[docs/lock-budgeted-passthrough.md](docs/lock-budgeted-passthrough.md#exit-codes); +every code's JSON shape is in +[docs/cli-output-examples.md](docs/cli-output-examples.md#exit-codes). + ## Demo A runnable tour of the CLI against a local PostgreSQL (Docker required): diff --git a/docs/capabilities.md b/docs/capabilities.md index cc479ce..77189b3 100644 --- a/docs/capabilities.md +++ b/docs/capabilities.md @@ -82,7 +82,11 @@ Two consequences follow, and they explain most of this page: when the change is executable through an online-safe path; a refusal exits 2 with a typed reason. CI can gate on the exit code alone. That contract is only worth something if pg-sprite never executes what it cannot vouch for — see - [Why typed refusal, not passthrough](#why-typed-refusal-not-passthrough). + [Why typed refusal, not passthrough](#why-typed-refusal-not-passthrough). The one + sanctioned exception keeps the gate honest: a blocking form the operator explicitly + accepts runs under bounded budgets and exits 3, never 0, so a gate that fails on any + non-zero status stays fail-closed + ([the full ladder](cli-output-examples.md#exit-codes)). ## The support model: three tiers diff --git a/docs/cli-output-examples.md b/docs/cli-output-examples.md index cc33a98..fb35a4d 100644 --- a/docs/cli-output-examples.md +++ b/docs/cli-output-examples.md @@ -19,9 +19,11 @@ CREATE TABLE users (id bigint PRIMARY KEY, email text); CREATE TABLE events (id bigint, created date) PARTITION BY RANGE (created); ``` -`migrate` exit codes follow the dry-run contract (0 = executable, 2 = refused — -including a target table that does not exist, so a typo'd name cannot gate green) -defined with the [diagnostic codes](postgres-online-ddl-reference.md#dry-run-diagnostic-codes); +`migrate` exit codes follow the four-code ladder under [Exit codes](#exit-codes): +the dry run exits 0 when every statement is executable and 2 when any would be +refused — including a target table that does not exist, so a typo'd name cannot +gate green — as defined with the +[diagnostic codes](postgres-online-ddl-reference.md#dry-run-diagnostic-codes); CI can gate on the exit code without parsing JSON. The gate is refusals only: a destructive-but-executable change (`DROP COLUMN`) warns and exits 0 — a gate that must stop drops checks `.statements[].destructive` in the JSON report. @@ -37,6 +39,7 @@ dry run. Statement kinds `migrate` does not support (`DROP INDEX`, a verdict, not a plan report — and exit 2. The JSON report schema is [plan-report.md](plan-report.md). +- [Exit codes](#exit-codes) - [Codes used in these examples](#codes-used-in-these-examples) - [Refusal reasons](#refusal-reasons) - [Migrate](#migrate) @@ -56,6 +59,31 @@ a verdict, not a plan report — and exit 2. The JSON report schema is - [Capabilities](#capabilities) - [Embedded support matrix with the binary version — exit 0](#embedded-support-matrix-with-the-binary-version--exit-0) +## Exit codes + +The process status is the part of the contract a shell reads without JSON. Each +code answers whether anything committed and whether the engine vouches for it +as online-safe; the section headings below name the code each example exits with. +The table is restated from the [README](../README.md#exit-codes), which is the +canonical statement of the ladder; a wording fix lands there first. + +| Exit | Meaning | Anything committed? | Online-safe? | +|---|---|---|---| +| 0 | Executed through an online-safe path; or a dry run, `diff`, or `pull` found nothing to refuse | yes (dry run, `diff`, `pull`: nothing runs) | yes | +| 1 | Failed: a PostgreSQL error after execution started (rolled back), or an operational or usage error before it — bad flags, an unreachable database, a mismatched `--force` acknowledgement | no; a safer sequence that stopped mid-flight keeps its committed prefix, named in `executed_sql` | not applicable | +| 2 | Refused: no online-safe path; the verdict names the typed `reason` and `class` | no | not applicable | +| 3 | Committed through the accepted-blocking passthrough, under bounded budgets, without an online-safety guarantee | yes | no | + +Refusals from every command — `migrate`, its dry run, `diff`, and `pull` — share +exit 2, so a gate branches on the status without caring which subcommand produced +it; exit 3 is `migrate`'s alone, because no other command executes DDL, and no +`migrate` flag reaches the passthrough yet, so no CLI invocation produces it today. +Exit 2 means nothing *committed*: an optimistic attempt that exceeded its statement +budget did run and was rolled back, and still exits 2. A gate that treats every +non-zero status as failure is fail-closed for all three non-zero codes; a caller +that deliberately permits the passthrough allows 3 explicitly. The rationale is in +[lock-budgeted-passthrough.md](lock-budgeted-passthrough.md#exit-codes). + ## Codes used in these examples Every diagnostic in the examples below carries one of these typed codes. Each @@ -209,8 +237,10 @@ $ pg-sprite migrate --alter 'ALTER TABLE users ADD CONSTRAINT users_email_key UN ``` An operator-accepted blocking refusal has a distinct marked outcome and exit -3; exit 0 remains exclusive to online-safe execution. Nothing produces this -outcome until `--accept-blocking` lands in a later change: +3; exit 0 remains exclusive to online-safe execution. The executor primitive +(`executor.ExecuteAcceptedBlocking`) and the verdict shape are shipped; the +`--accept-blocking` flag that reaches them from `migrate` is not, so the +command below shows the contract, not a runnable invocation yet: ```text executed without online safety (accepted blocking refusal) diff --git a/docs/execution-model.md b/docs/execution-model.md index 764905c..365e36b 100644 --- a/docs/execution-model.md +++ b/docs/execution-model.md @@ -172,8 +172,11 @@ Automation branches on `code` — the stable outcome vocabulary — never on prose, which is free to change. Reading the three surfaces: - **Exit codes** separate the cases: an execution failure exits 1 (as here); - a refusal — where nothing was ever attempted — exits 2. See - [cli-output-examples.md](cli-output-examples.md). + a refusal — where nothing committed, even if an optimistic attempt ran and + was rolled back — exits 2; a change the operator explicitly accepted as + blocking, which committed without an online-safety guarantee, exits 3. + Exit 0 is exclusive to online-safe execution. The full ladder is in + [cli-output-examples.md](cli-output-examples.md#exit-codes). - **Library callers** get the same facts typed: `errors.As` to `*executor.SequenceStepError` (`Step`, `Total`, `SQL`, `Kind` — the execution class, the `(brief)` in the line below — and the underlying diff --git a/docs/lock-budgeted-passthrough.md b/docs/lock-budgeted-passthrough.md index a61af2a..55a8dba 100644 --- a/docs/lock-budgeted-passthrough.md +++ b/docs/lock-budgeted-passthrough.md @@ -34,7 +34,7 @@ make the behavior online-safe. The current [capabilities matrix](capabilities.md#why-typed-refusal-not-passthrough) makes refusal valuable: the engine says what it cannot vouch for instead of silently falling back to a blocking form. The root README reserves exit code 0 for a schema change that ran through -an online-safe path. Exit code 2 means refused and nothing ran; exit code 1 means execution +an online-safe path. Exit code 2 means refused and nothing committed; exit code 1 means execution failed. Those meanings remain useful and remain unchanged. The new path does not weaken that contract. It splits “the engine refused to call this @@ -301,6 +301,8 @@ lookup that resolves the accepted table, which runs only on the execution path. retains disposition `refuse`, reason, class, cause, and guidance. Eligibility is permission to request a later execution path, not a reclassification as online-safe. +### Exit codes + Exit code 3 means the statement committed through this marked path. Exit code 0 remains online-safe success, 1 remains execution failure, and 2 remains refusal with nothing committed. Choosing exit 0 plus a marked outcome lost because shell CI would have to parse JSON or prose @@ -308,10 +310,14 @@ to distinguish accepted blocking execution from the product's online-safe succes A distinct code is intentionally non-zero: generic CI fails closed, while a caller that deliberately permits this path can allow 3 explicitly. -The full ladder once this ships is the binary's process contract, with the question each code -answers. Refusals from every command — `migrate`, `diff`, a dry run, and `pull` — share exit -2, so a CI author gates on the status without caring which subcommand produced it; exit 3 is -produced by `migrate` alone, because no other command executes DDL. +The full ladder is the binary's process contract, with the question each code answers. Its +canonical statement is the root README's [Exit codes](../README.md#exit-codes) section; it is +restated at the head of [cli-output-examples.md](cli-output-examples.md#exit-codes) and, with +this path's cells spelled out, in the table below — all three pinned by test to the constants in +`pkg/verdict`. This section holds the reasoning behind the two cells that are not obvious from +the table. Refusals from every command — `migrate`, `diff`, a dry run, and +`pull` — share exit 2, so a CI author gates on the status without caring which subcommand +produced it; exit 3 is produced by `migrate` alone, because no other command executes DDL. | Exit | Meaning | Anything committed? | Online-safe? | |------|---------|---------------------|--------------| @@ -481,11 +487,13 @@ Sequence implementation as follows: its per-statement branches: it invokes `migrate` without `--accept-blocking`, so it cannot produce a 3, and an unexpected 3 falls through every branch to `FAIL` and skips the `psql` re-apply, which is the safe direction for a statement that already committed. - *(done in part: the outcome, the retained identity and budget fields, the exit-3 constant - and sentinel, plan-report eligibility, and the `cli-output-examples.md` example shipped; - the exit-code contract paragraph at that page's head, the five surface edits, and the - exit-2 gloss tightening move to step 4 with the flag, because no command can produce exit 3 - until `--accept-blocking` exists.)* + *(done: the outcome, the retained identity and budget fields, the exit-3 constant and + sentinel, plan-report eligibility, and the `cli-output-examples.md` example shipped; the + four-code ladder is stated at that page's head and in the root README, pinned by test to + the `pkg/verdict` constants, with exit 3 marked as reserved until the flag exists; the five + surface edits and the exit-2 gloss tightening shipped with it. The T2 row's "exit 2" cell + in [capabilities.md](capabilities.md) stays as written until step 4, because it describes + what a caller sees today.)* 4. Add `--accept-blocking` to imperative `migrate`, reject its combination with `--force`, emit the pre-execution audit record, and add demo assertions for eligibility, success, lock-budget refusal, statement-budget failure, mismatched acknowledgement, and exit codes. diff --git a/docs/low-level-design.md b/docs/low-level-design.md index 61ef8c7..8bef873 100644 --- a/docs/low-level-design.md +++ b/docs/low-level-design.md @@ -820,7 +820,10 @@ At the library seam, each executor outcome maps to a stable string code embedding `pkg/executor` branches on one vocabulary; the CLI's verdict JSON carries the same codes — an execution failure ends in a `failed` verdict (exit 1, distinct from the refusal exit 2) with the code, the failed step, and the committed prefix in `executed_sql`, so -automation can distinguish nothing-committed from partial state left behind. Native execution +automation can distinguish nothing-committed from partial state left behind; an +operator-accepted blocking change that committed ends in an `executed-without-online-safety` +verdict (exit 3), the one case that is committed but not vouched for as online-safe +([the ladder](cli-output-examples.md#exit-codes)). Native execution exposes a caller-owned `progress.Tracker`. Embedders run a blocking executor call in their own bounded task and poll `Tracker.Progress(ctx)`: sequence position and elapsed time come from in-process state, while an active concurrent index build is read on demand from diff --git a/docs/optimistic-attempt.md b/docs/optimistic-attempt.md index 7c08ac8..028fe44 100644 --- a/docs/optimistic-attempt.md +++ b/docs/optimistic-attempt.md @@ -220,7 +220,13 @@ assertion separating "instant" from "copy" ### Exit inventory -| Exit | Outcome | Typed as | DDL executed? | +Every way the lane-B attempt can end, numbered as cases for the prose that +follows. The case numbers are not process exit codes: the process status for +each row is the one the [README's exit-code ladder](../README.md#exit-codes) +assigns to its outcome — 0 for a success verdict, 2 for a refusal, 1 for a +failed sequence. + +| Case | Outcome | Typed as | DDL executed? | |---|---|---|---| | 1 | Online idiom / substituted sequence completed | Success verdict | Yes — online by proof | | 2 | Bounded attempt completed within budget | Success verdict | Yes — it was catalog-only | diff --git a/docs/postgres-online-ddl-reference.md b/docs/postgres-online-ddl-reference.md index afd5700..09d95b1 100644 --- a/docs/postgres-online-ddl-reference.md +++ b/docs/postgres-online-ddl-reference.md @@ -298,7 +298,12 @@ table), so automation that needs the cause reads the typed `reason` and disposition from the `--json` report. A destructive-but-executable change (`DROP COLUMN`, `DROP TABLE`) is **not** a refusal: it warns and exits 0. A gate that must stop drops checks `.statements[].destructive` in the JSON -report. +report. A real run adds the two codes that need something to have executed — +**1** when execution failed (a dry run exits 1 only for an operational error +such as an unreachable database, because it executes nothing) and **3** when +the operator explicitly accepted a blocking form and it committed without an +online-safety guarantee; the full ladder is in +[cli-output-examples.md](cli-output-examples.md#exit-codes). ### `metadata-only` diff --git a/docs/refusal-classes.md b/docs/refusal-classes.md index 544bfb8..aa22b4a 100644 --- a/docs/refusal-classes.md +++ b/docs/refusal-classes.md @@ -6,7 +6,7 @@ answers the next question: wait for engine capability, hand the work to its owne named safer idiom, or change the run environment. Every refused statement has an `outcome`, a typed `reason`, a typed `class`, explanatory `detail`, and, -where one exists, a `safer_idiom`. Exit code 2 means nothing ran. That is enough to explain a +where one exists, a `safer_idiom`. Exit code 2 means nothing committed. That is enough to explain a single refusal, but not enough to route it: `unsupported-statement` alone covers a data backfill, an imperative `CREATE TABLE`, a permanently unsafe `CREATE INDEX IF NOT EXISTS`, and an admitted `ALTER TABLE` operation for which the planner has no route. Those are not @@ -127,7 +127,7 @@ giving consumers one closed routing vocabulary. Encoding the class into `reason` lost because it would rename every existing token, multiply otherwise identical reasons, and make consumers parse a compound convention. Assigning a new exit code to each class lost because exit code 2 has one valuable process-level meaning: -refused, nothing ran. Shell status is too small a surface for the reason, class, and owner +refused, nothing committed. Shell status is too small a surface for the reason, class, and owner axes, and changing it would break the existing gate. For example, an `UPDATE` backfill changes only by additive fields: diff --git a/internal/cli/migrate.go b/internal/cli/migrate.go index 5db5b89..d4bb4bf 100644 --- a/internal/cli/migrate.go +++ b/internal/cli/migrate.go @@ -102,9 +102,13 @@ func (c *MigrateCmd) retryPolicy() executor.RetryPolicy { return executor.RetryPolicy{MaxAttempts: c.LockAttempts, InitialBackoff: c.LockBackoff, MaxBackoff: c.LockBackoffMax} } -// emit prints the verdict in the selected format and returns ErrRefused for -// refusals so the exit code distinguishes them from operational errors. The -// JSON contract stays plain; the human rendering styles its labels. +// emit prints the verdict in the selected format and returns the sentinel +// the entry point maps to the outcome's exit code: ErrRefused for a refusal +// and ErrAcceptedBlocking for a commit without an online-safety guarantee, +// so neither can be read as online-safe exit 0 or as an operational error. A +// failed verdict returns nil here; its caller returns the run error, which +// is what exits 1. The JSON contract stays plain; the human rendering styles +// its labels. func (c *MigrateCmd) emit(out io.Writer, v verdict.Verdict) error { if c.JSON { text, err := v.JSON() @@ -117,8 +121,12 @@ func (c *MigrateCmd) emit(out io.Writer, v verdict.Verdict) error { } else if err := writeVerdictText(out, c.palette(out), v); err != nil { return err } - if v.Outcome == verdict.OutcomeRefused { + switch v.Outcome { + case verdict.OutcomeRefused: return verdict.ErrRefused + case verdict.OutcomeExecutedWithoutOnlineSafety: + return verdict.ErrAcceptedBlocking + default: + return nil } - return nil } diff --git a/internal/cli/migrate_test.go b/internal/cli/migrate_test.go index a7eb5ae..283138c 100644 --- a/internal/cli/migrate_test.go +++ b/internal/cli/migrate_test.go @@ -1,6 +1,8 @@ package cli import ( + "bytes" + "fmt" "testing" "time" @@ -10,6 +12,7 @@ import ( "github.com/block/pg-sprite/pkg/executor" "github.com/block/pg-sprite/pkg/migrate" + "github.com/block/pg-sprite/pkg/verdict" ) // parseMigrate runs args through the real command grammar so these tests @@ -75,6 +78,37 @@ func TestMigrateDefaultsMatchLibraryDefaults(t *testing.T) { assert.Equal(t, want.Retry, got.Retry) } +// The exit-code ladder is a constant↔behavior contract, not only a +// constant↔docs one: each verdict outcome must reach the entry point as the +// sentinel that maps to its documented code. The two committing outcomes +// are the pair that must never be confused — an accepted-blocking commit +// returning nil would exit 0 and claim online safety it does not have. A +// failed verdict returns nil from emit by design; its caller returns the run +// error, which exits 1. +func TestEmitReturnsTheSentinelForEachOutcome(t *testing.T) { + for _, tc := range []struct { + outcome verdict.Outcome + want error + }{ + {outcome: verdict.OutcomeExecuted, want: nil}, + {outcome: verdict.OutcomeExecutedWithoutOnlineSafety, want: verdict.ErrAcceptedBlocking}, + {outcome: verdict.OutcomeRefused, want: verdict.ErrRefused}, + {outcome: verdict.OutcomeFailed, want: nil}, + } { + t.Run(string(tc.outcome), func(t *testing.T) { + c := parseMigrate(t, "--json") + var out bytes.Buffer + got := c.emit(&out, verdict.Verdict{Outcome: tc.outcome, Statement: "ALTER TABLE t ADD COLUMN c int"}) + if tc.want == nil { + assert.NoError(t, got) + } else { + assert.ErrorIs(t, got, tc.want) + } + assert.Contains(t, out.String(), fmt.Sprintf(`"outcome": %q`, tc.outcome), "the verdict is printed before the sentinel returns") + }) + } +} + func TestRetryPolicyDefaults(t *testing.T) { t.Run("kong defaults match the executor defaults", func(t *testing.T) { c := parseMigrate(t) diff --git a/pkg/verdict/docs_test.go b/pkg/verdict/docs_test.go index bee3d98..7f72967 100644 --- a/pkg/verdict/docs_test.go +++ b/pkg/verdict/docs_test.go @@ -2,7 +2,11 @@ package verdict import ( "fmt" + "go/ast" + "go/parser" + "go/token" "os" + "strconv" "strings" "testing" @@ -19,6 +23,120 @@ const cliOutputExamplesDoc = "../../docs/cli-output-examples.md" // reason; a reason without a class row there cannot be routed by consumers. const refusalClassesDoc = "../../docs/refusal-classes.md" +// exitCodeLadderDocs are the pages that state the process exit-code ladder +// as a table whose first cell is the code; the README is where a CI author +// first meets it, the CLI examples page is the machine contract, and the +// passthrough design holds the reasoning behind the non-obvious cells. +var exitCodeLadderDocs = []string{ + "../../README.md", + cliOutputExamplesDoc, + "../../docs/lock-budgeted-passthrough.md", +} + +// exitCodeLadderHeader is the header row of a ladder table; only rows under +// this header count as the ladder, so a numbered list elsewhere on the page +// (acceptance criteria, invariants) cannot satisfy the check by accident. +const exitCodeLadderHeader = "| Exit | Meaning |" + +// exitCodeLadder is every process exit code the binary produces: the +// ExitCode* constants declared anywhere in this package's non-test source, +// read from the files so a new constant is in the ladder before anyone +// remembers to list it, whichever file its author opened, plus the two shell +// conventions the entry point inherits without a named constant — 0 for +// success and 1 for any error kong reports. +func exitCodeLadder(t *testing.T) map[int]struct{} { + t.Helper() + entries, err := os.ReadDir(".") + require.NoError(t, err) + + fset := token.NewFileSet() + ladder := map[int]struct{}{0: {}, 1: {}} + for _, entry := range entries { + name := entry.Name() + if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + file, err := parser.ParseFile(fset, name, nil, parser.SkipObjectResolution) + require.NoError(t, err) + for _, decl := range file.Decls { + gen, ok := decl.(*ast.GenDecl) + if !ok || gen.Tok != token.CONST { + continue + } + for _, spec := range gen.Specs { + collectExitCodes(t, ladder, spec) + } + } + } + require.Greater(t, len(ladder), 2, "the walker found at least one declared ExitCode constant") + return ladder +} + +// collectExitCodes adds every ExitCode* name in one const spec to the +// ladder. Each name must carry its own integer literal: a spec that repeats +// the previous one implicitly, or derives from iota or another constant, is +// rejected by name rather than indexed past the end of its values. +func collectExitCodes(t *testing.T, ladder map[int]struct{}, spec ast.Spec) { + t.Helper() + vs, ok := spec.(*ast.ValueSpec) + if !ok { + return + } + for i, name := range vs.Names { + if !strings.HasPrefix(name.Name, "ExitCode") { + continue + } + require.Lessf(t, i, len(vs.Values), "%s states its value explicitly", name.Name) + lit, ok := vs.Values[i].(*ast.BasicLit) + require.Truef(t, ok && lit.Kind == token.INT, "%s is an integer literal", name.Name) + code, err := strconv.Atoi(lit.Value) + require.NoError(t, err) + ladder[code] = struct{}{} + } +} + +// documentedExitCodes returns the first-cell integer of every row under +// every ladder header in the page. A doc with no ladder table yields an +// empty set, which the equality check then reports as missing every code. +func documentedExitCodes(t *testing.T, doc string) map[int]struct{} { + t.Helper() + codes := map[int]struct{}{} + lines := strings.Split(doc, "\n") + for i, line := range lines { + if !strings.HasPrefix(line, exitCodeLadderHeader) { + continue + } + // Skip the header and the separator row, then read rows until the + // table ends. + for _, row := range lines[i+2:] { + if !strings.HasPrefix(row, "|") { + break + } + cells := strings.SplitN(row, "|", 3) + require.Lenf(t, cells, 3, "ladder row has a first cell: %q", row) + code, err := strconv.Atoi(strings.TrimSpace(cells[1])) + require.NoErrorf(t, err, "ladder row's first cell is an exit code: %q", row) + codes[code] = struct{}{} + } + } + return codes +} + +// Every page that states the ladder must list exactly the exit codes the +// binary produces: an exit-code constant added to this package without a row +// saying what it means for a shell gate fails here, a ladder table that +// drops a code fails, and so does a row for a code the binary no longer +// exits with. +func TestExitCodeLadderDocsListEveryCode(t *testing.T) { + ladder := exitCodeLadder(t) + for _, path := range exitCodeLadderDocs { + raw, err := os.ReadFile(path) + require.NoError(t, err) + assert.Equalf(t, ladder, documentedExitCodes(t, string(raw)), + "%s must state the exit-code ladder as a table under %q with one row per code", path, exitCodeLadderHeader) + } +} + // Every refusal reason automation can meet must be documented: a Reason // constant added without a row in the doc's refusal-reason table fails here. func TestDocListsEveryRefusalReason(t *testing.T) {