Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,12 @@ refusal — never a silently wrong or incomplete result:
(`executor.RebuildAbandonedIndex`, or `executor.DropAbandonedIndex` when
the caller must not rebuild) is library-only; from the CLI the
[runbook](docs/invalid-index-recovery.md) applies.
- **The accepted-blocking passthrough has no CLI flag yet** — running an
otherwise-refused change deliberately under the engine's bounded lock
budget, with the verdict marked `executed-without-online-safety` (exit 3),
is library-only (`executor.ExecuteAcceptedBlocking`); from the CLI these
refusals exit 2. Design:
[docs/lock-budgeted-passthrough.md](docs/lock-budgeted-passthrough.md).
- **Non-table objects** — views, standalone sequences, enums, domains,
extensions, functions, triggers — are outside the declarative model,
which covers one ordinary table plus its indexes per file.
Expand Down
2 changes: 1 addition & 1 deletion SAFETY.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The invariant registry (invariant IDs referenced below) lives in
| `pkg/capabilities` — embedded, validated support matrix and Markdown rendering | ❌ periphery | exists | — |
| `pkg/diffplan` — desired schema → routed convergence plan, the declarative front door as a library (the CLI `diff` and embedding orchestrators share it) | ❌ periphery | exists | — |
| `pkg/migrate` — one gated statement → resolve, classify, route, execute → one verdict; the imperative front door as a library (the CLI `migrate` and embedding orchestrators share it), plus the desired-state execution loop (`RunDesired`: derive the convergence plan, admit it as a whole, run each planned statement back through the same pipeline) | ❌ periphery² | exists | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `status`, `diff`, `fmt`, `lint`, and `suggest` exist | — |
| `internal/cli` — CLI, flags, help, prompts | ❌ periphery | `migrate`, `pull`, `diff`, `fmt`, `lint`, `suggest`, `capabilities`, and `status` exist | — |
| `pkg/progress` — strategy-wide progress snapshots; the executors' observation seam (core imports it, so its locking discipline is core-critical); copy counters reserved for later | ✅ core | native progress exists | — |
| orchestrator adapter | ❌ periphery | planned (Phase 11) | OC-* hold *at* the boundary |
| `internal/testutil` | ❌ test-only | exists | — |
Expand Down
28 changes: 17 additions & 11 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,18 +21,19 @@ pg-sprite is a decoupled **planner → router → executor** engine. The planner
changes, the router decides *which strategy*, interchangeable executors decide *how*.

The planner is itself a pipeline of five distinct stages. The two front-ends enter it at
different points — an imperative `--alter` already *is* DDL, so it goes straight to parse;
a declarative `--desired` schema must first be compared against the live database to
different points — an imperative `migrate --alter` already *is* DDL, so it goes straight to parse;
a declarative `diff --desired` schema must first be compared against the live database to
*produce* DDL — and the derived statements then re-enter the parse boundary like any
hand-written statement, so both routes converge on the same parse → classify → lint tail
and every operation is judged by the same rules regardless of how it arrived:

```
user: --alter "ALTER TABLE …" user: --desired schema.sql
user: migrate --alter "ALTER TABLE …" user: diff --desired schema.sql
(imperative: statements) (declarative: whole schema)
│ │
╭─────────▼──────────────────────────────────────▼─────────╮
│ CLI: migrate · diff · fmt · lint · suggest · status │
│ CLI: migrate · pull · diff · fmt · lint · suggest · │
│ capabilities · status │
╰─────────┬──────────────────────────────────────┬─────────╯
│ │
┌──────────────▼───── PLANNER (shared front-end) ─────▼──────────────┐
Expand Down Expand Up @@ -161,10 +162,15 @@ architectural decision; this is the permission slip. Two integration surfaces ex
different levels of commitment:

- **The CLI and its JSON output** — the intended seam for orchestrators. `diff` and
`--dry-run` emit machine-readable verdicts and plans; the plan report freezes as a
single versioned contract (an explicit schema-version field, additive-only changes
within a version) in Phase 2.5. Until that lands its shape may change in any PR — wait
for the versioned report rather than pinning the interim shape.
`--dry-run` emit machine-readable verdicts and plans; the plan report is a single
versioned contract (`format_version`, `plan.FormatVersion`; additive-only changes
within a version, a consumer rejects a version it does not understand), documented in
[plan-report.md](plan-report.md). The suggest report carries its own `format_version`
on the same rule ([suggest-report.md](suggest-report.md)). `capabilities --json` is
deliberately not versioned separately: its `version` is the binary version string,
because the matrix is committed with and tested against one binary, so consumers pin
the binary rather than a matrix version and ignore fields they do not recognize
([capabilities-contract.md](capabilities-contract.md#versioning)).
- **The Go packages** — everything under `pkg/` is importable, and the front-end seams
(`pkg/statement`, `pkg/schemadiff`, `pkg/planner`, `pkg/router`, `pkg/verdict`) are
each designed as a standalone entry point; `internal/` is unimportable by
Expand All @@ -177,8 +183,8 @@ different levels of commitment:

| Package | Role | Status |
| --- | --- | --- |
| `cmd/pg-sprite` | CLI entry point (Kong): `migrate` · `diff` · `fmt` · `lint` · `suggest` · `status` | all six exist |
| `internal/cli` | Command tree and flag handling (including `migrate --dry-run`) | all six exist |
| `cmd/pg-sprite` | CLI entry point (Kong): `migrate` · `pull` · `diff` · `fmt` · `lint` · `suggest` · `capabilities` · `status` | all eight exist |
| `internal/cli` | Command tree and flag handling (including `migrate --dry-run`) | all eight exist |
| `internal/testutil` | Test harness: containerized PostgreSQL, throwaway schemas | exists |
| `pkg/dbconn` | Pool with bounded session timeouts, retries, RDS/Aurora auto-TLS (embedded CA bundle), terminate-blockers; advisory-lock mutual exclusion lands here | exists |
| `pkg/statement` | `go-pgquery` (Wasm `libpg_query`) parse boundary, typed per-operation descriptors, and advisory rewrites (never hand-parse SQL); shadow DDL is validated by executing the retargeted statement on the empty shadow, and fingerprints come from `pkg/schemadiff`'s transaction-scoped scratch schema — execute-and-introspect, never AST surgery | exists |
Expand All @@ -192,7 +198,7 @@ different levels of commitment:
| `pkg/diffplan` | The declarative front door as a library: desired schema in, routed `plan.Report` out — the CLI `diff` and embedding orchestrators share this one pipeline | exists |
| `pkg/migrate` | The imperative front door as a library: one parsed statement in — gate, resolve, classify, route, execute — one `verdict.Verdict` out; the CLI `migrate` and embedding orchestrators share this one pipeline. Also the desired-state execution loop: `RunDesired` derives the convergence plan (`diffplan.Plan`), admits it as a whole (existence, destructive guard, dispositions, optional fingerprint pin), and runs each planned statement back through `Run` — per-statement verdicts, committed-prefix semantics | exists |
| `pkg/router` | Route classified statements to native / copy-and-swap / refuse dispositions; copy-and-swap reports unavailable until that backend lands | exists (Phase 2.4) |
| `pkg/executor` | Bounded optimistic native attempt, the concurrent index build, and the autocommit safer-sequence runner, with stable outcome codes; the full `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) arrives with the copy-and-swap backend | native execution exists |
| `pkg/executor` | Native backend with stable outcome codes: the bounded optimistic attempt, the concurrent index build and its invalid-index recovery, the autocommit safer-sequence runner, the greenfield `CREATE TABLE` path, and the accepted-blocking passthrough primitive; the full `Executor` contract (`Plan`/`Execute`/`Status`/`Abort`) arrives with the copy-and-swap backend | native execution exists |
| `pkg/progress` | Strategy-wide, pollable progress snapshots: native phase/elapsed time, sequence position, retry attempt, and server-reported concurrent-index work; optional copy counters are reserved for copy-and-swap | native progress exists |
| `pkg/copier` | PK-range chunker over one integer-family primary key with dynamic time-based sizing (produces `Chunk` and `Watermark`; composite keys refused in v1), and the parallel chunked copy into the shadow table (never overwrites) — there is no separate chunker package | contracts exist; copy loop Phase 4 |
| `pkg/checksum` | The mandatory correctness gate; continuous checker; repair primitive | Phase 5 |
Expand Down
9 changes: 5 additions & 4 deletions docs/capabilities-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,10 +203,11 @@ understand.

## Shared refusal vocabulary

A refusal `class` on the verdict JSON is a separate decision, recorded in
[refusal-classes.md](refusal-classes.md); `pkg/verdict` does not emit one today. When that
field ships, that contract owns the vocabulary and the matrix uses the same words, so a
consumer reading a row and a consumer reading a verdict reach the same route:
A refusal `class` on the verdict JSON is a separate contract, recorded in
[refusal-classes.md](refusal-classes.md) and emitted by `pkg/verdict` on every refusal
(`Verdict.WithRefusal`; the reason → class mapping is the registry in `pkg/migrate`). That
contract owns the vocabulary and the matrix uses the same words, so a consumer reading a
row and a consumer reading a verdict reach the same route:

| Matrix row | Refusal `class` |
| --- | --- |
Expand Down
19 changes: 12 additions & 7 deletions docs/capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,13 +337,18 @@ where one exists — the exact safer sequence or the statement an operator can r
deliberately, outside the engine, in a maintenance window. The operator stays in
control; the engine stays honest. `--force` never bypasses a policy refusal.

A constrained variant is **planned**: an explicit, dedicated flag (distinct from
`--force`) that executes an otherwise-refused change through the engine's own bounded
`lock_timeout` sessions ("unsafe DDL under a bounded lock budget", which raw psql does
not give you), with the refusal analysis still printed before execution and the verdict
unmistakably marked as executed without an online-safety guarantee. The plain success
contract stays reserved for online-safe paths, and refusals for unrecognized SQL are
never eligible — only changes the engine understands but cannot run *safely*.
A constrained variant exists as a **library primitive, with no CLI flag yet**: an
explicit, dedicated acceptance (distinct from `--force`) executes an otherwise-refused
change through the engine's own bounded `lock_timeout` sessions ("unsafe DDL under a
bounded lock budget", which raw psql does not give you), with the refusal analysis still
produced before execution and the verdict unmistakably marked as executed without an
online-safety guarantee (`outcome: executed-without-online-safety`, exit 3 — never 0).
`executor.ExecuteAcceptedBlocking` runs it and `Verdict.WithAcceptedBlocking` records it;
the refused statement's `blocking_passthrough_eligible` field on the plan report says
whether a refusal qualifies. Only changes the engine understands but cannot run *safely*
are eligible — refusals for unrecognized SQL never are. A `migrate` flag that reaches the
primitive is the remaining step; until it lands the CLI exits 2 for these refusals. The
design is [lock-budgeted-passthrough.md](lock-budgeted-passthrough.md).

## Deliberately operator-owned

Expand Down
12 changes: 10 additions & 2 deletions docs/cli-output-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ is a one-line summary; the linked reference entry is authoritative.
| [`unsupported-partitioned-parent`](postgres-online-ddl-reference.md#unsupported-partitioned-parent) | The routed plan builds an index concurrently but the target is a partitioned parent, where PostgreSQL cannot `CREATE INDEX CONCURRENTLY`. Refused. |
| [`unsupported-statement`](postgres-online-ddl-reference.md#unsupported-statement) | The planner knows no safe path for the statement (for example `SET UNLOGGED`, `CLUSTER ON`). Refused — the same typed reason the run path's refusal verdict carries. |
| [`table-not-found`](postgres-online-ddl-reference.md#table-not-found) | The target table does not exist, so classification fell back to zero facts; running without `--dry-run` would fail. The dry run exits 2 and the report carries `table_exists: false`. |
| [`destructive`](postgres-online-ddl-reference.md#destructive) | The change discards live data or structure (`DROP COLUMN`, `DROP TABLE`, truncating conversions). A warning alongside the routing decision, not a refusal. |
| [`destructive`](postgres-online-ddl-reference.md#destructive) | The change discards live data or structure — a dropped column, constraint, index, or `NOT NULL` (dropping a `DEFAULT` is not destructive). A warning alongside the routing decision, not a refusal; `DROP TABLE` never reaches classification, it refuses as `unsupported-statement`. |
| [`blocking-idiom`](lint-report.md#codes-code) | Lint-only code: the submitted form blocks readers or writers and a safer native form exists; the finding's `suggestion` carries the safer SQL when the linter can construct it. |

## Refusal reasons
Expand All @@ -121,7 +121,15 @@ in `detail`. The set is closed and pinned by test (`verdict.Reasons()`).
| `destructive-change` | The desired-state plan discards live structure — a dropped column, constraint, index, or `NOT NULL` — and desired-state execution runs no destructive statement; run the drop deliberately instead ([execution model](execution-model.md)). |
| `plan-fingerprint-mismatch` | The plan recomputed at execution time does not carry the pinned fingerprint: the plan a reviewer approved is not the plan that would execute, so nothing runs ([execution model](execution-model.md)). |
| `create-collision` | The greenfield create plan's table name or a claimed index, constraint-index, or sequence name is occupied. Nothing runs; re-derive the plan against the live catalog to see what holds the name, then drop or rename the occupant, name a constraint's index explicitly, or for a sequence use an explicitly named sequence or a non-serial column — re-planning alone reproduces the refusal. Catalog absence checks handle existing occupants. Duplicate-name SQLSTATEs backstop races for explicit names; for server-chosen names, the probe narrows the race to the time-of-check window, and after the `CREATE TABLE` commits the executor reads the constraint-index and sequence names the table actually owns and compares them against the claimed first-choice names — a name taken inside the window makes the server pick a suffixed replacement, which surfaces as a typed `create-name-mismatch` failure at step 1 with the born table left in place for an operator to rename the relation or drop, then re-diff. |
| `create-names-unverified` | The `CREATE TABLE` committed but the read of the constraint-index and sequence names the table owns did not complete, so whether every first-choice claim was honoured is unknown. The born table is left in place; compare its names against the desired file, rename or drop, then re-diff. |

Two codes that look like refusals are not: `create-name-mismatch` and
`create-names-unverified` are executor failure codes on a `failed` verdict (exit 1,
`failed_step` 1), because the `CREATE TABLE` has already committed when they arise.
The first means a claimed first-choice constraint-index or sequence name went to an
occupant inside the probe's window and the server suffixed it; the second means the
read of the names the table owns did not complete, so the claims are unproven, not
failed. In both the born table is left in place — compare its names against the
desired file, rename or drop, then re-diff.

## Migrate

Expand Down
12 changes: 7 additions & 5 deletions docs/high-level-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,12 +127,14 @@ is the right design on PostgreSQL specifically — is
## Architecture at a glance

```diagram
user: --alter "..." OR --desired schema.sql
user: migrate --alter "ALTER TABLE …" OR diff --desired schema.sql
╭──────────▼──────────-╮
│ CLI: migrate · diff ·│
│ fmt · lint · status │
╰──────────┬──────────-╯
╭──────────▼───────────────╮
│ CLI: migrate · pull · │
│ diff · fmt · lint · │
│ suggest · capabilities · │
│ status │
╰──────────┬───────────────╯
╭───────────────╮ shared front-end:
│ PLANNER │ parse · introspect ·
Expand Down
2 changes: 1 addition & 1 deletion docs/limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ not escape hatches:

| Change | Current behavior |
| --- | --- |
| Index builds on a partitioned parent | PostgreSQL cannot build an index concurrently at the parent level. PostgreSQL supports a plain blocking build, but pg-sprite refuses it by policy because it takes `ACCESS EXCLUSIVE`; `--force` does not bypass this decision. An operator who chooses a maintenance-window blocking build must run it outside pg-sprite. The partition-aware `CREATE INDEX ON ONLY` → per-partition `CREATE INDEX CONCURRENTLY` → `ATTACH PARTITION` flow is planned but not yet implemented. |
| Index builds on a partitioned parent | PostgreSQL cannot build an index concurrently at the parent level. PostgreSQL supports a plain blocking build, but pg-sprite refuses it by policy because it takes `ACCESS EXCLUSIVE`; `--force` does not bypass this decision. An operator who chooses a maintenance-window blocking build runs it outside the CLI today; a library caller can run it through `executor.ExecuteAcceptedBlocking`, which marks the verdict as executed without online safety ([lock-budgeted-passthrough.md](lock-budgeted-passthrough.md)). The partition-aware `CREATE INDEX ON ONLY` → per-partition `CREATE INDEX CONCURRENTLY` → `ATTACH PARTITION` flow is planned but not yet implemented. |
| `ADD CONSTRAINT ... USING INDEX` on a partitioned parent | PostgreSQL does not support adopting an existing index on a partitioned parent in any supported version. pg-sprite refuses before execution. |
| `ADD FOREIGN KEY ... NOT VALID` on a partitioned parent | PostgreSQL does not support this before version 18, so pg-sprite refuses it on versions 14–17. It is supported on version 18 and later. |
| Copy-and-swap | The copy-and-swap backend is not yet available. Statements that require it route to `refuse`; pg-sprite never falls through to a blocking rewrite. |
Expand Down
Loading
Loading