feat(cli): use stack EphemeralPostgres for schema-tooling shadows

[avallete]

Summary

When [experimental].stack / SUPABASE_EXPERIMENTAL_STACK=1 is on:

• supabase stack is gated on the flag, including help and completion. Top-level start/stop/status become stack aliases. Invalid project config leaves the feature off; invalid env overrides are still errors. Flag off keeps legacy Compose start/stop/status and Docker SQL-template shadows.
• --local uses the project stack. db, migration, test db, gen types, and inspect resolve --local through stack credentials instead of Compose names (supabase_db_*, db:5432). gen types --local and inspect db … --local do not start a stack. Linked and explicit --db-url stay URL / Management API targets (a URL that happens to match config.toml ports is still a URL). Functions, storage, and top-level pull stay on their existing backends.
• Schema tooling uses EphemeralPostgres. db diff, db pull, declarative generate/sync, and migration squash provision throwaway shadows through @supabase/stack. The cluster is a scoped second instance (native host PGDATA or a named volume). Snapshots are runtime-kind specific (stack-shadow-baseline-*.tar, same keep-3 / 2-day TTL as the legacy namespace). The runtime marker can carry snapshotKey; keyless tars still restore. Archive flags are portable (GNU tar and Darwin bsdtar). Ephemeral env files are 0600; the cache dir is 0700.
• The stack backend requires in-process pg-delta. --use-migra, --use-pgadmin, --use-pg-schema, and --diff-engine migra are rejected for every stack runtime because the shadow is always EphemeralPostgres, including --linked / --db-url.
• Live start is Compose-faithful. First create (unconfigured): schema init, overlay (webhooks, grants, vault, roles.sql), then migrate-and-seed. Existing cluster: webhooks only — no second overlay or migrate-and-seed. A failed first setup leaves the engine up and does not retry on the next start; recover with db reset. Disabled capabilities keep nested pins (version, settings).
• db start is postgres-only on first create. Ingress follows listener intents, so a database-only plan does not require API-key material. An existing stack resumes its persisted services. --from-backup is unsupported. db reset --local and declarative --apply wipe through resetDatabase (a failed wipe stays start-eligible), then migrate or seed on stack credentials. Bucket seed talks to the stack API endpoint and waits only while storage is enabled and bouncing.
• Native tool clients. db dump --local, db test / test db, and migration squash use PATH pg_dump / pg_prove on native engines (must match the stack Postgres major). Docker/Podman keeps the one-shot tool container against published credentials, never PGHOST=db. On Linux, omit / empty / host --network-id keeps loopback instead of rewriting to host.docker.internal.
• Native second stack. Realtime rpc / GEN_RPC_* binds a unique host port on native runtimes only. Docker publications omit rpc.
• Catalog overlay for shadows and first create. Local first-create and shadow Postgres get the auth, storage, and realtime schema Compose would after migrate, without starting those services. --exclude does not skip that catalog set. Docker catalog SQL matches Compose; native schema-init is migrate-only (Realtime bin/prepare omits SEED_SELF_HOST).
• Shadow cache / teardown. Keys resolve the same default runtime as create (Docker if installed). Remotes *EnabledForSetup overlay onto ephemeral catalog config so db pull does not schema-init services remotes turned off. Interrupted Postgres is torn down with the start fiber; in-flight Docker/Podman create IDs are registered so Ctrl-C cannot orphan them. Native spawn drains pipes.

Docs: ADR 0025, stack-commands, and the SIDE_EFFECTS.md pages for the switched commands (including ~/.supabase/ephemeral-postgres/<identity>/).

[github-actions]

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@bcc62728d19c93a1cd748e7dd832c2a3c9c05f25

Preview package for commit bcc6272.

[Coly010]

Reviewed at ae7a323 (review agents across architecture, engineering, security and DX, plus manual dogfooding on the cli-staging bench: macOS, Docker Desktop 29.7, container runtime). Requesting changes. The ephemeral-shadow core is solid and I could not make it leak, but the PR's own db start path does not work on a real stack, three command families silently stay on the legacy backend with the flag on, and the snapshot cache dropped the integrity checks the legacy cache had.

What I dogfooded

Scenario	Result
db diff --linked, db pull, migration list --linked	✅ ephemeral shadow, ~5s warm
migration squash with two migrations	✅ cold shadow + Docker pg_dump one-shot, 8s
Cold → warm shadow cache	✅ stack-shadow-baseline-*.tar 0600 in a 0700 dir, mtime touched on warm hit, 7.6s → 2.5s
Ctrl-C 3.5s into a cold shadow (ephemeral Postgres + storage schema-init up)	✅ no containers, volumes, networks or .partial left
Ephemeral cluster publish binding	✅ 127.0.0.1:<port> only
--use-migra / --use-pgadmin / db start --from-backup / invalid env or config values	✅ rejected as documented
db start	❌ failed to start local database: Persisted API gateway material is incomplete; leaves a registered stack with desired lifecycle stopped; --debug adds nothing
gen types --local with the stack running	❌ supabase start is not running. from the legacy Compose check — routes to legacy under the flag
db reset --local	❌ wiped and re-bootstrapped the database, then failed on the overlay connect; status kept reporting database ready
every other --local command	⛔ blocked on my machine by a pre-existing stack gateway bug (below), worked around with --db-url on the private port

Blockers

1. db start cannot work on a real stack. The postgres-only overlay excludes auth, and packages/stack/src/supervisor/Ingress.ts:276 gates API-key material on definition.listeners.api.enabled rather than on auth/rest being enabled, so the four auth key slots are missing and the supervisor fails. stack start -x auth fails identically on develop, so the bug is pre-existing, but this PR is its first mainstream caller and the db start integration tests mock StackApi, so nothing exercises it. Fix the gate (or make the API listener follow the auth/rest exclusion) here or in a prerequisite PR, and add one real container-runtime db start test.
2. Routing allow-list misses shared consumers (pull, gen, inspect). Inline.
3. Catalog overlay re-runs roles.sql on already-running stacks. Inline.
4. Snapshot tar restored without a key-bound marker, extracted with default tar flags. Inline.

Should fix

• failed resetDatabase leaves the stack claiming running with a dead database and a stale activation cache (inline)
• no reaper for orphaned supabase-eph-* containers/volumes/PGDATA after a hard kill (inline)
• loopback port allocation TOCTOU, matters once declarative sync provisions two shadows (inline)
• ephemeral Postgres env file created world-readable then chmod'd with the failure ignored (inline)
• cache directory mode only applied at creation (inline)
• --runtime docker advice is unreachable for an existing native stack (inline)
• db dump --linked / --db-url drop SUPABASE_NETWORK_ID with the flag on (inline)
• stack db reset skips bucket seeding whenever storage is still bouncing (inline)
• db pull's seeded-from-dump branch still needs a Docker pg_dump container (inline)
• engine-flag rejection applies to --linked / --db-url too, contradicting "linked targets are unchanged" (inline)

Docs and contract

• SIDE_EFFECTS.md is stale for five commands whose behaviour changed under the flag: db reset, db start, db pull (and top-level pull), migration squash, db schema declarative generate/sync. Nothing documents the new ~/.supabase/ephemeral-postgres/<identity>/ state location either.
• ADR 0025 says "Top-level status is not switched"; root.ts aliases it.
• stack-commands.md describes db start as "starts the database without persisting --exclude", but db start has no --exclude flag and in practice resumes the full persisted stack when one exists. Say that plainly.
• packages/config/src/experimental.ts description omits test db and should say --local targets only (inline).
• (or Git Bash) in the Windows dump/squash suggestion is wrong: Git Bash ships no pg_dump. test db on the same path has no suggestion at all.
• WARNING: skipped analytics schema init: analytics schema init requires the activated analytics process reached me during db reset. "Schema init" is ADR vocabulary; users need "auth/storage/analytics database setup was skipped because the service is not running".
• StackCatalogSetupError has no [ErrorActionabilityId] (inline).

Pre-existing stack bugs surfaced by this dogfood (not this PR's fault, but they block it)

• Database TCP gateway never proxies on macOS. With develop's supervisor as well as this PR's, Postgres answers directly on its published private port in ~17ms, but the stack's own 127.0.0.1:54322 listener times out for host pg clients and returns "server closed the connection unexpectedly" for the pg_dump/pg_prove containers via host.docker.internal. The HTTP gateway (REST, auth health) works. Since every --local command in this PR goes through credentials().database.url, none of them work on a Mac today. macOS CI only runs the port tests, which would explain why this is green.
• stack start -x auth fails as described in blocker 1.

Done well

ADR decisions (a)–(e) each map to code; the auth/storage/realtime policy lives in the CLI and SchemaInit stays generic; runtime.kind + engine + artifact identity in the cache key with the separate stack-shadow-baseline- namespace means legacy and stack tars cannot collide; addFinalizer(destroyCluster) registered before acquisition and uninterruptibleMask scoped to the handoff, which the Ctrl-C test bore out; the ephemeral-postgres and reset-database integration tests run real Postgres rather than mocks; db diff's SIDE_EFFECTS.md update is exemplary.

[jgoux]

Requesting changes for the six runtime defects detailed inline. The ADR routing correction is nonblocking.

The findings were independently cross-checked and supported by code traces and targeted reproductions. Build, type checks, changed-file lint/format checks, and 951 targeted tests passed (one platform-specific skip), but those checks do not cover the failures identified here. Full E2E and the complete platform matrix were not run.