Support more native datatypes in Walshadow

Cargo.toml

@@ -46,6 +46,7 @@ astral-tokio-tar = { version = "0.6", default-features = false }
 async-channel = "2.5"
 async-trait = "0.1"
 backon = { version = "1", features = ["tokio-sleep"] }
+bumpalo = "3"
 bytes = "1"
 chrono = { version = "0.4", default-features = false, features = ["clock"] }
 clap = { version = "4", features = ["derive", "env"] }

docker/Dockerfile

@@ -49,8 +49,9 @@ RUN make -C pgext clean \
 
 FROM postgres:${PG_MAJOR}-alpine AS runtime
 # bash: entrypoint shebang. ca-certificates: TLS to clickhouse.
-# liblz4/libzstd already present as PG deps.
-RUN apk add --no-cache bash ca-certificates
+# postgis: so the daemon-owned shadow PG can render the schema's geography
+# column via typoutput. liblz4/libzstd already present as PG deps.
+RUN apk add --no-cache bash ca-certificates postgis
 
 COPY --from=rust-builder /usr/local/bin/walshadow-stream /usr/local/bin/walshadow-stream
 COPY --from=ext-builder /src/pgext/walshadow.so \

docker/Dockerfile.source

@@ -0,0 +1,8 @@
+# Source Postgres for the stress schema: stock postgres:18 + PostGIS, so the
+# gist dump's geography column loads unchanged.
+FROM postgres:18-bookworm
+RUN apt-get update \
+    && apt-get install -y --no-install-recommends \
+        postgresql-18-postgis-3 \
+        postgresql-18-pgvector \
+    && rm -rf /var/lib/apt/lists/*

docs/destination-tables.md

@@ -64,7 +64,12 @@ Common mappings include:
 | `time` | `Time64(6)` |
 | `timestamp`, `timestamptz` | `DateTime64(..., 'UTC')` |
 | `uuid` | `UUID` |
-| `json`, `jsonb`, `inet`, `cidr`, `interval`, arrays, unknown types | `String` |
+| `json`, `jsonb` | `JSON` |
+| `hstore` | `Map(String, Nullable(String))` |
+| `vector`, `halfvec` (pgvector) | `Array(Float32)` |
+| `geography`, `geometry` (PostGIS) | `String` (WKT) |
+| `<elem>[]` arrays | `Array(Nullable(<elem>))`; unknown elem → `Array(Nullable(String))` |
+| `inet`, `cidr`, `interval`, unknown types | `String` |
 
 Nullable source columns become `Nullable(...)` unless used as ClickHouse sort
 keys. ClickHouse deployments using PostgreSQL `time` columns must enable

plans/INDEX.md

@@ -52,9 +52,10 @@ components. User workflows and supported behavior live under
 [future/INDEX.md](future/INDEX.md) collects design docs for unbuilt work:
 runtime-config signals and net-new knobs, two-phase commit,
 sequence-state replication, cross-table ordering, CH-bounce recovery,
-parked operational polish. Once built, keep behavior in code and tests, move
-user-facing consequences into `docs/`, and retain only rationale or invariants
-which code cannot express
+greenfield tier-3 oracle (throwaway bootstrap PG), oracle emitting CH-native
+blocks, parked operational polish. Once built, keep behavior in code and tests,
+move user-facing consequences into `docs/`, and retain only rationale or
+invariants which code cannot express
 
 ## Architecture diagrams
 

plans/bootstrap.md

@@ -53,7 +53,11 @@ for rendered diagram. Five clusters top→bottom:
    [emitter.md](emitter.md)). One synthetic ack seq per rfn flip;
    `tail.finish` seals partial batches and waits all seqs durable
    before handoff. Metrics-only runs (no `--ch-config`) instead drain
-   through `drain_backfill` into a counting `TupleObserver`
+   through `drain_backfill` into a counting `TupleObserver`.
+   Bridge-routed tier-3 values (jsonb, arrays, hstore, …) can't be
+   resolved here — the shadow/bridge don't exist until after bootstrap —
+   so they currently land empty; in-tree types (geography, vector) are
+   fine. Fix in [future/greenfield_oracle.md](future/greenfield_oracle.md)
 4. **Shadow handoff** — `BootstrapOutcome { start, end }` returned;
    daemon writes `standby.signal` and calls `materialize_conf` to
    replace shadow's config files. Config includes walshadow settings,

plans/emitter.md

@@ -356,7 +356,10 @@ today is `String`, anything else dies cleanly at `append`
 `pk_member = true` strips `Nullable(_)` wrap because CH refuses
 `Nullable` in `ORDER BY`. User-visible matrix lives in
 [`docs/destination-tables.md`](../docs/destination-tables.md#default-type-mapping),
-hard-coded by `base_type_for`
+hard-coded by `base_type_for`. Dynamic-OID types (hstore, pgvector, arrays,
+PostGIS) are matched on `RelAttr.type_name`, not OID; CH forbids `Nullable`
+over `Array`/`Map` so those stay bare (a NULL source value lands as an empty
+one), while `Nullable(JSON)` is allowed.
 
 `numeric` needs `1 ≤ p ≤ 76` for `Decimal`; `p = 0`, scale outside
 `0 ≤ s ≤ p`, or unconstrained `numeric` (which can carry NaN/±Inf) fall

plans/future/INDEX.md

@@ -16,6 +16,8 @@ rationale under `plans/` only when code cannot express it
 * [failover.md](failover.md) — beyond the switchover crossing in [../failover.md](../failover.md): unplanned promotion (transaction-state fence, overwrite contrecord), slotless pause windows, timeline-aware archive and base-backup replay
 * [sync_commit_witness.md](sync_commit_witness.md) — walshadow as RPO=0 durability standby
 * [two_phase_commit.md](two_phase_commit.md) — `XLOG_XACT_PREPARE` handling and gxid-keyed buffer
+* [greenfield_oracle.md](greenfield_oracle.md) — resolve bridge-routed tier-3 (jsonb/arrays/hstore) during greenfield bootstrap via a throwaway PG (walshadow module + source extensions) torn down after; OID-matching constraint (restore-from-backup vs resolve-by-name)
+* [oracle_native_blocks.md](oracle_native_blocks.md) — bridge/pgext emits CH-native column bytes instead of `typoutput` text, batching `DECODE` from row-at-a-time to a column-major list of rows (matches CH's columnar block, amortizes the round trip); removes the emitter-side composite re-parse (`ColumnBuf::{Array,Map,Json}`, text parsers, `NodeArena`); does not solve greenfield
 * [ch_bounce_recovery.md](ch_bounce_recovery.md) — deeper re-emit-from-spill on retry-budget exhaustion
 * [pinned_ddl_baseline.md](pinned_ddl_baseline.md) — schema-event outcome must be a function of config + baseline, not cache warmth: CH-existence / persisted-baseline options for cross-restart consistency, drop detection across downtime, opt-in mapping vs republish
 * [coverage100.md](coverage100.md) — drive `cargo llvm-cov` line coverage toward 100%: tiered work list (pure units → fixtures → live e2e → hard tail)

plans/future/greenfield_oracle.md

@@ -0,0 +1,57 @@
+# greenfield_oracle — a bridge for tier-3 during bootstrap
+
+## Problem
+
+Greenfield bootstrap ([../bootstrap.md](../bootstrap.md)) page-walks a
+base/object-store backup and drains rows through `pipeline::bootstrap::drain`
+**before the shadow PG and its bridge exist** (bridge is created after
+`run_bootstrap` returns). So bridge-routed tier-3 values — jsonb, arrays,
+hstore, tsvector, ranges, domains — have no PG to render them and land empty.
+In-tree types (geography, vector; see [../oracle.md](../oracle.md)) are fine.
+
+`BackupSource` is a page reader, not a running PG, so there is nothing to
+decode on-disk tier-3 Datums during the drain.
+
+## Approach
+
+Stand up a **throwaway Postgres with the walshadow module + the source's
+extensions**, point the bridge/oracle at it for the bootstrap drain, then tear
+it down once bootstrap completes. `resolve_decoded_heap` already takes an
+`Option<&Oracle>`; wire this oracle in for the greenfield drain and the whole
+tier-3 path resolves exactly like live.
+
+Lifecycle: create → (extensions/module ready) → pass `Some(oracle)` to
+`bootstrap::drain` → drain → drop the temp PG + its datadir/socket before the
+real shadow is materialized for streaming.
+
+## The OID-matching constraint (the hard part)
+
+`ws_decode_datum_text` renders a Datum by running the type's `typoutput`,
+looked up by **OID**. Built-in tier-3 OIDs are stable across clusters (jsonb
+3802, `int4[]` 1007, …) so a fresh `initdb` + `CREATE EXTENSION` handles them.
+But **extension type OIDs are assigned at `CREATE EXTENSION` time and differ
+per cluster** — a fresh temp PG's `hstore`/`geography`/`vector` OID won't match
+the source OID carried in the on-disk bytes, so typoutput lookup misfires.
+
+Two ways to satisfy it:
+
+- **Restore the temp PG from the backup** (it then carries the source catalog,
+  OIDs match) — essentially a short-lived shadow. Reuses the base-backup we
+  already fetched; heaviest but exact. Overlaps with the Option-A framing in
+  the earlier analysis.
+- **Resolve by type name, not OID** — extend the bridge `DECODE` protocol to
+  carry the type name; the worker looks up `typoutput` via
+  `regtype`/`pg_type.typname` in the temp PG (which has the same-named
+  extensions installed). Lets a plain `initdb` + extensions work regardless of
+  OID drift. Smaller PG, but a protocol + worker change.
+
+## Open questions
+
+- Which extensions to install: derive from the source catalog (types actually
+  present) vs a fixed set; fail-soft when one isn't available.
+- Cost/timing: temp-PG spin-up vs bootstrap duration; only worth it when tier-3
+  columns exist in the mapped set.
+- Interaction with restart/resume: bootstrap re-runs must recreate/tear down
+  the temp PG idempotently.
+- Does not change the emitter contract; it only makes `Some(oracle)` available
+  earlier. Orthogonal to [oracle_native_blocks.md](oracle_native_blocks.md).

plans/future/oracle_native_blocks.md

@@ -0,0 +1,78 @@
+# oracle_native_blocks — bridge emits CH-native, not text
+
+## Problem
+
+Today bridge-routed tier-3 values round-trip through **text**: the worker runs
+`typoutput` → `ColumnValue::Text`, and the emitter then **re-parses** that text
+back into ClickHouse's columnar form. The emitter-side machinery that does this
+re-parse — `ColumnBuf::{Array,Map,Json}`, `encode_array`/`encode_map`/
+`encode_json` (with `parse_pg_array_1d` / `parse_vector_list` / hstore + JSON
+parsers), and the `NodeArena` nested-`ColumnBuilder` builder — is
+decode-to-text-then-parse-back, which is wasteful and brittle (PG text quoting,
+CH `JSON` object-only + null-slot rules, etc.).
+
+## Approach
+
+Have the resolver produce ClickHouse column data directly, so the emitter
+splices it in without a text detour. The extension links a CH serializer
+(`clickhouse-c` / `pg-clickhouse-c`); the `DECODE` op carries the **target CH
+type** per item and returns native column bytes instead of a `typoutput`
+string. The emitter keeps its columnar assembly for scalars but drops the
+composite text parsers.
+
+## Batch the DECODE (row-at-a-time → list of rows)
+
+Today the bridge is **per-tuple**: `resolve_pending_tuple` bundles one row's
+pending columns into a single `DECODE` request, and the decode pool calls it
+once per new/old tuple — one socket round trip per row. That's fine for
+scattered text but wrong-shaped for native output: a ClickHouse column is
+**many rows of one type laid out together**, so a row-at-a-time worker can't
+build a column and every row pays a round trip.
+
+Change the unit of resolution from one row to a **list of rows** (a
+column-major batch):
+
+- `DECODE` takes, per pending column, the target CH type + the raw on-disk
+  Datum for *every row in the batch* (nulls marked); the worker decodes down a
+  column and returns that column's native bytes (values + null map, plus
+  offsets for Array/Map) in one shot.
+- Resolve at **batch granularity, not tuple granularity**. The batcher already
+  accumulates per-table row batches (`InsertBatch`/chunk — see
+  [../emitter.md](../emitter.md)); hand a whole table-chunk to the bridge and
+  get back native columns ready to append to the block. This amortizes the
+  round trip over the batch and matches CH's columnar layout end-to-end.
+- Ordering/back-pressure: one in-flight batch request per table-chunk keeps the
+  existing seq/ack accounting; size the batch to the inserter's block size.
+
+## Scope / cost
+
+- **pgext** (`pgext/decode.c`, `worker.c`, `walshadow.h`): link the CH
+  serializer; `DECODE` request becomes column-batched (target CH type + a list
+  of raw Datums per column); response returns the serialized native column
+  (values + null map + offsets). Reimplement PG-Datum → CH-native for the
+  tier-3 matrix in C.
+- **Protocol/coupling**: a PG-side component must now know the destination CH
+  type (it knows nothing about ClickHouse today) — carry it in the request or a
+  negotiated intermediate.
+- **Rust** (`ops/bridge.rs`, `ops/oracle.rs`, `emit/…`): `Bridge::decode`
+  takes a batch and returns whole native columns; resolution moves from the
+  per-tuple call in `emit/pipeline/decode.rs` to batch granularity alongside
+  the batcher. Delete the emitter composite `ColumnBuf` variants + text parsers
+  + `NodeArena`; scalar `ColumnBuf`/`build_column` stays.
+- Estimated large + higher-risk (new C dep in the PGXS `.so`, native-format
+  edge cases across many types, batched-protocol reframe).
+
+## Does NOT solve greenfield
+
+This only changes *where* serialization happens; it still needs a live PG with
+the walshadow module to do the decoding. Greenfield has none until after
+bootstrap — that gap is [greenfield_oracle.md](greenfield_oracle.md), and the
+two are independent.
+
+## Alternative considered
+
+In-tree Rust decoders for PG array/hstore/jsonb on-disk binary (no bridge at
+all) — self-contained and greenfield-friendly, but re-implements PG's
+varlena/alignment/null-bitmap/JEntry formats byte-exact, which is its own large
+correctness surface. Native-from-the-extension reuses PG's own rendering and is
+preferred where a PG is available.

plans/oracle.md

@@ -36,14 +36,34 @@ Why these:
 disambiguation lives at type-OID level not body bytes (on-disk vs wire
 confusion surfaced here historically)
 
+## In-tree extension types
+
+PostGIS `geography`/`geometry` and pgvector `vector`/`halfvec` are rendered
+in-tree from their on-disk bytes by `render_ext_columns` (`src/ops/oracle.rs`),
+matched on `RelAttr.type_name` (dynamic OIDs). geography 2-D points →
+WKT `POINT(x y)` (`gserialized_point_to_wkt`); vector → `[a,b,c]`
+(`vector_to_text`). No bridge round-trip, so these resolve even where the
+shadow worker is unavailable (e.g. greenfield bootstrap).
+
 ## Bridge-routed Tier 3
 
-`jsonb`, arrays, `tsvector`, ranges, domains. Heap decoder emits
+`jsonb`, arrays, `hstore`, `tsvector`, ranges, domains. Heap decoder emits
 [`ColumnValue::PgPending { type_oid, raw }`](../src/heap_decoder.rs);
 [`resolve_pending_tuple`](../src/oracle.rs) collects every pending column of a
 tuple into one `DECODE` request to shadow's bridge worker, swaps `PgPending`
 for `Text` on each item that rendered
 
+## Shared resolve step
+
+`resolve_decoded_heap(oracle, attrs, decoded)` runs `render_ext_columns` then
+(if an oracle is present) `resolve_pending_tuple` over a heap's new/old tuples.
+Both the live decode pool (`emit/pipeline/decode.rs`) and the object-store /
+COPY backfill paths call it, so backfilled rows resolve identically to
+streamed ones. **Greenfield bootstrap is the exception**: it runs before the
+shadow/bridge exist, so bridge-routed types there resolve to nothing (in-tree
+types still work) — the fix is
+[future/greenfield_oracle.md](future/greenfield_oracle.md).
+
 Two alternatives considered (insert + select round-trip;
 `SELECT $1::bytea::<typ>::text`) require reconstructing wire format from
 on-disk format — same codec work the worker elides