From f724af648b36004ca31981318b94530b6e4a1125 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 13 Oct 2026 07:10:18 -0400 Subject: [PATCH 01/10] docs: reorganize ColdFront docs into the standard pgedge nav structure Splits usage.md's three operating modes into their own guides, moves shared operational topics into a new Administration nav section, and converts the Using ColdFront and Architecture Guides sections to real index.md pages so their nav headers open directly without a duplicate child link. --- .gitignore | 1 + CLAUDE.md | 6 +- CONTRIBUTING.md | 4 +- README.md | 60 +- docs/architecture_decoupled.md | 25 +- .../index.md} | 51 +- docs/architecture_tiered.md | 26 +- docs/architecture_vectors.md | 2 +- docs/caveats.md | 138 ++ docs/compaction.md | 6 +- docs/configuration.md | 107 ++ docs/distributed_setup.md | 198 +++ docs/index.md | 53 +- docs/installation.md | 151 +- docs/object_store.md | 453 +++-- docs/one_time_setup.md | 253 +++ docs/reading_writing.md | 39 + docs/source.md | 27 + docs/stylesheets/extra.css | 6 + docs/supported_types.md | 153 ++ docs/usage.md | 1460 ----------------- docs/usage_decoupled.md | 205 +++ docs/usage_partitioner.md | 312 ++++ docs/usage_tiered.md | 134 ++ docs/usage_vectors.md | 4 +- docs/using_coldfront/index.md | 32 + docs/vended_credentials.md | 51 + docs/walkthrough.md | 6 +- docs/walkthrough_decoupled.md | 4 +- docs/walkthrough_distributed.md | 4 +- docs/walkthrough_partitioner.md | 4 +- docs/walkthrough_tiered.md | 6 +- mkdocs.yml | 47 +- 33 files changed, 2176 insertions(+), 1852 deletions(-) rename docs/{architecture.md => architecture_guides/index.md} (96%) create mode 100644 docs/caveats.md create mode 100644 docs/configuration.md create mode 100644 docs/distributed_setup.md create mode 100644 docs/one_time_setup.md create mode 100644 docs/reading_writing.md create mode 100644 docs/source.md create mode 100644 docs/supported_types.md delete mode 100644 docs/usage.md create mode 100644 docs/usage_decoupled.md create mode 100644 docs/usage_partitioner.md create mode 100644 docs/usage_tiered.md create mode 100644 docs/using_coldfront/index.md create mode 100644 docs/vended_credentials.md diff --git a/.gitignore b/.gitignore index be965a52..a3944970 100644 --- a/.gitignore +++ b/.gitignore @@ -24,6 +24,7 @@ REPO_READY_AUDIT.md # MkDocs build output site/ +.venv-docs/ # pg_regress expected outputs — tracked; the global *.out would exclude them. !extension/coldfront/test/expected/*.out diff --git a/CLAUDE.md b/CLAUDE.md index b59dfccb..d71dab12 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -85,7 +85,7 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on - duckdb-iceberg's 2.0-line commit path is rewritten (a retry loop with `RefreshFromCatalog` firing reactively after a 409, rather than proactively before the first attempt). Our bakery patch's anchors (`RefreshExistingManifestList`, `GetTransactionRequest`) no longer exist there — the proactive pre-commit refresh needs reimplementing, not dropping. - Two of our interop patches (manifest-list `format-version`, lowercase parquet `file_format`) are still needed against the 2.0-line `main`. - The DuckDB temp-spill collision above (`duckdb#15173`, `pg_duckdb#887`) is fixed on the 2.0 line: spill files are instance-qualified, so the per-backend temp directory stops being a correctness requirement once the stack moves to 2.0. -- Partitioned `INSERT` and `CREATE TABLE ... PARTITIONED BY (...)` already work at our current `ICEBERG_REF`; `docs/architecture.md` and the extension's partition-column notice calling this unsupported are stale and should be corrected independently of the 2.0 move. +- Partitioned `INSERT` and `CREATE TABLE ... PARTITIONED BY (...)` already work at our current `ICEBERG_REF`; `docs/architecture_guides/index.md` and the extension's partition-column notice calling this unsupported are stale and should be corrected independently of the 2.0 move. **DuckDB 1.5.6 (`v1.5-variegata` repin)** (checked 2026-10-04 at head `5dcf5070`): moving to the branch head forces a DuckDB newer than 1.5.4 for the whole stack — pg_duckdb loads extensions only for its exact DuckDB version, and pg_duckdb PR #1071 bumps to 1.5.5 while nothing upstream has tried 1.5.6 yet. All five `docker/iceberg-*-v15.patch` files still apply cleanly at the head and none of their fixes have landed upstream; `referenced_data_file` is still reader-only and `RETURNING` on a cold `INSERT`/`UPDATE`/`DELETE`/`MERGE` is still refused. The gains are vended-credential refresh, equality-delete scans with extra columns, truncated UTF-8 string bounds, an Azure SAS endpoint-suffix fix, and a `write.metadata.metrics` backport — treat a repin as a DuckDB stack move (base republish, full matrix), not a quick ref bump. @@ -109,8 +109,8 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on ## Documentation - User docs live in `docs/` as a MkDocs site (pgEdge MkDocs structure); `README.md` is the GitHub landing/TOC. Build with `mkdocs build --strict` — it must pass with no warnings. - Keep `README.md` and `docs/index.md` near-identical, EXCEPT `docs/index.md` is user-facing and omits the developer-specific content that belongs only in a GitHub README: CI badges, build-from-source internals, the project-structure tree, dependency/build tables, and author/contributor info. -- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Documentation table (one terse row per guide), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README table, and the list in `docs/architecture.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. +- Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Documentation table (one terse row per guide), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README table, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. - Internal/scratch docs (bench, experimental, hardening, comparison, backlog, scale-test, E2E) are gitignored and NEVER pushed to GitHub; never reference them from published docs. - ARCHITECTURE docs are reference material, not work logs: brief, present-tense, no history ("we used to…", version churn), no rejected-alternative narration. Keep the *why* (rationale that explains the design); cut the journey. -- One home per topic, no duplication across docs: usage/modes/config/CLI and one-time setup in `docs/usage.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture*.md`. When moving a section, fix every cross-anchor that pointed at it. +- One home per topic, no duplication across docs: shared day-to-day usage and the partition CLI in `docs/using_coldfront/index.md`; each mode's own guide in `docs/usage_tiered.md`, `docs/usage_decoupled.md`, and `docs/usage_partitioner.md`; PostgreSQL/ColdFront config in `docs/configuration.md`; Lakekeeper install/config/bootstrap in `docs/one_time_setup.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture_guides/index.md` and `docs/architecture_tiered.md`/`docs/architecture_decoupled.md`/`docs/architecture_vectors.md`. When moving a section, fix every cross-anchor that pointed at it. - Tracked build-journal docs (`DUCKDB_1.5_PATCHED.md`, `DUCKDB_1.5_UNPATCHED.md`) are kept as history and never deleted, but they are not part of the docs site: never link them from `README.md` or `docs/`. Point build internals at the real recipe, `docker/Dockerfile.duckdb15-base`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ae9ef0d8..c978262a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,8 +5,8 @@ Thanks for your interest in improving ColdFront. ## Getting started - Build and run the stack: [Installation](docs/installation.md). -- Day-to-day usage and the two operating modes: [Usage](docs/usage.md). -- Architecture: [Architecture](docs/architecture.md) (plus the tiered/decoupled +- Day-to-day usage and the two operating modes: [Usage](docs/using_coldfront/index.md). +- Architecture: [Architecture](docs/architecture_guides/index.md) (plus the tiered/decoupled deep dives). ## Development workflow diff --git a/README.md b/README.md index c83e1e17..810369e3 100644 --- a/README.md +++ b/README.md @@ -16,11 +16,11 @@ The ColdFront documentation consists of the following guides: - [Building ColdFront from Source](docs/installation.md) - [Setting Up an Object Store](docs/object_store.md) - Architecture - - [Architecture Overview](docs/architecture.md) + - [Architecture Overview](docs/architecture_guides/index.md) - [Tiered Mode](docs/architecture_tiered.md) - [Decoupled Mode](docs/architecture_decoupled.md) - [Vector Storage](docs/architecture_vectors.md) -- [Using ColdFront](docs/usage.md) +- [Using ColdFront](docs/using_coldfront/index.md) - [Storing and Searching Embeddings](docs/usage_vectors.md) - [Compacting the Cold Tier](docs/compaction.md) - Developer Resources @@ -133,9 +133,10 @@ settings above. The `import` command takes a deployment YAML, modeled on [config.example.yaml](config.example.yaml), and writes it into the server once. ColdFront has no configuration file. A YAML passed to any later run is checked against the server, and the run takes only `postgres.dsn` from the file. -For every setting, see the [One-Time Setup](docs/usage.md#one-time-setup) and -[Tuning Knobs](docs/usage.md#tuning-knobs) sections of the Using ColdFront -guide. +For every setting, see the +[Configuring Lakekeeper](docs/one_time_setup.md) guide and the +[Configuring PostgreSQL](docs/configuration.md#configuring-postgresql) +section of the Configuring ColdFront guide. ## Quickstart @@ -147,7 +148,7 @@ docker compose --profile local-store up -d --build ``` Bootstrap Lakekeeper and create a warehouse (see the one-time setup in the -[Using ColdFront](docs/usage.md) guide), then create a table in psql: +[Using ColdFront](docs/using_coldfront/index.md) guide), then create a table in psql: ```sql CREATE EXTENSION IF NOT EXISTS pg_duckdb; @@ -167,28 +168,33 @@ than creating it: `coldfront.adopt_iceberg_table()` reads its schema from the catalog and gives it the same wrapper view and registry row, read-only unless you pass `p_writable => true`. `coldfront.release_iceberg_table()` hands the table back with the Iceberg table untouched. See -[Adopting a Table That Already Exists in the Catalog](docs/usage.md#adopting-a-table-that-already-exists-in-the-catalog). +[Adopting a Table That Already Exists in the Catalog](docs/usage_decoupled.md#adopting-a-table-that-already-exists-in-the-catalog). To remove a table again, `coldfront.drop_iceberg_table()` unregisters it and drops the Iceberg table, deleting the stored objects only when asked to. See -[Dropping an Iceberg Table](docs/usage.md#dropping-an-iceberg-table-both-modes). +[Dropping an Iceberg Table](docs/usage_decoupled.md#dropping-an-iceberg-table-both-modes). For compliance environments that cannot store an object-store credential, `coldfront.set_storage_secret_vended()` runs with no credential in the database: Lakekeeper issues short-lived per-table credentials at access time. -See [Vended Credentials](docs/usage.md#vended-credentials). +See [Vended Credentials](docs/vended_credentials.md). ## Using ColdFront -The Quickstart covers a decoupled table from start to finish. The -[Using ColdFront](docs/usage.md) guide covers both modes in depth, the -standalone partition manager and its CLI, the storage backends, and the -distributed setup; the [walkthrough](docs/walkthrough.md) demos run each mode -on a sample table. +The Quickstart walks through a decoupled table from start to finish. +[Using ColdFront in Tiered Mode](docs/usage_tiered.md), +[Using ColdFront in Decoupled Mode](docs/usage_decoupled.md), and +[Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) +document each mode in depth; the [Using ColdFront](docs/using_coldfront/index.md) guide +helps you pick one, and the [walkthrough](docs/walkthrough.md) demos run +each mode on a sample table. Operational topics shared across modes - +caveats, vended credentials, reading and writing, supported types, and +distributed (mesh) setup - are documented under Administration. ## Documentation -The following table lists the ColdFront guides and what each one covers: +The following table lists the ColdFront guides and what each one +documents: | Doc | Contents | |---|---| @@ -197,12 +203,22 @@ The following table lists the ColdFront guides and what each one covers: | [Decoupled mode demo](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | | [Partitioner demo](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | | [Distributed demo](docs/walkthrough_distributed.md) | Points two PostgreSQL nodes at one shared lake. | -| [Embeddings](docs/usage_vectors.md) | Covers storing and searching embeddings with the pgvector interface. | -| [Usage](docs/usage.md) | Covers day-to-day use: both modes plus the standalone partition manager, one-time setup, reading and writing, supported types, the partition CLI, storage backends, distributed (mesh) setup, and tuning. | -| [Installation](docs/installation.md) | Covers building from source (Docker or bare-metal), and testing and CI. | -| [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running on cloud S3 (virtual-hosted), end to end. | -| [Compaction](docs/compaction.md) | Covers cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | -| [Architecture](docs/architecture.md) | Describes the shared architecture and core mechanics. | +| [Embeddings](docs/usage_vectors.md) | Documents storing and searching embeddings with the pgvector interface. | +| [Usage](docs/using_coldfront/index.md) | Compares the three operating modes and links to each mode's guide. | +| [Tiered mode usage](docs/usage_tiered.md) | Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. | +| [Decoupled mode usage](docs/usage_decoupled.md) | Documents tables that live entirely in Iceberg from the first row, and dropping or releasing a registered table. | +| [Standalone partitioner usage](docs/usage_partitioner.md) | Documents managing PostgreSQL partitions with no cold tier at all, and the partition CLI shared with tiered mode. | +| [Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other sharp edges shared between tiered and decoupled tables. | +| [Vended Credentials](docs/vended_credentials.md) | Documents running with no stored object-store credential, for AWS S3 and Azure ADLS Gen2. | +| [Reading and Writing](docs/reading_writing.md) | Documents the SQL surface shared between tiered and decoupled tables. | +| [Supported Column Types](docs/supported_types.md) | Documents the PostgreSQL column types, including arrays, that a cold tier supports. | +| [Distributed Setup](docs/distributed_setup.md) | Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. | +| [Installation](docs/installation.md) | Documents installing from a package or building from source (Docker or bare-metal), and testing and CI. | +| [Configuring ColdFront](docs/configuration.md) | Documents configuring PostgreSQL and ColdFront's own configuration. | +| [Configuring Lakekeeper](docs/one_time_setup.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | +| [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running end to end, for any of four supported backends: S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. | +| [Compaction](docs/compaction.md) | Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | +| [Architecture](docs/architecture_guides/index.md) | Describes the shared architecture and core mechanics. | | [Architecture: tiered](docs/architecture_tiered.md) | Describes tiered mode (hot PG plus cold Iceberg) in depth. | | [Architecture: decoupled](docs/architecture_decoupled.md) | Describes decoupled (iceberg-only) mode in depth. | | [Architecture: vectors](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | @@ -239,7 +255,7 @@ For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the `duckdb.postgres_role` default that the image sets up, and how least privilege holds across a Spock mesh - see -[Architecture: non-superuser app roles](docs/architecture.md#non-superuser-app-roles-least-privilege). +[Architecture: non-superuser app roles](docs/architecture_guides/index.md#non-superuser-app-roles-least-privilege). ## Caveats diff --git a/docs/architecture_decoupled.md b/docs/architecture_decoupled.md index 952d4729..c225b608 100644 --- a/docs/architecture_decoupled.md +++ b/docs/architecture_decoupled.md @@ -6,7 +6,7 @@ archiver. PostgreSQL becomes a stateless compute front-end. Lakekeeper and the underlying object store own the storage. Decoupled mode shares the same codebase, docker stack and extension as tiered mode. The shared mechanics - pg_duckdb Iceberg I/O, the rewrite hook, the bakery protocol, the registry - -are in [architecture.md](architecture.md); tiered mode is in +are in [architecture.md](architecture_guides/index.md); tiered mode is in [architecture_tiered.md](architecture_tiered.md). This document covers what is specific to decoupled mode. @@ -162,7 +162,7 @@ which would lose precision or identity: - custom enums, `xml`, `tsvector`/`tsquery`, range types, and multirange types. - composite types. - arrays of `timestamptz`, `jsonb`, `json`, `interval` or a vector type, and - arrays of arrays; [Arrays](usage.md#arrays) gives the reasons. + arrays of arrays; [Arrays](supported_types.md#arrays) gives the reasons. - pgvector's `sparsevec`. - any type not enumerated above. @@ -201,8 +201,8 @@ DELETE FROM events WHERE id = 1; The fourth argument, `p_partition_cols`, is the table's Iceberg partitioning as a `text[]` of `PARTITIONED BY` terms, passed to DuckDB as written; the terms, and how a term with a comma or a quoted name is written in the array literal, -are in the [Mode 2](usage.md#mode-2-decoupled-iceberg-only) section of the -Using ColdFront guide. DuckDB refuses an unknown transform, a bad argument or a +are in the [Using ColdFront in Decoupled Mode](usage_decoupled.md) guide. +DuckDB refuses an unknown transform, a bad argument or a column outside the schema at `CREATE TABLE`. The one check DuckDB leaves to the first `INSERT`, a time transform on a column that is not a timestamp or date (for `hour`, not a timestamp), `coldfront._partition_clause()` makes at the @@ -517,7 +517,7 @@ Three properties are worth knowing: ## ACID Model -(This section summarizes material from [architecture.md](architecture.md) +(This section summarizes material from [architecture.md](architecture_guides/index.md) §Concurrency and §Known Limitations applied to the decoupled scenario.) The following table shows the status of each ACID property in decoupled mode: @@ -724,8 +724,8 @@ pointing at the same Lakekeeper endpoint and S3 bucket: - For DDL, Spock's `ddl_sql` repset replicates `CREATE/ALTER/DROP` of the wrapper view, and the `default` repset replicates the `coldfront.tiered_views` registry row (see - [Distributed Setup](usage.md#distributed-setup-3-node-mesh-decoupled-mode)), - so one node provisions the table and replication enables every peer's hook. + [Distributed Setup](distributed_setup.md)), so one node provisions the + table and replication enables every peer's hook. A column `ALTER TABLE` reaches the peers the same way: the node that runs it changes the shared Iceberg table, and its rebuilt view and registry row replicate. @@ -792,8 +792,7 @@ The call is idempotent. It puts every ColdFront table that replicates by value in the node's replication sets: the two bakery tables, the registry and the watermark, the storage secret, the lifecycle config and the vector routing state (the list and the reason for each table are in the -[Distributed Setup](usage.md#what-coldfrontensure_replicated-does) section of -the Using ColdFront guide). +[Distributed Setup](distributed_setup.md#what-coldfrontensure_replicated-does) guide). If it has not run on a peer, that peer's ack `INSERT`s are local-only and never replicate back to the originating writer: every claim on the originator waits at the ack barrier for an ack that never arrives. @@ -847,16 +846,16 @@ Decoupled mode has the following limitations: - A transaction block that writes a decoupled table cannot also write a PostgreSQL table unless `duckdb.unsafe_allow_mixed_transactions` is on: pg_duckdb refuses the PostgreSQL write, or the `COMMIT` when that write came - first. The [Caveats](usage.md#caveats) describe the rule and the risk of - setting the parameter. + first. The [Caveats](caveats.md) describe the rule and the risk of setting + the parameter. ## Next Steps To go further with ColdFront, consult the following documents: -- The [Architecture](architecture.md) overview describes the mechanics both +- The [Architecture](architecture_guides/index.md) overview describes the mechanics both modes share. - The [Formal Verification](formal/README.md) document describes the TLA+ model of the bakery protocol and how to check it. -- The [Using ColdFront](usage.md) guide covers creating decoupled tables and +- The [Using ColdFront](using_coldfront/index.md) guide covers creating decoupled tables and setting up a distributed mesh. diff --git a/docs/architecture.md b/docs/architecture_guides/index.md similarity index 96% rename from docs/architecture.md rename to docs/architecture_guides/index.md index 00500b4e..f1cdb674 100644 --- a/docs/architecture.md +++ b/docs/architecture_guides/index.md @@ -7,10 +7,10 @@ rewrites reads and writes to hit the right tier, and **all Iceberg I/O goes through `pg_duckdb` running in-process inside PostgreSQL** - no external query engine, no Go Iceberg libraries. -For mode-specific design, see [architecture_tiered.md](architecture_tiered.md) +For mode-specific design, see [architecture_tiered.md](../architecture_tiered.md) (hot PG + cold Iceberg) · -[architecture_decoupled.md](architecture_decoupled.md) (all-Iceberg) · -[architecture_vectors.md](architecture_vectors.md) (vector storage). +[architecture_decoupled.md](../architecture_decoupled.md) (all-Iceberg) · +[architecture_vectors.md](../architecture_vectors.md) (vector storage). ## Operating Modes and Topologies @@ -33,7 +33,7 @@ path in the hook takes, mostly through `_exec_iceberg_with_claim`. Decoupled mode always classifies as `TIER_COLD` and never reaches `emit_hot`; vanilla and mesh differ only in how that claim serializes cold writes. This document covers the shared mechanics and the tiered path; see -[architecture_decoupled.md](architecture_decoupled.md) for the decoupled mode's +[architecture_decoupled.md](../architecture_decoupled.md) for the decoupled mode's ACID model and distributed scaling story. ### Read Target: Primary or Physical Standby @@ -104,9 +104,9 @@ The following table describes each component, its role, and its license: How rows move through this depends on the storage mode: the tiered hot heap + archiver + `UNION ALL` data-flow is in the -[Data Flow](architecture_tiered.md#data-flow) section of the Tiered Mode page; +[Data Flow](../architecture_tiered.md#data-flow) section of the Tiered Mode page; the all-Iceberg flow is in -[architecture_decoupled.md](architecture_decoupled.md). +[architecture_decoupled.md](../architecture_decoupled.md). ## Core Mechanics: pg_duckdb @@ -211,7 +211,7 @@ with the privilege it requires. `_exec_iceberg_with_claim` deliberately stays `SECURITY INVOKER` - it runs the caller's cold DML, which must execute as the caller. This `SECURITY DEFINER` setting is **protocol-neutral**: it changes the PG execution privilege, not the claim/ack/lock/ticket protocol, re-verified -against [the TLA+ model](formal/Bakery.tla) (all safe configs pass; the race +against [the TLA+ model](../formal/Bakery.tla) (all safe configs pass; the race config still violates `NoLakekeeperConflict`). See README "Security"; the setting is asserted by the journey's @@ -256,9 +256,8 @@ The same parse-analyze hook prepares a `SELECT` that DuckDB will run, wherever in the statement the view is named (a CTE, a sub-select, a set-operation branch): `date_bin`, the `::jsonb` cast, `jsonb_array_length` and the JSON builders (`jsonb_build_object`, `jsonb_agg` and their `json_` twins) are -rewritten into spellings both engines accept (see the -[Supported Column Types](usage.md#supported-column-types) section of the Using -ColdFront guide). A +rewritten into spellings both engines accept (see +[Supported Column Types](../supported_types.md)). A `planner_hook` folds bound parameters into such a read before pg_duckdb plans it when a parameter sits where DuckDB cannot type a placeholder (a direct argument of a pg_duckdb function, any argument of a table function); the plan @@ -267,8 +266,7 @@ above any custom plan, so under the plan cache's cost-based selection the read is planned from its values on every execution. `plan_cache_mode = force_generic_plan` bypasses that selection and picks the value-less generic plan, which fails with `only works with DuckDB execution` -(see the [Supported Column Types](usage.md#supported-column-types) section of -the Using ColdFront guide). +(see [Supported Column Types](../supported_types.md)). The following table maps each operation to its interface and routing path: @@ -287,12 +285,12 @@ How the hook splits a write is mode-specific: - In tiered mode, the hook routes by the partition-column watermark - hot heap vs cold Iceberg, with dual-tier writes for ambiguous predicates. See - [Transparent `INSERT`](architecture_tiered.md#transparent-insert) and - [Transparent `UPDATE`/`DELETE`](architecture_tiered.md#transparent-updatedelete) + [Transparent `INSERT`](../architecture_tiered.md#transparent-insert) and + [Transparent `UPDATE`/`DELETE`](../architecture_tiered.md#transparent-updatedelete) in the Tiered Mode page. - In decoupled mode, the hook always classifies `TIER_COLD`, so every write is a single-tier Iceberg write. See - [architecture_decoupled.md](architecture_decoupled.md). + [architecture_decoupled.md](../architecture_decoupled.md). ### Cold-Tier DML from Inside PL/pgSQL (Functions, DO Blocks, Triggers) @@ -345,7 +343,7 @@ ColdFront coordinates concurrent writes across the cluster as follows: Snowflake ticket via the Spock-replicated `coldfront.claims` table and waits for its turn before issuing the iceberg commit. The bakery produces no 409s and needs no retry in the application. See the - [Concurrency](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol) + [Concurrency](../architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol) section of the Decoupled Mode page for the full design and benchmarks. ### Cold-Write Strategy: Stock vs Patched duckdb-iceberg @@ -361,7 +359,7 @@ The following table shows the two orderings: | Ordering | Applies to | Behavior | |---|---|---| | stock (default) | every write on stock upstream duckdb-iceberg; a `DELETE`, `UPDATE` or `MERGE` (the cross-tier move and `vector_train`'s assignment included) on every build | The writer takes the serializer first, *then* uploads parquet and commits inside it. On an unpatched binary this is what keeps a second writer from capturing a stale parent. A write that references rows the table holds needs it on every build: its position deletes name data files by path, and a compaction or another such write committing between its scan and its commit would rewrite or remove those files, after which the deletes apply to nothing and the rows come back. | -| async (both GUCs `on`) | appends (`INSERT`, `COPY`, the archiver's exports, the tiered `INSERT`'s cold sink) on the patched build (`iceberg-bakery-aware-commit-refresh-v15.patch`) | The statement uploads its parquet and queues the claim; the extension's transaction callback takes it at PRE_COMMIT, just before pg_duckdb's own PRE_COMMIT callback commits the Iceberg transaction, and releases it at COMMIT. Uploads overlap on one node and across nodes, an open transaction blocks no other writer, and only the short commit POST is serialized. A new data file conflicts with nothing committed in between. An append to a clustered table is checked at commit against the live centroid generation and fails with `serialization_failure` if a retrain replaced it (see [architecture_vectors.md](architecture_vectors.md)). | +| async (both GUCs `on`) | appends (`INSERT`, `COPY`, the archiver's exports, the tiered `INSERT`'s cold sink) on the patched build (`iceberg-bakery-aware-commit-refresh-v15.patch`) | The statement uploads its parquet and queues the claim; the extension's transaction callback takes it at PRE_COMMIT, just before pg_duckdb's own PRE_COMMIT callback commits the Iceberg transaction, and releases it at COMMIT. Uploads overlap on one node and across nodes, an open transaction blocks no other writer, and only the short commit POST is serialized. A new data file conflicts with nothing committed in between. An append to a clustered table is checked at commit against the live centroid generation and fails with `serialization_failure` if a retrain replaced it (see [architecture_vectors.md](../architecture_vectors.md)). | For every write but a clustered table's append, the code path and the application-visible behavior are identical, so the GUCs are a performance knob; @@ -438,7 +436,7 @@ to be usable on a peer is covered next. The tiered-specific cross-node behavior - what replicates so a tiered table is usable on every peer, and why both the registry and the watermark join the replication set - is in the -[Tiered Tables in a Spock Mesh](architecture_tiered.md#tiered-tables-in-a-spock-mesh) +[Tiered Tables in a Spock Mesh](../architecture_tiered.md#tiered-tables-in-a-spock-mesh) section of the Tiered Mode page. ### Registry Keying: By Name, Not OID @@ -463,7 +461,7 @@ node-independent, so the registry row is identical on every node and the replication set copies it by value (an OID is node-local and could not be). That is what makes cross-node tiered tables work with no per-node re-resolution - see the -[Tiered Tables in a Spock Mesh](architecture_tiered.md#tiered-tables-in-a-spock-mesh) +[Tiered Tables in a Spock Mesh](../architecture_tiered.md#tiered-tables-in-a-spock-mesh) section of the Tiered Mode page. Lower-level operations that genuinely need an OID - catalog lookups, the DDL @@ -523,14 +521,14 @@ mode (the archiver those with a `hot_period`, the partitioner those without), and exit with an error when they find none; tables enter it via `register`, or from a YAML `archiver.tables` list fed to `import`. Both expose a management CLI - `register` / `list` / `set` / `remove` / `import` / `export` - documented -in [usage.md](usage.md#managing-partitioned-tables-cli). +in [usage_partitioner.md](../usage_partitioner.md#managing-partitioned-tables-cli). ## Known Limitations These apply to both storage modes. Tiered-only limitations (cold `RETURNING`, dual-tier command tag, crash-safety of permissive writes, partition-scheme constraints, autovacuum-vs-cutover) are in the -[Tiered-Specific Limitations](architecture_tiered.md#tiered-specific-limitations) +[Tiered-Specific Limitations](../architecture_tiered.md#tiered-specific-limitations) section of the Tiered Mode page. The cross-cutting limitations are: @@ -733,7 +731,8 @@ and the directory and reporting what that freed. That sweep runs once per session and costs single-digit milliseconds per spill file, so even a directory abandoned with 600 GB in it (about 600 files at DuckDB's file size) is a couple of seconds on the one statement that finds it. See the -[Tuning Knobs](usage.md#tuning-knobs) section of the Using ColdFront guide. +[Configuring PostgreSQL](../configuration.md#configuring-postgresql) section +of the Configuring ColdFront guide. The PID is what makes that reclaim possible, and is why ColdFront does not apply pg_duckdb#887, which names the directory with a random uuid and removes @@ -801,11 +800,11 @@ with fleet benchmarking rather than as a patch ColdFront maintains. To go further with ColdFront, consult the following documents: -- The [Tiered Mode](architecture_tiered.md) deep dive describes the archive +- The [Tiered Mode](../architecture_tiered.md) deep dive describes the archive pipeline, transparent DML, and tiered tables in a Spock mesh. -- The [Decoupled Mode](architecture_decoupled.md) deep dive describes +- The [Decoupled Mode](../architecture_decoupled.md) deep dive describes iceberg-only tables, their ACID model, and the bakery protocol. -- The [Vector Storage](architecture_vectors.md) deep dive describes how +- The [Vector Storage](../architecture_vectors.md) deep dive describes how embeddings are stored, assigned to clusters, and searched. -- The [Using ColdFront](usage.md) guide covers day-to-day use of both modes. +- The [Using ColdFront](../using_coldfront/index.md) guide covers day-to-day use of both modes. diff --git a/docs/architecture_tiered.md b/docs/architecture_tiered.md index e827449b..a20c6b54 100644 --- a/docs/architecture_tiered.md +++ b/docs/architecture_tiered.md @@ -1,4 +1,4 @@ -# ColdFront - Tiered Operating Mode +# Tiered Operating Mode Tiered mode keeps recent rows in the PostgreSQL heap and archives older rows to Apache Iceberg, presenting both as one table through a `UNION ALL` view. An @@ -6,7 +6,7 @@ archiver moves rows hot→cold on a cron. This document covers the **tiered-specific** design. The shared mechanics - pg_duckdb Iceberg I/O, the rewrite hook, the bakery protocol, the registry, DDL -handling, infrastructure - live in [architecture.md](architecture.md); the +handling, infrastructure - live in [architecture.md](architecture_guides/index.md); the all-Iceberg alternative is [architecture_decoupled.md](architecture_decoupled.md). @@ -66,7 +66,7 @@ A single Go binary runs via cron. The binary converts an existing partitioned table into a tiered table on first run, then manages the ongoing lifecycle. The archiver is a thin SQL orchestrator - no DuckDB/Iceberg/Arrow Go libraries; all Iceberg I/O goes through `pg_duckdb` (see the -[Core Mechanics](architecture.md#core-mechanics-pg_duckdb) section of the +[Core Mechanics](architecture_guides/index.md#core-mechanics-pg_duckdb) section of the Architecture overview). ### Prerequisites @@ -76,7 +76,7 @@ The archiver requires the following before its first run: - PostgreSQL 16+ with pg_duckdb. - Lakekeeper, bootstrapped with a warehouse. - a configured storage secret (see the - [Session Setup](architecture.md#session-setup) section of the Architecture + [Session Setup](architecture_guides/index.md#session-setup) section of the Architecture overview). - static `s3:` or `azure:` credentials in the archiver config, unless the cold store uses vended credentials. @@ -170,7 +170,7 @@ six-phase pipeline: 2. The bulk export copies the partition PG → Iceberg under a captured snapshot, using the temp table bridge (see the - [Temp Table Bridge](architecture.md#temp-table-bridge-pg-iceberg) section of + [Temp Table Bridge](architecture_guides/index.md#temp-table-bridge-pg-iceberg) section of the Architecture overview) and a single bakery-claimed Iceberg `INSERT`. Each cycle has already created the Iceberg namespace and table, if missing, before the per-partition loop. @@ -281,7 +281,7 @@ the flat path. ## Transparent `INSERT` The `post_parse_analyze_hook` (see the -[Application Interface](architecture.md#application-interface) section of the +[Application Interface](architecture_guides/index.md#application-interface) section of the Architecture overview) intercepts `INSERT` on a registered tiered view and rewrites it into a single statement that splits the input by the partition-column watermark. The hot half is always a plain set-based @@ -554,7 +554,7 @@ them. Strict mode avoids this path entirely. A cold-only `UPDATE` or `DELETE` sets nothing, so a transaction block that holds one cannot also write a PostgreSQL table unless the session sets the -parameter itself; the [Caveats](usage.md#caveats) state the rule. +parameter itself; the [Caveats](caveats.md) state the rule. ### Strict (`off`) @@ -566,7 +566,7 @@ applications to supply a tier-deterministic WHERE clause. ## Tiered Tables in a Spock Mesh The bakery protocol that serializes cold writes cluster-wide is described in -the [Concurrency](architecture.md#concurrency-and-pgedge-spock-deployments) +the [Concurrency](architecture_guides/index.md#concurrency-and-pgedge-spock-deployments) section of the Architecture overview. This section covers what is specific to a *tiered* table across a mesh. @@ -584,7 +584,7 @@ So alongside the bakery substrate (`coldfront.claims` / `coldfront.claim_acks`), **both `coldfront.tiered_views` and `coldfront.archive_watermark` are in the Spock replication set**: `coldfront.ensure_replicated()`, the one-time per-node mesh step described in -[usage.md](usage.md#what-coldfrontensure_replicated-does), adds them with the +[distributed_setup.md](distributed_setup.md#what-coldfrontensure_replicated-does), adds them with the rest. The archiver runs on one node and registers a table, its view and its registry row, in one transaction, so a peer applies them together and only gets these rows by replication; without `tiered_views` a peer cannot read the cold @@ -593,7 +593,7 @@ tier, and `INSERT`/`UPDATE`/`DELETE`/DDL-blocking stop recognizing the view. Both tables are **name-keyed** - `tiered_views` by `(schema_name, relname)`, `archive_watermark` by `table_name` - so each row replicates verbatim and correct on every node, with no OID divergence to reason about. See the -[Registry Keying](architecture.md#registry-keying-by-name-not-oid) section of +[Registry Keying](architecture_guides/index.md#registry-keying-by-name-not-oid) section of the Architecture overview. ## Partition Scheme Compatibility @@ -710,7 +710,7 @@ not make deletes less expensive. These are specific to the dual-tier model. Cross-cutting limitations (the planner-level takeover, jsonb-as-json, single-node execution, S3 compatibility, one-time secret setup) are in the -[Known Limitations](architecture.md#known-limitations) section of the +[Known Limitations](architecture_guides/index.md#known-limitations) section of the Architecture overview. The dual-tier model has the following limitations: @@ -763,9 +763,9 @@ The dual-tier model has the following limitations: To go further with ColdFront, consult the following documents: -- The [Architecture](architecture.md) overview describes the mechanics both +- The [Architecture](architecture_guides/index.md) overview describes the mechanics both modes share. - The [Decoupled Mode](architecture_decoupled.md) deep dive describes the bakery protocol that serializes cold writes. -- The [Using ColdFront](usage.md) guide covers tiering a table and managing it +- The [Using ColdFront](using_coldfront/index.md) guide covers tiering a table and managing it with the partition CLI. diff --git a/docs/architecture_vectors.md b/docs/architecture_vectors.md index 36678457..16856ac0 100644 --- a/docs/architecture_vectors.md +++ b/docs/architecture_vectors.md @@ -566,5 +566,5 @@ To go further with ColdFront, consult the following documents: searching vector columns. - The [Compaction](compaction.md) guide covers the maintenance that restores cluster order. -- The [Architecture](architecture.md) overview describes the mechanics both +- The [Architecture](architecture_guides/index.md) overview describes the mechanics both modes share. diff --git a/docs/caveats.md b/docs/caveats.md new file mode 100644 index 00000000..9461725a --- /dev/null +++ b/docs/caveats.md @@ -0,0 +1,138 @@ +# Understanding ColdFront Caveats + +Keep the following caveats in mind when running either mode: + +- `jsonb` reads come back as `json`. The `->`/`->>` operators work, and + ColdFront translates the `::jsonb` cast, `jsonb_array_length`, + `jsonb_build_object` / `jsonb_agg` and `date_bin`; other jsonb operators and + functions are unsupported on cold or cross-tier reads. A hot-only read of the + view runs in PostgreSQL with full jsonb (see + [Supported Column Types](supported_types.md)). +- Cross-tier isolation differs from PG-native isolation. The cold tier keeps + one Iceberg snapshot for the whole transaction, even at READ COMMITTED, so a + commit by another session after the transaction's first cold read stays + invisible until the transaction ends. The hot tier follows PostgreSQL's own + isolation level, so at READ COMMITTED a later statement can see new hot rows + but not new cold ones. Within one transaction, a read of a tiered or + decoupled table sees the transaction's own writes on both tiers. The held + snapshot is also why the compactor's `--expire-older-than` must stay longer + than the longest transaction that reads the table (see + [compaction.md](compaction.md)). +- One transaction block cannot hold both a PostgreSQL write and a cold-tier + write unless `duckdb.unsafe_allow_mixed_transactions` is on. pg_duckdb + refuses the pair ("Writing to DuckDB and Postgres tables in the same + transaction block is not supported"), at the PostgreSQL write when it comes + second and at `COMMIT` when it came first. A cold-tier write here is a + tiered `UPDATE` or `DELETE` whose `WHERE` bounds the cold tier, or any write + to a decoupled table. The statements that write both tiers themselves (a + dual-tier `UPDATE` or `DELETE`, a tiered `INSERT` with cold rows, a + cross-tier move) set the parameter `LOCAL` for the rest of their + transaction, so the block they ran in accepts later writes of either kind. + You can set it yourself, `SET LOCAL duckdb.unsafe_allow_mixed_transactions = + on`, at your own risk: pg_duckdb commits the Iceberg snapshot at + `PRE_COMMIT`, so a backend crash between that and the PostgreSQL commit + record keeps the cold write and loses the PostgreSQL writes. +- In decoupled mode, pg_duckdb commits the Iceberg snapshot at PRE_COMMIT, so a + backend crash after that but before the PG commit record leaves the Iceberg + write committed and the PG side lost. A crash after the Parquet upload but + before the commit POST leaves unreferenced objects, which the compactor's + orphan pass reclaims. +- In decoupled mode, concurrent writes from multiple PG nodes are serialized + PG-side by the bakery protocol: every iceberg-only `INSERT` goes through + `coldfront._exec_iceberg_with_claim`, which holds a globally-ordered + Snowflake ticket and waits for its turn before committing to Lakekeeper. + There are no 409 conflicts and no app-level retries. The protocol is + Lamport-1978 mutex with the Ricart-Agrawala (1981) deferred-reply + optimization; claims and acks replicate as Spock rows and it stays safe under + Spock's asymmetric apply (modeled in + [docs/formal/Bakery.tla](https://github.com/pgEdge/ColdFront/blob/main/docs/formal/Bakery.tla)). + The bakery requires the `snowflake` extension, the `coldfront.loopback_dsn` + GUC (the node's loopback, a unix-socket DSN), and a one-time + `SELECT coldfront.ensure_replicated()` call on every node after Spock mesh + setup; see + [architecture_decoupled.md](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). + Sync-rep is **not** required. The throughput ceiling is Lakekeeper's commit + rate, not the writer count. +- For direct table access, `_events` is the hot heap (tiered mode only). + `ice.public.` is the Iceberg table - only addressable via + `iceberg_scan(...)` or `duckdb.raw_query('… ice.… …')`, never via + PG-native 3-part names. +- A tiered `INSERT` streams its cold rows: the cold projection runs in + PostgreSQL, in a second session the DuckDB `postgres` extension opens from + `coldfront.local_pg_dsn`, and its rows go straight into the Iceberg writer + in one pass. An omitted IDENTITY column takes `nextval()` on the hot table's + sequence, so cold ids share it with the hot side, and an omitted column with + a DEFAULT takes it. The hot rows are one set-based `INSERT`. The second + session reads the source at the statement's own snapshot, so both tiers see + one state of the tables whatever another session commits meanwhile, and a + REPEATABLE READ transaction reads its snapshot on both tiers. A source the + stream cannot read the same way twice (a volatile or stable function such as + `now()`, a temporary table, a `WITH` entry, a table the transaction has + written, a SERIALIZABLE transaction, or a second table-reading `INSERT` in + one READ COMMITTED transaction) goes through + `coldfront._cold_sink` instead, which renders each row in plpgsql and writes + Iceberg in batches of `coldfront.cold_write_batch_size` rows, as does a + server without `coldfront.local_pg_dsn`. The statement's `cold_rows` column + is `NULL` when the cold rows streamed. A value that reaches the second + session or DuckDB as text is rendered in a form every reader takes as the + same value, whatever the caller's `DateStyle`, `IntervalStyle` or + `extra_float_digits`. +- `COPY FROM` reads the rows with PostgreSQL's `COPY` reader and writes + them through that same `INSERT` path, `coldfront.cold_write_batch_size` rows + per `INSERT`. The format options are the reader's, and a supplied value for a + `GENERATED ALWAYS` identity column is kept, as `COPY` into a table keeps it. + `COPY ... WHERE` and the `FREEZE`, `ON_ERROR`, `REJECT_LIMIT` and `DEFAULT` + options are refused. +- An `INSERT`, `UPDATE` or `DELETE` nested in a `WITH` entry goes through the + same rewrite as a top-level one. With a watermark an `INSERT`'s row may go + cold, so `RETURNING` on it is refused, as on a top-level `INSERT`; a hot + `UPDATE` or `DELETE` keeps `RETURNING`, a cold or dual-tier one cannot + return rows or read another `WITH` entry (its DuckDB half does not see + them), and a cross-tier move cannot be a `WITH` entry. A statement may write + a tiered view once, and the nested write may not have a `WITH` clause of + its own. On a decoupled view the source runs in DuckDB, so a `WITH` entry + that modifies data is refused (DuckDB 1.5 has no data-modifying `WITH`), and + a nested `INSERT` may not read another `WITH` entry. A statement that holds + a nested write and also reads a tiered view runs in DuckDB as a whole, and + pg_duckdb refuses it ("DuckDB does not support modifying CTEs"); read the + hot table, or split the statement. +- `MERGE INTO `, on PostgreSQL 17 and later (16 allows `MERGE` on tables + alone), runs on the tier its `ON` condition bounds the partition column to + (`AND t.ts >= ''` for the hot tier, `AND t.ts < ''` for the + cold tier): a hot `MERGE` runs in PostgreSQL against the hot table + and keeps `RETURNING`, a cold one in DuckDB against the Iceberg table. Each + tier sees its own rows alone, so a `MERGE` that bounds neither tier is + refused, and an `INSERT` action's row must belong to the statement's tier: a + hot `MERGE` refuses a row below the cutoff, a cold one a row at or after it; + insert such rows with `INSERT`, which splits them. A cold `INSERT` action + must give the identity column a value (`OVERRIDING SYSTEM VALUE` for a + `GENERATED ALWAYS` one), a cold `MERGE` cannot return rows and runs one + `UPDATE` or `DELETE` action per statement (duckdb-iceberg's limit), and a + `WHEN NOT MATCHED BY SOURCE` action must bound the same tier in its own + condition. A `MERGE` that sets the partition column is refused (run the + change as an `UPDATE`), as is one on a table with a clustered vector column + and one whose source reads the view. A `MERGE` nested in a `WITH` entry + takes the same path as a nested `UPDATE` or `DELETE`. On a decoupled view + every `MERGE` runs in DuckDB. +- `TRUNCATE` on a registered relation, or on the hot table behind a tiered one, + fails with an error, because the cold rows in Iceberg would stay visible + through the view. +- A table or column with a cold tier cannot have a name that PostgreSQL leaves + unquoted but DuckDB parses as a keyword. With DuckDB 1.5.4 those names are + `anti`, `asof`, `at`, `by`, `describe`, `glob`, `lambda`, `pivot`, + `pivot_longer`, `pivot_wider`, `positional`, `qualify`, `semi`, `show`, + `summarize`, `unpack` and `unpivot`. pg_duckdb passes a table or column name + to DuckDB quoted the way PostgreSQL quotes it, so such a name reaches DuckDB + bare even when the query quotes it. Every query pg_duckdb runs against the + table or column then fails with a DuckDB `Parser Error` + ([duckdb/pg_duckdb#1019](https://github.com/duckdb/pg_duckdb/issues/1019)). + ColdFront refuses such a column name wherever a column gets a cold tier: + tiered registration, an archive pass that tiers or expires data, + `coldfront.create_iceberg_table()`, `coldfront.adopt_iceberg_table()`, and an + `ADD COLUMN` or `RENAME COLUMN` on a table that already has a cold tier. + ColdFront refuses such a table name at tiered registration, in + `coldfront.create_iceberg_table()` and `coldfront.adopt_iceberg_table()`, and + in a `RENAME` of the view or the hot table of a table that already has a cold + tier. Partition-only registration, without a hot period, accepts a table with + such a name or column. ColdFront does not refuse a DuckDB keyword that the + DuckDB parser accepts as a name, such as `columns`. diff --git a/docs/compaction.md b/docs/compaction.md index 55cd3214..94133734 100644 --- a/docs/compaction.md +++ b/docs/compaction.md @@ -1,4 +1,4 @@ -# COMPACTOR - Cold-Tier Table Maintenance +# Compacting the Cold Tier `cmd/compactor` keeps a cold-tier Iceberg table healthy: it compacts each partition's small Parquet files into fewer large ones, expires old snapshots, @@ -129,10 +129,10 @@ tables directly. Build the binary with `make compactor`, which vets, lints To go further with ColdFront, consult the following documents: -- The [Using ColdFront](usage.md) guide covers the server configuration the +- The [Using ColdFront](using_coldfront/index.md) guide covers the server configuration the compactor shares with the archiver, and the deployment YAML `import` writes into it. - The [Vector Storage](architecture_vectors.md) deep dive describes the sort key the compactor applies to clustered tables. -- The [Architecture](architecture.md) overview describes the bakery claim each +- The [Architecture](architecture_guides/index.md) overview describes the bakery claim each compaction step takes. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 00000000..0dc22f37 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,107 @@ +# Configuring ColdFront + +After installing ColdFront packages or building from source, several +components still need configuring before the cold tier comes online: + +- PostgreSQL and ColdFront's own configuration, detailed in this + guide, regardless of which installation path you followed. +- your object store, documented in + [Configuring your Object Store](object_store.md). +- installing, configuring, and bootstrapping Lakekeeper, and the + one-time per-database setup that ties everything together, + documented in the [Configuring Lakekeeper](one_time_setup.md) + guide. + +## Configuring PostgreSQL + +A package installation does not configure PostgreSQL. Add the following +settings to `postgresql.conf`, then restart PostgreSQL: + +```ini +shared_preload_libraries = 'pg_duckdb,coldfront' +duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' +duckdb.allow_unsigned_extensions = true +duckdb.autoinstall_known_extensions = false +coldfront.iceberg_async_parquet = on +coldfront.iceberg_bakery_patch = on +coldfront.warehouse = '' +coldfront.lakekeeper_endpoint = 'http://:8181/catalog' +coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' +``` + +Together, the three `duckdb.*` settings point pg_duckdb at the directory +where `pgedge-coldfront-duckdb-extensions` installs ColdFront's patched +DuckDB extensions, rather than letting it fetch the unpatched upstream +build - the same substitution that reintroduces HTTP 409 failures under +concurrent cold writes. The patched extensions are unsigned, so +`duckdb.allow_unsigned_extensions` must stay on; and because a missing +extension file would otherwise trigger that same fallback, +`duckdb.autoinstall_known_extensions` stays off. + +The two `coldfront.iceberg_*` settings make a node upload the Parquet files of +an `INSERT` or `COPY` outside its write lock (the bakery claim on a Spock mesh, +an advisory lock on a single node) and serialize only the catalog commit. The +packaged duckdb-iceberg includes the patch that this ordering requires, and the +[Distributed Setup](distributed_setup.md) guide describes the mesh settings. + +ColdFront also exposes the following settings, which adjust write +behavior and execution; tune them as needed: + +| Setting | Description | +|---|---| +| `coldfront.allow_mixed_writes` | Controls tiered-mode `UPDATE`/`DELETE` whose WHERE cannot be proven to target one tier. `on` emits a dual-tier CTE; `off` rejects with an error and a hint. Not relevant in decoupled mode (every write is single-tier by definition). The default is `on`. | +| `coldfront.cold_write_batch_size` | Sets how many cold rows a tiered `INSERT` gathers (see [Caveats](caveats.md)) before writing them to Iceberg as one `INSERT`. A larger value writes fewer, larger Parquet files; the remainder always flushes, so a small write stays one file. The default is `10000`, with a minimum of `1`. | +| `coldfront.vector_probe` | Sets whether a recognized similarity search reads only the clusters nearest its query vector. `off` gives an exact scan of the whole corpus. Affects only a table with a trained vector column ([usage_vectors.md](usage_vectors.md)). The default is `on`. | +| `coldfront.vector_nprobe` | Sets how many clusters such a search reads, overriding the column's own `nprobe`; at or above the column's `nlist` the search is exhaustive. The default is `0`, which uses the configured value. | +| `duckdb.force_execution` | Benchmark before enabling: on a mixed workload it helps `count(distinct)` and similar but regresses index lookups, top-K with PK ordering, and JSON access. The default is `off`. | +| `duckdb.temporary_directory` | Sets where DuckDB spills. Each backend gets its own subdirectory there, named after its process id, so concurrent spills cannot collide; one left by a departed backend is reclaimed. See [architecture.md](architecture_guides/index.md#duckdb-spill-files-are-not-namespaced-per-instance). | +| `duckdb.max_temp_directory_size` | A cap per connection, not a cluster total. For a total budget, divide it by the concurrent sessions, or give the temp path its own filesystem or quota. The default is 90% of free space per session. | + +## Writing ColdFront's Configuration + +Server-level settings live in `postgresql.conf`, each managed table's +configuration is a row in `coldfront.partition_config`, and the +cold-store credential lives in `coldfront.storage_secret`. + +Write the configuration with `coldfront.set_storage_secret()` and the +`register` command. `set_storage_secret()` writes the following to +`coldfront.storage_secret`; the +[Configuring your Object Store](object_store.md) guide shares the exact +values for each supported backend: + +- `` - your object store's access key. +- `` - the matching secret. +- `` - your object store's endpoint. + +```sql +SELECT coldfront.set_storage_secret('', '', ''); +``` + +`register` writes the following to `coldfront.partition_config`: + +- `` - the table name. +- `` - the partition cadence, e.g. `monthly`. +- `` - how long a partition stays hot before the archiver + moves it to the cold tier. + +```bash +./bin/archiver register --table
\ + --period --hot-period "" +``` + +Or do both in one step by importing a deployment YAML: + +```bash +archiver import --config deploy.yaml +``` + +The `pgedge-coldfront` package installs an example deployment YAML at +`/etc/pgedge/coldfront/config.yaml`. The file is only an example; edit it and pass it to `import` - no +ColdFront tool reads it unless `--config` names it. +After an import, the server holds the configuration. A later run that is given +a YAML checks the file against the server and refuses to run if any value +differs. The only value such a run takes from the file is `postgres.dsn`, which +connects when `--dsn` is unset. The +[Managing Partitioned Tables (CLI)](usage_partitioner.md#managing-partitioned-tables-cli) +section of the Using ColdFront in Standalone Partitioned Mode guide describes +`register`, `import`, and `export`. diff --git a/docs/distributed_setup.md b/docs/distributed_setup.md new file mode 100644 index 00000000..a91dfb09 --- /dev/null +++ b/docs/distributed_setup.md @@ -0,0 +1,198 @@ +# Setting Up a Distributed Deployment + +For multi-writer iceberg workloads, run ColdFront on N PG nodes in a Spock mesh +against the same Lakekeeper + S3. The bakery serializes commits PG-side so +writers never collide at the catalog. + +### Per-Node `postgresql.conf` + +The configuration below applies to each node in the mesh: + +```ini +wal_level = logical +shared_preload_libraries = 'snowflake,spock,pg_duckdb,coldfront' + +# The bakery rules a peer dead when its walsender's last reply is older than +# coldfront.peer_alive_window_ms (default 10 s), and an idle peer replies only +# to the walsender's keepalive, every wal_sender_timeout/2. The claim refuses +# to run unless wal_sender_timeout is positive and below twice the window. +wal_sender_timeout = 15s + +# Spock prereq + DDL replication. The ddl_sql repset (and with it the +# one-node provisioning promise for wrapper views) only works with DDL +# replication on; allow_ddl_from_functions covers DDL issued inside +# coldfront's plpgsql helpers. +track_commit_timestamp = on +spock.enable_ddl_replication = on +spock.allow_ddl_from_functions = on +spock.include_ddl_repset = on + +# Only on a server that has this setting (PostgreSQL 16.15, 17.11 and 18.6 in +# pgEdge's builds): +# it lists the libraries allowed as logical decoding output plugins, and its +# default leaves out Spock's, so no subscription can create its slot. A server +# without the setting refuses to start with this line in place. +output_plugin_libraries = 'pgoutput, test_decoding, spock_output' + +# Per-node - any distinct integer 1..1023 (must be unique per node; the value +# is otherwise arbitrary - the bakery matches acks by spock node name, not by id). +snowflake.node = 1 + +# DSN of the node's loopback, which runs the bakery's autonomous claim +# INSERT/DELETE. It must name a unix socket: every onboarded role can read +# this setting, so it must never need a password. The claim session only +# touches the coldfront.claims/claim_acks heap tables, so it never attaches +# the Iceberg catalog (the lazy catalog-attach hook fires only on a tiered +# view). application_name=coldfront_loopback marks the session as bakery +# traffic. Only a superuser can set it. +coldfront.loopback_dsn = 'host=/tmp dbname=coldfront user=coldfront application_name=coldfront_loopback' + +coldfront.warehouse = 'wh' +coldfront.lakekeeper_endpoint = 'http://lakekeeper:8181/catalog' + +# Upload the Parquet outside the bakery claim and serialize only the catalog +# commit. Set both, and only where duckdb-iceberg includes ColdFront's patch (the +# ColdFront image does). With either off, the upload is serialized too. +coldfront.iceberg_async_parquet = on +coldfront.iceberg_bakery_patch = on +``` + +The bakery has no peer-ack timeout. R-A's only failure mode is a dead peer +(would wait forever), and a liveness check inside the wait loop closes it: +the bakery treats a peer whose `pg_stat_replication.reply_time` is older than +`coldfront.peer_alive_window_ms` as already acked, whatever the walsender's +state (a peer that has just reconnected is catching up and still alive). The +peer's apply worker sends that reply after it applies, every +`spock.feedback_frequency` messages, and in answer to the walsender's +keepalive; `wal_receiver_status_interval` plays no part, since Spock's apply +worker does not read it. The window and `wal_sender_timeout` go together; set +them as [Timeouts](#timeouts) below describes. An alive peer that has +not acked is either deferring legitimately (R-A's defer rule) or about to +ack, and waiting for it is correct, not a failure. + +A claim whose owner is gone (a hard backend crash) is reaped without operator +action: by that node's next cold write, to any table, by a peer's arriving +claim, or by the waiting peer's poke, a no-op `UPDATE` of its own claim row +about once a second for as long as it waits. That poke is the only replication +traffic the bakery generates while a writer waits, and there is none when +nothing waits. See [architecture_decoupled.md](architecture_decoupled.md), +*Orphan reaping*. + +Sync-rep (`synchronous_standby_names`) is **not required** by the bakery - the +R-A ack barrier is what serializes iceberg commits. You can still enable +sync-rep cluster-wide if you want stronger durability for non-bakery writes, +but it plays no part in iceberg-commit serialization. + +Run this one-time mesh setup in this order on every node, because step 3 calls +`spock.repset_add_table`, which requires the local Spock node to already exist: + +```sql +-- 1. Extensions, in dependency order. snowflake is a bakery prereq +-- (claim tickets come from snowflake.nextval). +CREATE EXTENSION IF NOT EXISTS snowflake; +CREATE EXTENSION IF NOT EXISTS spock; +CREATE EXTENSION IF NOT EXISTS pg_duckdb; +CREATE EXTENSION IF NOT EXISTS coldfront; + +-- 2. Spock node + full-mesh subscriptions (each node has N-1 subs). Name each +-- subscription sub__from_ (on n1: sub_n1_from_n2): the +-- bakery's peer-liveness check looks the walsender up by that name, and a peer +-- whose subscription is named any other way counts as dead and is treated as +-- already acked. +SELECT spock.node_create('n', 'host= user=coldfront dbname=coldfront port=5432'); + +-- on n1 (n2 and n3 symmetric): +SELECT spock.sub_create('sub_n1_from_n2', 'host= user=coldfront dbname=coldfront port=5432', + ARRAY['default','default_insert_only','ddl_sql'], + false, false, '{}', '0', false); +SELECT spock.sub_create('sub_n1_from_n3', 'host= user=coldfront dbname=coldfront port=5432', + ARRAY['default','default_insert_only','ddl_sql'], + false, false, '{}', '0', false); + +-- 3. **Required** on every node, after its subscriptions exist and before +-- the first cold write, storage secret or table registration: put every +-- ColdFront table that replicates by value in this node's replication sets. +SELECT coldfront.ensure_replicated(); +``` + +The subscriptions set +`synchronize_structure := false, synchronize_data := false` because the tables +already exist on every node from the coldfront extension, so no initial copy is +needed. + +### Timeouts + +The following table lists the two settings that decide how long a cold write +waits for a peer; set them together, on every node: + +| Setting | Recommended | Where | What it does | +|---|---|---|---| +| `wal_sender_timeout` | `15s` | `postgresql.conf` on every node; the image sets it in mesh mode | The node's walsender asks each peer for a reply after half of it (7.5 s) and drops a peer that has not replied for the whole of it. | +| `coldfront.peer_alive_window_ms` | `10000` (the default) | `postgresql.conf`, superuser-only | The bakery treats a peer whose last reply is older than this as dead and stops waiting for its ack. | + +The rule: half of `wal_sender_timeout` must stay below the window, with a +round trip to spare; the claim enforces the first part. An idle peer replies +only when the walsender asks, so the +PostgreSQL default of 60 s against a 10 s window would rule a live idle peer +dead; a claim refuses to run with such settings and names both. With the +recommended values a writer waits at most 10 s for a peer that is really +dead, and the walsender disconnects a peer silent for 15 s, which then +reconnects on its own. + +On a slow or lossy WAN raise both together and keep the rule, for example +`wal_sender_timeout = 30s` with `coldfront.peer_alive_window_ms = 20000`. +Both take effect on a reload (`ALTER SYSTEM SET ...;` then +`SELECT pg_reload_conf();`), no restart: walsenders re-read the timeout, and +each claim reads the window once. + +### What `coldfront.ensure_replicated()` Does + +The call is the one ColdFront-specific step of the mesh setup. It adds eight +tables to the node's replication sets (`claim_acks` to `default_insert_only`, +the rest to `default`), each keyed by name, so a row is identical on every node +and replicates by value: + +| Table | What replicates, and why a peer needs it | +|---|---| +| `coldfront.claims`, `coldfront.claim_acks` | The bakery's tickets and acknowledgements. A peer acknowledges an originator's claim by inserting into `claim_acks` on its own node, and the ack reaches the originator only if the table is in the peer's set. `claim_acks` goes in the insert-only set, because each node deletes only the ack copies it holds. | +| `coldfront.tiered_views` | The registry. A peer's hook recognizes a view by its row; without it the peer's writes through the view fail in PostgreSQL ("cannot insert into view") and its reads cannot attach the cold tier. | +| `coldfront.archive_watermark` | The hot/cold cutoff that routes each tiered write. | +| `coldfront.storage_secret` | The cold-store credential, so one `set_storage_secret` call reaches every node. | +| `coldfront.partition_config` | The per-table lifecycle. The archiver and the partitioner add this table themselves as well, because the partitioner runs without the extension. | +| `coldfront.vector_config`, `coldfront.vector_centroids` | The cluster routing state, so every node resolves a vector to the same cluster. | + +Two tables stay local by design and are not added: `coldfront.deferred_acks` +(each node's queue of the acks it owes) and `coldfront._dummy_dml_target`. + +The call runs on every node because a replication set is a property of the +provider: each node sends the rows of the tables in its own set, and a node +that skipped the step keeps its acks, registry rows and secret to itself. It +cannot run at `CREATE EXTENSION` time, because `spock.repset_add_table` needs +the local Spock node, and nothing runs it later on a node's behalf: +provisioning a table, setting the secret and writing cold rows all assume it +has run. It is idempotent, so running it again is harmless, and a node added to +the mesh later runs the same three steps. + +A skipped step shows up late and away from the node that skipped it. If a peer +never ran it, the originator of every cold write waits at the ack barrier for +an ack that never arrives. If the node that registers a table never ran it, +the view reaches each peer as replicated DDL but the registry row does not, so +the peer's writes through the view fail with "cannot insert into view". A +secret set on such a node stays on that node. + +Verify the step on every node. The query lists the eight tables: + +```sql +SELECT c.relname + FROM spock.replication_set rs + JOIN spock.replication_set_table rst ON rst.set_id = rs.set_id + JOIN pg_class c ON c.oid = rst.set_reloid + WHERE rs.set_name = 'default' AND c.relnamespace = 'coldfront'::regnamespace + ORDER BY 1; +``` + +Then verify the mesh before benching - insert a sentinel claim on each node +and read it back from every other node. All N×(N-1) directions must show the +row before traffic starts. `ci/journey.sh` `story_mesh_substrate` is a +copyable reference: it checks the membership above on every node and then the +sentinels. diff --git a/docs/index.md b/docs/index.md index b927aa3d..a9e6653a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -85,7 +85,7 @@ docker compose --profile local-store up -d --build ``` Bootstrap Lakekeeper and create a warehouse (see the one-time setup in the -[Using ColdFront](usage.md) guide), then create a table in psql: +[Using ColdFront](using_coldfront/index.md) guide), then create a table in psql: ```sql CREATE EXTENSION IF NOT EXISTS pg_duckdb; @@ -105,16 +105,16 @@ than creating it: `coldfront.adopt_iceberg_table()` reads its schema from the catalog and gives it the same wrapper view and registry row, read-only unless you pass `p_writable => true`. `coldfront.release_iceberg_table()` hands the table back with the Iceberg table untouched. See -[Adopting a Table That Already Exists in the Catalog](usage.md#adopting-a-table-that-already-exists-in-the-catalog). +[Adopting a Table That Already Exists in the Catalog](usage_decoupled.md#adopting-a-table-that-already-exists-in-the-catalog). To remove a table again, `coldfront.drop_iceberg_table()` unregisters it and drops the Iceberg table, deleting the stored objects only when asked to. See -[Dropping an Iceberg Table](usage.md#dropping-an-iceberg-table-both-modes). +[Dropping an Iceberg Table](usage_decoupled.md#dropping-an-iceberg-table-both-modes). For compliance environments that cannot store an object-store credential, `coldfront.set_storage_secret_vended()` runs with no credential in the database: Lakekeeper issues short-lived per-table credentials at access time. -See [Vended Credentials](usage.md#vended-credentials). +See [Vended Credentials](vended_credentials.md). ## Least-Privilege Application Roles @@ -148,7 +148,7 @@ For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the `duckdb.postgres_role` default that the image sets up, and how least privilege holds across a Spock mesh - see -[Architecture: non-superuser app roles](architecture.md#non-superuser-app-roles-least-privilege). +[Architecture: non-superuser app roles](architecture_guides/index.md#non-superuser-app-roles-least-privilege). ## Caveats @@ -165,15 +165,40 @@ see all three modes in action with copy-pasteable commands. To go further with ColdFront, consult the following guides: -- The [Installation](installation.md) guide covers building ColdFront and - bringing up the stack. -- The [Using ColdFront](usage.md) guide covers both modes, the standalone - partition manager, supported types, and tuning. -- The [Embeddings](usage_vectors.md) guide covers storing and searching +- The [Installation](installation.md) guide details installing ColdFront + from a package or building it from source. +- The [Configuring ColdFront](configuration.md) guide documents PostgreSQL + and ColdFront's own configuration. +- The [Configuring Lakekeeper](one_time_setup.md) guide installs, + configures, and bootstraps it, then installs the extensions + each database needs. +- The [Using ColdFront](using_coldfront/index.md) guide compares the three operating modes + and links to each mode's guide. +- The [Using ColdFront in Tiered Mode](usage_tiered.md) guide documents + archiving aging partitions to Iceberg. +- The [Using ColdFront in Decoupled Mode](usage_decoupled.md) guide + documents tables that live entirely in Iceberg, and dropping or + releasing a registered table. +- The [Using ColdFront in Standalone Partitioned Mode](usage_partitioner.md) + guide documents managing PostgreSQL partitions with no cold tier, and + the partition CLI shared with tiered mode. +- The [Caveats](caveats.md) guide documents cross-tier isolation, mixed + transactions, and other sharp edges shared between tiered and decoupled + tables. +- The [Vended Credentials](vended_credentials.md) guide documents running + with no stored object-store credential. +- The [Reading and Writing](reading_writing.md) guide documents the SQL + surface shared between tiered and decoupled tables. +- The [Supported Column Types](supported_types.md) guide documents the + PostgreSQL column types, including arrays, that a cold tier supports. +- The [Distributed Setup](distributed_setup.md) guide documents running a + multi-writer decoupled-mode Spock mesh against one shared lake. +- The [Embeddings](usage_vectors.md) guide documents storing and searching embeddings through the pgvector interface. -- The [Configuring your Object Store](object_store.md) guide takes you from an - empty bucket to a working cold tier on cloud S3. -- The [Compaction](compaction.md) guide covers cold-tier maintenance: +- The [Configuring your Object Store](object_store.md) guide takes you from + an empty bucket or container to a working cold tier, for any of four + supported backends. +- The [Compaction](compaction.md) guide documents cold-tier maintenance: compaction, snapshot expiry, and orphan-file removal. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. diff --git a/docs/installation.md b/docs/installation.md index 9deaf814..e4bd8e9a 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,21 +1,23 @@ -# Installing and Configuring ColdFront +# Installing ColdFront -This guide covers installing ColdFront from the published package or -building it from source, and configuring it once installed. +This guide details installing ColdFront from the published package or +building it from source. See [Configuring ColdFront](configuration.md) +for setting up PostgreSQL, Lakekeeper, and ColdFront itself once +installed. ## Using a Package to Install ColdFront ColdFront's packages come from the pgEdge package repository. Before you -install them, configure the repository by following the page for your platform -in the Enterprise Postgres documentation: +can use a package, you must configure the pgEdge repository by following +the steps in the Enterprise Postgres documentation: - For RHEL, Oracle Linux, AlmaLinux, and Rocky Linux, see [Configuring the Repository on Enterprise Linux](https://docs.pgedge.com/enterprise/el/configure-repo/). - For Debian and Ubuntu, see [Configuring the Repository on Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/configure-repo/). -Those pages cover the platform prerequisites, disabling any PGDG repository, -and creating the pgEdge repository. +The documentation details the platform prerequisites, disabling any PGDG +repository, and creating the pgEdge repository. With the repository in place, install the ColdFront extension package for your PostgreSQL major version. It depends on the pgEdge PostgreSQL server, @@ -48,115 +50,6 @@ The extension package does not install the command-line tools or Lakekeeper. Install `pgedge-coldfront` and `pgedge-lakekeeper` separately on the hosts that need them. -### Configuring PostgreSQL - -A package installation does not configure PostgreSQL. Add the following -settings to `postgresql.conf`, then restart PostgreSQL: - -```ini -shared_preload_libraries = 'pg_duckdb,coldfront' -duckdb.extension_directory = '/usr/lib/pgedge/coldfront/duckdb-extensions' -duckdb.allow_unsigned_extensions = true -duckdb.autoinstall_known_extensions = false -coldfront.iceberg_async_parquet = on -coldfront.iceberg_bakery_patch = on -coldfront.warehouse = '' -coldfront.lakekeeper_endpoint = 'http://:8181/catalog' -coldfront.local_pg_dsn = 'host=/var/run/postgresql dbname= user=' -``` - -The three `duckdb.*` settings make pg_duckdb load ColdFront's patched DuckDB -extensions from the directory where `pgedge-coldfront-duckdb-extensions` -installs them. Without these settings, pg_duckdb downloads the unpatched -upstream extensions, and concurrent cold writes can then fail with HTTP 409. -The patched extensions are unsigned, so `duckdb.allow_unsigned_extensions` must -be on. Keeping `duckdb.autoinstall_known_extensions` off stops DuckDB from -downloading an unpatched upstream copy when an extension file is missing. - -The two `coldfront.iceberg_*` settings make a node upload the Parquet files of -an `INSERT` or `COPY` outside its write lock (the bakery claim on a Spock mesh, -an advisory lock on a single node) and serialize only the catalog commit. The -packaged duckdb-iceberg includes the patch that this ordering requires, and the -[Distributed Setup](usage.md#distributed-setup-3-node-mesh-decoupled-mode) -section of the Using ColdFront guide describes the mesh settings. - -The [One-Time Setup](usage.md#one-time-setup) section of the Using ColdFront -guide describes the `coldfront.*` settings. Follow that section from the -Lakekeeper bootstrap onward to create the warehouse, the extensions, and the -cold-store credential. - -### Setting Up Lakekeeper - -The `pgedge-lakekeeper` package installs the `lakekeeper` binary and a -`lakekeeper` systemd service, but it does not enable or start the service. -Lakekeeper stores its catalog in a PostgreSQL 15 or later database and reads -its settings from `/etc/lakekeeper/lakekeeper.env`. The following steps -prepare and start the service: - -1. Create a role and a database for the catalog: - - ```sql - CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; - CREATE DATABASE lakekeeper OWNER lakekeeper; - ``` - - The migration in step 3 creates the `uuid-ossp`, `pgcrypto`, `pg_trgm`, - `btree_gin`, and `btree_gist` extensions, so either the role must be - allowed to run `CREATE EXTENSION` or a superuser must create them first. - -2. In `/etc/lakekeeper/lakekeeper.env`, set - `LAKEKEEPER__PG_DATABASE_URL_WRITE` to the database's connection string and - `LAKEKEEPER__PG_ENCRYPTION_KEY` to a random secret, such as the output of - `openssl rand -base64 32`. Lakekeeper encrypts stored credentials with the - key, so keep the key stable and backed up. Every node that shares the - catalog needs the same key. - -3. Run the one-time database migration as the `lakekeeper` user: - - ```bash - set -a; . /etc/lakekeeper/lakekeeper.env; set +a - sudo -E -u lakekeeper /usr/bin/lakekeeper migrate - ``` - -4. Enable and start the service: - - ```bash - sudo systemctl enable --now lakekeeper - ``` - -Lakekeeper listens on port 8181 on every address by default. Without an -authorization backend in `lakekeeper.env`, the catalog accepts every request, -so configure authentication and authorization before you expose the service -beyond a trusted network. - -### Configuring ColdFront - -ColdFront has no configuration file, because the database holds its -configuration. The server settings are the `postgresql.conf` lines above, each -managed table is a row in `coldfront.partition_config`, and the cold-store -credential is in `coldfront.storage_secret`. The archiver, partitioner, and -compactor connect the way psql does, from the libpq environment (`PGHOST`, -`PGDATABASE`, `PGUSER`, `PGPASSWORD`, `PGSERVICE`) or `--dsn`, and read every -other setting from the server. - -You write that configuration with `coldfront.set_storage_secret()` and the -`register` command, or in one step by importing a deployment YAML: - -```bash -archiver import --config deploy.yaml -``` - -The `pgedge-coldfront` package installs an example deployment YAML at -`/etc/pgedge/coldfront/config.yaml`. That file is only an example to edit and -pass to `import`, and no ColdFront tool reads it unless `--config` names it. -After an import, the server holds the configuration. A later run that is given -a YAML checks the file against the server and refuses to run if any value -differs. The only value such a run takes from the file is `postgres.dsn`, which -connects when `--dsn` is unset. The -[Managing Partitioned Tables (CLI)](usage.md#managing-partitioned-tables-cli) -section of the Using ColdFront guide describes `register`, `import`, and -`export`. - ## Building ColdFront from Source This section walks you through building ColdFront from source, either in @@ -267,8 +160,8 @@ publishes it. The floating `pg` tag moves to the same image only when the base-image workflow is dispatched from `main` with `push=true` and both architectures. -Then follow the [One-Time Setup](usage.md#one-time-setup) section of the Using -ColdFront guide: bootstrap Lakekeeper, create a table, tier it, and verify. +Then follow the [Configuring Lakekeeper](one_time_setup.md) guide: +bootstrap it, create a table, tier it, and verify. !!! note "pg_duckdb pin" @@ -359,8 +252,8 @@ SQL A row count of 1 read back through Iceberg confirms the full path. For a real cloud store, drop the `local-store` profile, point the warehouse at your own -bucket, and follow the [One-Time Setup](usage.md#one-time-setup) section of the -Using ColdFront guide for the full tier-and-verify journey. +bucket, and follow the [Configuring Lakekeeper](one_time_setup.md) guide +for the full tier-and-verify journey. ## Building ColdFront on Bare Metal @@ -394,9 +287,9 @@ coldfront.iceberg_bakery_patch = on coldfront refuses to load any other way: `CREATE EXTENSION coldfront` fails on a server that does not preload it, with an error that names the setting. -See the [Tuning Knobs](usage.md#tuning-knobs) section of the Using ColdFront -guide for the remaining GUCs, and the README for the optional non-superuser -role that the image sets up. +See the [Configuring PostgreSQL](configuration.md#configuring-postgresql) +section of the Configuring ColdFront guide for the remaining GUCs, and +the README for the optional non-superuser role that the image sets up. ## Testing & CI @@ -462,8 +355,12 @@ SeaweedFS-only. To go further with ColdFront, consult the following guides: +- The [Configuring ColdFront](configuration.md) guide details PostgreSQL + and ColdFront's own configuration. +- The [Configuring Lakekeeper](one_time_setup.md) guide installs, + configures, and bootstraps it, then installs the extensions each + database needs. - The [Walkthrough](walkthrough.md) guide runs the demo stack hands-on. -- The [Using ColdFront](usage.md) guide covers the one-time setup and both - modes. -- The [Configuring your Object Store](object_store.md) guide connects the cold - tier to AWS S3. +- The [Using ColdFront](using_coldfront/index.md) guide documents both operating modes. +- The [Configuring your Object Store](object_store.md) + guide connects the cold tier to AWS S3. diff --git a/docs/object_store.md b/docs/object_store.md index dc46723b..22f7898d 100644 --- a/docs/object_store.md +++ b/docs/object_store.md @@ -1,87 +1,95 @@ # Configuring your Object Store -This walkthrough takes you from an empty S3 bucket to a working ColdFront cold -tier in one sitting. You stand up the ColdFront stack (PostgreSQL + the -Lakekeeper Iceberg catalog) with Docker, point it at your bucket, and write -rows that land as Apache Iceberg tables in S3 - readable straight back through -Postgres. - -This walkthrough targets a real cloud S3 service that uses virtual-hosted -addressing. For a path-style S3-compatible store (MinIO, SeaweedFS) or GCS, see -the [Storage Backends](usage.md#storage-backends) section of the Using -ColdFront guide instead. - -No prior ColdFront knowledge is assumed. Copy and paste the commands from top -to bottom. The examples use the following placeholders throughout: bucket -`my-iceberg-bucket`, region `eu-west-1`, key `AKIAEXAMPLE...`, secret -`` - substitute your own. +This walkthrough guides you from an empty bucket or container to a +working ColdFront cold tier. You stand up the ColdFront stack - +PostgreSQL and the Lakekeeper Iceberg catalog - with Docker, and point +it at your object store. ColdFront supports four cold-store backends: +S3-compatible stores (SeaweedFS, MinIO), virtual-hosted cloud S3 (AWS), +Google Cloud Storage, and Azure ADLS Gen2. Configure **exactly one** of +them for a given deployment; this guide shows the bucket/container +setup, the Lakekeeper warehouse, and the ColdFront credential for each. --- -## 1. Prerequisites +## Prerequisites + +Before you begin, decide which backend you are using, since the +bucket/container details, credential type, and Lakekeeper warehouse +profile differ by backend: + +- **S3-compatible stores** (SeaweedFS, MinIO) need a bucket, the + store's endpoint URL, and a long-term access key and secret. +- **Virtual-hosted cloud S3** (AWS) needs a bucket, the bucket's real + Region, and a long-term access key and secret. +- **Google Cloud Storage** needs a bucket and an + [HMAC key pair](https://cloud.google.com/storage/docs/authentication/hmackeys) + (GCS has no native Lakekeeper profile; it is configured as an S3 + backend pointed at GCS's S3-interoperability endpoint). +- **Azure ADLS Gen2** needs a storage account, a filesystem (container) + in that account, and the account's access key. + +For any backend that takes a long-term key (S3-compatible, virtual-hosted +S3, and GCS): + +> **Only long-term keys work.** Lakekeeper's warehouse credential has no +> field for a session token, so single sign-on (SSO) or temporary +> session-token credentials do **not** work here. Use a permanent +> access-key pair with no expiry. +> +> This applies to the warehouse's own credential. A deployment that must +> not store any object-store credential in the database can use vended +> credentials instead, where the warehouse issues short-lived per-table +> credentials at access time. Vended credentials are available for AWS S3 +> and Azure ADLS Gen2, but not for GCS or other S3-compatible stores; see +> [vended_credentials.md](vended_credentials.md). + +For AWS S3 and S3-compatible stores, grant the key permission to read, +write, and list the bucket (`GetObject` / `PutObject` / `DeleteObject` / +`ListBucket`) and to abort a failed multipart upload +(`AbortMultipartUpload`), as in the following example policy: -Before you begin, gather the following: - -- an S3 bucket (`my-iceberg-bucket` in the examples below). -- the bucket's real region (`eu-west-1` in the examples below). -- a long-term access key, meaning an access key id (`AKIAEXAMPLE...`) and its - secret (``). - - > **Only long-term keys work.** Lakekeeper's warehouse credential has no - > field for a session token, so single sign-on (SSO) or temporary - > session-token credentials do **not** work here. Use a permanent - > access-key pair with no expiry. - > - > This applies to the warehouse's own credential. A deployment that must - > not store any object-store credential in the database can use vended - > credentials instead, where the warehouse issues short-lived per-table - > credentials at access time; see [usage.md](usage.md#vended-credentials). - -- permission for the key to read, write and list the bucket (`GetObject` / - `PutObject` / `DeleteObject` / `ListBucket`) and to abort a failed multipart - upload (`AbortMultipartUpload`), as in the following example policy: - - ```json +```json +{ + "Version": "2012-10-17", + "Statement": [ { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"], - "Resource": [ - "arn:aws:s3:::my-iceberg-bucket", - "arn:aws:s3:::my-iceberg-bucket/*" - ] - } + "Effect": "Allow", + "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:AbortMultipartUpload"], + "Resource": [ + "arn:aws:s3:::my-iceberg-bucket", + "arn:aws:s3:::my-iceberg-bucket/*" ] } - ``` + ] +} +``` - DuckDB never aborts a failed multipart upload, so add an - `AbortIncompleteMultipartUpload` lifecycle rule to the bucket to delete - leftover parts. +DuckDB never aborts a failed multipart upload, so add an +`AbortIncompleteMultipartUpload` lifecycle rule to the bucket to delete +leftover parts. -- the ColdFront image, built once by following - [installation.md](installation.md), which notes the registry access the base - image needs. Run the commands below from the repo root. +You also need the ColdFront image, built once by following +[installation.md](installation.md), which notes the registry access the +base image needs. Run the commands below from the repo root. + +No prior ColdFront knowledge is required to perform the steps on this +page; perform the commands in the order they are presented. The +examples use placeholder bucket, account, and credential values +throughout - substitute your own. --- -## 2. Bring Up the Stack (Postgres + Lakekeeper) +## Bringing Up the Stack (Postgres + Lakekeeper) -Start the stack from the repo root with a single Compose command: +Our first step begins by starting the stack from the repo root with +a single Compose command. The `docker-compose.yml` file ships in the +ColdFront repository root, so `docker compose` finds it automatically +when you run the command from there: ```bash docker compose up -d --build ``` -This starts **Postgres** (with `pg_duckdb` and `coldfront` preloaded), -**Lakekeeper** (the Iceberg REST catalog), Lakekeeper's **own catalog -Postgres**, and a one-shot **migrate** job. The command does **not** start the -bundled SeaweedFS - that is gated behind the `local-store` profile for -credential-free local evaluation. For cloud S3 you connect to the bucket -directly, so you do not need the bundled SeaweedFS. - If a local Postgres already owns port 5432, pick another host port: ```bash @@ -90,7 +98,7 @@ COLDFRONT_PG_PORT=55432 docker compose up -d --build If port 8181 is taken, set `COLDFRONT_LK_PORT` the same way to move Lakekeeper's host port. Then use that port in place of `8181` in every `curl` -command in Section 3. +command in [Configuring Lakekeeper](#configuring-lakekeeper) below. Wait for Postgres to report healthy (the container name is derived from your directory, so resolve it at runtime): @@ -99,15 +107,30 @@ directory, so resolve it at runtime): docker inspect -f '{{.State.Health.Status}}' "$(docker compose ps -q db)" # => healthy ``` ---- +The command brings up four containers: + +- PostgreSQL, with `pg_duckdb` and `coldfront` already preloaded. +- Lakekeeper, the Iceberg REST catalog. +- Lakekeeper's own catalog database. +- a one-shot migrate job that applies Lakekeeper's schema migrations + and then exits. + +The command also starts a bundled SeaweedFS object-store emulator, +gated behind the `local-store` profile, reserved for credential-free +local evaluation of the S3-compatible path. If you are connecting to a +real S3-compatible store, AWS S3, GCS, or Azure, you connect to it +directly and never need SeaweedFS. -## 3. One-Time Lakekeeper Setup +--- -Run the following three calls once each. +## Configuring Lakekeeper -### 3a. Bootstrap Lakekeeper +Next, bootstrap Lakekeeper, then create a warehouse and pre-create the +namespace your Iceberg tables will live in. The warehouse and +credential shape depend on which backend you chose in +[Prerequisites](#prerequisites); follow the matching subsection below. -Bootstrap Lakekeeper and accept its terms of use: +First, bootstrap Lakekeeper and accept its terms of use: ```bash curl -X POST http://localhost:8181/management/v1/bootstrap \ @@ -115,11 +138,49 @@ curl -X POST http://localhost:8181/management/v1/bootstrap \ -d '{"accept-terms-of-use":true}' ``` -### 3b. Create the S3 Warehouse +### S3-Compatible Stores (SeaweedFS, MinIO) + +Create a warehouse with a path-style profile pointed at your store's +endpoint: + +```bash +curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" \ + -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", + "bucket": "my-iceberg-bucket", + "key-prefix": "coldfront", + "region": "us-east-1", + "endpoint": "https://my-store.example.com:9000", + "path-style-access": true, + "flavor": "s3-compat", + "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", + "credential-type": "access-key", + "aws-access-key-id": "", + "aws-secret-access-key": "" + } + }' +# => HTTP 201 +``` + +`region` defaults to `us-east-1` if your store does not use Region +semantics. Set the ColdFront credential with an explicit endpoint: + +```sql +SELECT coldfront.set_storage_secret('', '', + 'my-store.example.com:9000', 'us-east-1', 'path', true); +``` + +### Virtual-Hosted Cloud S3 (AWS) -A warehouse tells Lakekeeper where on S3 your Iceberg tables live and which -credential to use. This is the **virtual-hosted cloud-S3** profile, and the -following flags matter: +This is the **virtual-hosted cloud-S3** profile, and the following +flags matter: - omitting `endpoint` selects native per-Region virtual-hosted addressing over HTTPS. @@ -160,8 +221,23 @@ curl -X POST http://localhost:8181/management/v1/warehouse \ The `key-prefix` is an arbitrary path inside the bucket. `coldfront` is only an example. +Set the cold-tier credential with a `NULL` endpoint, which selects +DuckDB's native per-Region virtual-hosted addressing: + +```sql +SELECT coldfront.set_storage_secret('AKIAEXAMPLE...', '', NULL, 'eu-west-1'); +``` + +!!! warning "The 3rd argument (endpoint) must be NULL for a cloud S3 endpoint" + + A `NULL` endpoint selects DuckDB's native per-Region virtual-hosted + HTTPS + addressing. The 4th argument is your bucket's region, from which DuckDB + derives the endpoint. A non-NULL endpoint applies `url_style` and + `use_ssl`, which default to path-style over plain HTTP - do **not** use + that shape for cloud S3. + A vended-credentials variant is also available. To run ColdFront with no stored -credential ([usage.md](usage.md#vended-credentials)), the warehouse issues +credential ([vended_credentials.md](vended_credentials.md)), the warehouse issues per-table temporary credentials from AWS Security Token Service (STS) instead of handing the client a static key. Two things change from the warehouse above. @@ -206,16 +282,110 @@ The role's trust policy above requires `sts:ExternalId`, so the `external-id` in the warehouse credential must match the `` in that condition; Lakekeeper itself makes `external-id` mandatory only for `aws-system-identity` credentials. On the database side, replace the `set_storage_secret(...)` call -in Section 4 with `SELECT coldfront.set_storage_secret_vended();`. +above with `SELECT coldfront.set_storage_secret_vended();`. -### 3c. Pre-Create the `public` Namespace +### Google Cloud Storage -Resolve the warehouse id, then create the `public` namespace under it: +Google Cloud Storage is *not a separate Lakekeeper profile*: it uses an +`s3` storage profile pointed at GCS's S3-interoperability endpoint, +with an HMAC key pair standing in for an access key: + +```bash +curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" \ + -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", + "bucket": "my-iceberg-bucket", + "key-prefix": "coldfront", + "region": "us-east-1", + "endpoint": "https://storage.googleapis.com", + "path-style-access": true, + "flavor": "s3-compat", + "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", + "credential-type": "access-key", + "aws-access-key-id": "", + "aws-secret-access-key": "" + } + }' +# => HTTP 201 +``` + +Lakekeeper's native `gcs` profile is service-account only and is +**not** used here. Set the ColdFront credential the same way as any +other S3-compatible endpoint: + +```sql +SELECT coldfront.set_storage_secret('', '', + 'storage.googleapis.com', 'us-east-1', 'path', true); +``` + +GCS has no STS equivalent on the interoperability endpoint, so it has +no vended-credentials variant; it always uses this static HMAC key +pair. + +### Azure ADLS Gen2 + +Azure ADLS Gen2 uses its own Lakekeeper warehouse type, `adls`, naming +a storage account and a filesystem (container) within it: + +```bash +curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" \ + -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "adls", + "filesystem": "my-iceberg-fs", + "account-name": "myaccount" + }, + "storage-credential": { + "type": "az", + "credential-type": "shared-access-key", + "key": "" + } + }' +# => HTTP 201 +``` + +The warehouse's `storage-credential` is Lakekeeper's own credential for +reaching the account; it is separate from the credential ColdFront +itself uses. Set ColdFront's credential with +`set_storage_secret_azure()` instead of `set_storage_secret()` - it +takes a CONFIG-provider connection string, with the account key riding +inside `AccountKey=…`: + +```sql +SELECT coldfront.set_storage_secret_azure( + 'DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=;EndpointSuffix=core.windows.net'); +``` + +`set_storage_secret_azure()` writes the same `coldfront.storage_secret` row +(replicated, `pg_dump`-excluded) and materializes a `TYPE azure` PERSISTENT +SECRET. The Azure cold tier is subject to the soft-delete / change-feed +restriction in [Caveats](index.md#caveats). + +A vended-credentials variant is also available: set `sas-enabled` on the +warehouse (on by default) and Lakekeeper vends a per-container SAS +token; the warehouse credential can then be a `shared-access-key`, +`client-credentials`, or `azure-system-identity`. On the database side, +use `SELECT coldfront.set_storage_secret_vended('azure');` instead of +`set_storage_secret_azure()`. + +### Pre-Creating the Namespace + +Regardless of backend, resolve the warehouse id and pre-create the +`public` namespace under it: ```bash WID=$(curl -s http://localhost:8181/management/v1/warehouse \ | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) -[ -n "$WID" ] || { echo "no warehouse id - did step 3b return 201?"; exit 1; } +[ -n "$WID" ] || { echo "no warehouse id - did the warehouse create return 201?"; exit 1; } curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ -H "Content-Type: application/json" \ @@ -224,19 +394,22 @@ curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ !!! note "Why this step is required (decoupled mode)" - `coldfront.create_iceberg_table()` (Section 5) runs `CREATE SCHEMA` and - `CREATE TABLE` in one transaction. duckdb-iceberg defers the schema create - to `COMMIT` but sends the table-create POST immediately, so against a - namespace-less warehouse it fails with HTTP 404. Pre-creating `public` + `coldfront.create_iceberg_table()` (in + [Testing with Decoupled Mode](#testing-with-decoupled-mode) below) runs + `CREATE SCHEMA` and `CREATE TABLE` in one transaction. duckdb-iceberg + defers the schema create to `COMMIT` but sends the table-create POST + immediately, so against a namespace-less warehouse it fails with HTTP + 404. Pre-creating `public` makes the in-transaction `CREATE SCHEMA IF NOT EXISTS` a no-op. (Tiered mode's archiver creates the namespace itself, so this is only needed for the decoupled demo below.) --- -## 4. Database Setup +## Creating the Supporting Extensions -Open psql inside the Postgres container: +Next, connect to the server with psql and open a session inside the +Postgres container: ```bash docker exec -it "$(docker compose ps -q db)" psql -U coldfront -d coldfront @@ -258,32 +431,22 @@ CREATE EXTENSION IF NOT EXISTS coldfront; `CREATE EXTENSION` it once. Skip that step and the next call fails with `schema "coldfront" does not exist`. -Set the cold-tier S3 credential once. The signature is -`set_storage_secret(key_id, secret, endpoint, region, url_style, use_ssl)`; the -last four default to `NULL`, `'us-east-1'`, `'path'`, `false`: - -```sql -SELECT coldfront.set_storage_secret('AKIAEXAMPLE...', '', NULL, 'eu-west-1'); -``` - -!!! warning "The 3rd argument (endpoint) must be NULL for a cloud S3 endpoint" - - A `NULL` endpoint selects DuckDB's native per-Region virtual-hosted + HTTPS - addressing. The 4th argument is your bucket's region, from which DuckDB - derives the endpoint. The SeaweedFS form you may have seen elsewhere passes - a non-NULL endpoint and no region - do **not** use that shape for cloud S3. +Now run the `set_storage_secret` (or `set_storage_secret_azure` / +`set_storage_secret_vended`) call from whichever backend subsection you +followed in [Configuring Lakekeeper](#configuring-lakekeeper) above. --- -## 5. Fastest Demo - Decoupled Mode (The Table Lives Entirely in S3) +## Testing with Decoupled Mode A decoupled (iceberg-only) table has no Postgres hot tier; every row lives in -Iceberg on S3, and you read and write it through a normal-looking Postgres -relation. This is the quickest way to prove the whole path works: +Iceberg on your object store, and you read and write it through a +normal-looking Postgres relation. This is the quickest way to prove the whole +path works: ```sql SELECT coldfront.create_iceberg_table( - 'public', 's3_demo', + 'public', 'cold_demo', '[ {"name": "id", "type": "bigint"}, {"name": "ts", "type": "timestamptz"}, @@ -291,73 +454,85 @@ SELECT coldfront.create_iceberg_table( ]'::jsonb, '{month(ts)}'); -INSERT INTO public.s3_demo VALUES (1, now(), 'hello from S3'); -INSERT INTO public.s3_demo VALUES (2, now(), 'second row'); +INSERT INTO public.cold_demo VALUES (1, now(), 'hello from the cold tier'); +INSERT INTO public.cold_demo VALUES (2, now(), 'second row'); -SELECT count(*) AS n, max(note) AS last FROM public.s3_demo; +SELECT count(*) AS n, max(note) AS last FROM public.cold_demo; -- => n = 2, last = 'second row' ``` -That `count(*) = 2` is read back through `iceberg_scan` from your real S3 -bucket - the round trip is complete. +That `count(*) = 2` is read back through `iceberg_scan` from your real +object store - the round trip is complete. -### Tiered Mode - The Headline Feature +### Tiered Mode is the Headline Feature -Decoupled mode is the warm-up. ColdFront's real purpose is **tiered** tables: a -partitioned Postgres table whose hot partitions automatically age out to -Iceberg on S3 once they pass a retention window, after which reads -transparently union live Postgres data with cold S3 data and writes route to -the correct tier. +Decoupled mode is just the warm-up. ColdFront's real purpose is +**tiered** tables: a partitioned Postgres table whose hot partitions +automatically age out to the cold tier once they pass a retention +window, after which reads transparently union live Postgres data with +cold data and writes route to the correct tier. You drive tiered tables with the `archiver` binary against a small YAML config -(Postgres DSN, the `wh` warehouse, your S3 region/keys); each table's lifecycle -is registered with `archiver register`. The credential and warehouse you set up -above are exactly what it needs. See [usage.md](usage.md) for the archiver -config and the partition CLI. +(Postgres DSN, your warehouse name, and your backend's credentials); each +table's lifecycle is registered with `archiver register`. The credential and +warehouse you set up above are exactly what it needs. See +[usage.md](using_coldfront/index.md) for the archiver config and the partition CLI. --- -## 6. Verify in S3 and Troubleshoot +## Verifying the Configuration + +You can confirm the objects physically landed in your object store +using that backend's own client. The following table shows an example +command per backend: + +| Backend | Example verification command | +|---|---| +| S3-compatible / AWS S3 | `aws s3 ls s3://my-iceberg-bucket/coldfront/ --recursive` | +| Google Cloud Storage | `gsutil ls -r gs://my-iceberg-bucket/coldfront/` | +| Azure ADLS Gen2 | `az storage blob list --account-name myaccount --container-name my-iceberg-fs --prefix coldfront --output table` | -Confirm objects physically landed (an S3 client such as the `aws` CLI, region -exported): +For AWS S3, export your region first: ```bash export AWS_DEFAULT_REGION=eu-west-1 aws s3 ls s3://my-iceberg-bucket/coldfront/ --recursive ``` -Iceberg stores each table directly under your `key-prefix`, as -`coldfront//`, with a `data/` directory (parquet) and a `metadata/` -directory (metadata JSON, `*.avro` manifests, snapshot files) - UUID paths, not -your table name. (A namespace created on a Lakekeeper release before 0.13 keeps -its `coldfront///` layout.) "Where did `s3_demo` -go?" → look under the UUID path beneath your `key-prefix`. +Iceberg stores each table directly under your `key-prefix` (S3-family +backends) or filesystem root (Azure), as `coldfront//`, +with a `data/` directory (parquet) and a `metadata/` directory +(metadata JSON, `*.avro` manifests, snapshot files) - UUID paths, not +your table name. (A namespace created on a Lakekeeper release before +0.13 keeps its `coldfront///` layout.) +"Where did `cold_demo` go?" → look under the UUID path beneath your +`key-prefix`. Work through the following checklist if something failed: - Check that you ran `CREATE EXTENSION coldfront` once in your database. Preloading is not the same as creating the extension, and a missing extension produces `schema "coldfront" does not exist`. -- Check that `set_storage_secret(..., NULL, 'eu-west-1')` has `NULL` as its - third argument (native virtual-hosted addressing over HTTPS) and your real - region as its fourth. A non-NULL endpoint applies `url_style` and `use_ssl`, - which default to path-style over plain HTTP. A wrong region sends requests to - another Region's endpoint, which S3 rejects (with HTTP 400 for a bucket in a - Region launched after 2019-03-20). +- For S3-family backends, check that `set_storage_secret(...)` has the right + endpoint argument for your backend: `NULL` for virtual-hosted cloud S3, or an + explicit host for an S3-compatible store or GCS. A wrong region sends + requests to another Region's endpoint, which S3 rejects (with HTTP 400 for a + bucket in a Region launched after 2019-03-20). +- For Azure, check that the connection string passed to + `set_storage_secret_azure()` has the right `AccountName` and `AccountKey`. - Check that the namespace `public` exists in Lakekeeper before `create_iceberg_table` runs. Without the namespace, the decoupled create fails with HTTP 404. - Check that the key is a long-term key, not an SSO or temporary session-token - credential. + credential (S3-family backends). ## Next Steps To go further with ColdFront, consult the following guides: -- The [Using ColdFront](usage.md) guide covers the archiver config, the - partition CLI, and the other storage backends. -- The [Compaction](compaction.md) guide covers cold-tier maintenance on the - bucket. -- The [Architecture](architecture.md) overview describes how the cold tier is +- The [Using ColdFront](using_coldfront/index.md) guide details the archiver config, the + partition CLI, and vended credentials in more detail. +- The [Compaction](compaction.md) guide documents cold-tier maintenance on + the object store. +- The [Architecture](architecture_guides/index.md) overview describes how the cold tier is read and written. diff --git a/docs/one_time_setup.md b/docs/one_time_setup.md new file mode 100644 index 00000000..10c88e7b --- /dev/null +++ b/docs/one_time_setup.md @@ -0,0 +1,253 @@ +# Configuring Lakekeeper + +The setup needs three services configured once per deployment: +PostgreSQL with the pg_duckdb and coldfront extensions, Lakekeeper, and +any object store (SeaweedFS, MinIO, AWS S3, GCS, or Azure ADLS Gen2). +This guide brings up and bootstraps Lakekeeper, then installs and +configures the PostgreSQL side; do this before creating any table. + +Lakekeeper is not part of ColdFront's own build; it is a separate +upstream project ([github.com/lakekeeper/lakekeeper](https://github.com/lakekeeper/lakekeeper)). +Choose how you are running it below. + +## With Docker + +Bring up the end-user stack first (the example uses SeaweedFS, gated +behind the `local-store` compose profile; host ports are published so +the `localhost` commands below work directly). For the image build +itself, see [installation.md](installation.md): + +```bash +docker compose --profile local-store up -d --build +``` + +Lakekeeper is now reachable at `localhost:8181` with no further setup. +Bootstrap it, create the warehouse, and pre-create the Iceberg +namespace: + +```bash +# 1. Bootstrap Lakekeeper +curl -X POST http://localhost:8181/management/v1/bootstrap \ + -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' + +# 2. Create warehouse (adjust endpoint/credentials for your S3 store) +curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", "bucket": "iceberg", "region": "us-east-1", + "endpoint": "http://seaweedfs:8333", "path-style-access": true, + "flavor": "s3-compat", "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", "credential-type": "access-key", + "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" + } + }' + +# 3. Create the Iceberg namespace in the new warehouse. +# REQUIRED for decoupled (iceberg-only) mode: the Iceberg CREATE SCHEMA is +# deferred to transaction COMMIT but the CREATE TABLE POST is sent +# immediately, so coldfront.create_iceberg_table - which runs both in one +# transaction - would fail with HTTP 404 against a cold warehouse. +# Pre-creating the namespace here (its own committed REST call) makes the +# function's in-txn CREATE SCHEMA IF NOT EXISTS a no-op so the table create +# succeeds. The archiver (tiered mode) creates the namespace itself and does +# not need this. +WID=$(curl -s http://localhost:8181/management/v1/warehouse \ + | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) +curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' +``` + +## Installing Lakekeeper + +On bare metal, how you get the `lakekeeper` binary in place depends on +your installation path; continue with Preparing and Starting Lakekeeper +below either way. + +### Performing a Package Installation + +The `pgedge-lakekeeper` package installs the `lakekeeper` binary and a +`lakekeeper` systemd service, but it does not enable or start the +service. + +### Building from Source + +Download the matching upstream prebuilt release directly - there is no +source build; the release ships a single stripped binary (glibc 2.34 or +later required): + +```bash +ARCH=$(uname -m) # x86_64 or aarch64 +LAKEKEEPER_VERSION=0.13.1 +curl -LO "https://github.com/lakekeeper/lakekeeper/releases/download/v${LAKEKEEPER_VERSION}/lakekeeper-${ARCH}-unknown-linux-gnu.tar.gz" +tar xzf "lakekeeper-${ARCH}-unknown-linux-gnu.tar.gz" +sudo install -m 0755 lakekeeper /usr/bin/lakekeeper +``` + +Create the system user the steps below run as: + +```bash +sudo groupadd -r lakekeeper +sudo useradd -r -g lakekeeper -d /var/lib/lakekeeper -s /sbin/nologin \ + -c "Lakekeeper Iceberg REST Catalog" lakekeeper +``` + +This repository's own packaging templates make convenient starting +points for the environment file and systemd unit, since you already +have the repo cloned to build the `coldfront` extension: +`packaging/lakekeeper/common/lakekeeper.env` and +`packaging/lakekeeper/common/lakekeeper.service`. Copy them into +`/etc/lakekeeper/lakekeeper.env` and the systemd unit directory. + +## Preparing and Starting Lakekeeper + +Lakekeeper stores its catalog in a PostgreSQL 15 or later database and +reads its settings from `/etc/lakekeeper/lakekeeper.env`. The +following steps prepare and start the service, whichever way you +installed it: + +1. Create a role and a database for the catalog: + + ```sql + CREATE ROLE lakekeeper LOGIN PASSWORD 'change-me'; + CREATE DATABASE lakekeeper OWNER lakekeeper; + ``` + + The migration in step 3 creates the `uuid-ossp`, `pgcrypto`, `pg_trgm`, + `btree_gin`, and `btree_gist` extensions, so either the role must be + allowed to run `CREATE EXTENSION` or a superuser must create them first. + +2. In `/etc/lakekeeper/lakekeeper.env`, set + `LAKEKEEPER__PG_DATABASE_URL_WRITE` to the database's connection string and + `LAKEKEEPER__PG_ENCRYPTION_KEY` to a random secret, such as the output of + `openssl rand -base64 32`. Lakekeeper encrypts stored credentials with the + key, so keep the key stable and backed up. Every node that shares the + catalog needs the same key. + +3. Run the one-time database migration as the `lakekeeper` user: + + ```bash + set -a; . /etc/lakekeeper/lakekeeper.env; set +a + sudo -E -u lakekeeper /usr/bin/lakekeeper migrate + ``` + +4. Enable and start the service: + + ```bash + sudo systemctl enable --now lakekeeper + ``` + +Lakekeeper listens on port 8181 on every address by default. Without an +authorization backend in `lakekeeper.env`, the catalog accepts every request, +so configure authentication and authorization before you expose the service +beyond a trusted network. + +## Bootstrapping Lakekeeper + +Bootstrap Lakekeeper, create the warehouse, and pre-create the Iceberg +namespace: + +```bash +# 1. Bootstrap Lakekeeper +curl -X POST http://localhost:8181/management/v1/bootstrap \ + -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' + +# 2. Create warehouse (adjust endpoint/credentials for your S3 store) +curl -X POST http://localhost:8181/management/v1/warehouse \ + -H "Content-Type: application/json" -d '{ + "warehouse-name": "wh", + "storage-profile": { + "type": "s3", "bucket": "iceberg", "region": "us-east-1", + "endpoint": "http://seaweedfs:8333", "path-style-access": true, + "flavor": "s3-compat", "sts-enabled": false, + "remote-signing-enabled": false + }, + "storage-credential": { + "type": "s3", "credential-type": "access-key", + "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" + } + }' + +# 3. Create the Iceberg namespace in the new warehouse. +# REQUIRED for decoupled (iceberg-only) mode: the Iceberg CREATE SCHEMA is +# deferred to transaction COMMIT but the CREATE TABLE POST is sent +# immediately, so coldfront.create_iceberg_table - which runs both in one +# transaction - would fail with HTTP 404 against a cold warehouse. +# Pre-creating the namespace here (its own committed REST call) makes the +# function's in-txn CREATE SCHEMA IF NOT EXISTS a no-op so the table create +# succeeds. The archiver (tiered mode) creates the namespace itself and does +# not need this. +WID=$(curl -s http://localhost:8181/management/v1/warehouse \ + | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) +curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ + -H "Content-Type: application/json" -d '{"namespace":["public"]}' +``` + +## Configuring PostgreSQL for Lakekeeper + +Once Lakekeeper is bootstrapped, install the extensions and set the +cold-tier credentials, once per database: + +```sql +CREATE EXTENSION IF NOT EXISTS pg_duckdb; +CREATE EXTENSION IF NOT EXISTS coldfront; + +SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); +``` + +The secret is stored in the `coldfront.storage_secret` table (excluded from +`pg_dump`; in a Spock mesh it replicates by value once added to the default +repset - one-time mesh setup step 4) and materialized as a DuckDB PERSISTENT +SECRET that loads at instance init. There is **no per-session setup**: the +Iceberg catalog `ice` attaches **lazily** by the coldfront C hook on the first +query that touches a tiered/decoupled view (read or write). + +Every `set_storage_secret()` argument after the secret is optional. +`p_endpoint` defaults to `NULL`, `p_region` to `'us-east-1'`, `p_url_style` to +`'path'` (the other value is `'vhost'`), and `p_use_ssl` to `false`. +`p_url_style` and `p_use_ssl` apply only when an endpoint is given. A TLS +endpoint such as GCS's needs `p_use_ssl => true`, and AWS S3 omits the endpoint +and sets `p_region` to the bucket's Region. + +The image writes the server settings ColdFront needs into `postgresql.conf` +when it initializes a new data directory. A server installed from packages or +built another way sets them itself, as the +[package](configuration.md#configuring-postgresql) and +[bare-metal](installation.md#building-coldfront-on-bare-metal) sections of +installation.md show: + +- `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at + server start. coldfront refuses to load any other way: `CREATE EXTENSION + coldfront` fails on a server that does not preload it, with an error that + names the setting. +- `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` name the Lakekeeper + warehouse and its catalog endpoint, which the Iceberg catalog `ice` attaches + to. Only a superuser can set them, and while either is empty the catalog does + not attach. +- `coldfront.local_pg_dsn` is the connection string DuckDB uses to read + PostgreSQL tables from this server, which a decoupled table's + `INSERT … SELECT` from a PostgreSQL table needs, as does a cold write to a + table with a clustered vector column. A tiered `INSERT` streams its cold + rows through it, and renders them row by row when it is not set. Only a + superuser can set or read it. + Set it before calling `set_storage_secret()`, which installs the DuckDB + `postgres` extension this path loads only when the setting is present. + +An application that connects as a non-superuser needs its role granted access +with `coldfront.grant_app_access()`, which takes an existing role: + +```sql +SELECT coldfront.grant_app_access('alice'); +``` + +`coldfront.grant_app_access()` derives the grants from the registry when it +runs, so run it again after you create or adopt a table, or tier one for the +first time. The call fails while `duckdb.postgres_role` is unset, because only +superusers can then run DuckDB. +[Least-Privilege Application Roles](index.md#least-privilege-application-roles) +lists what it grants. + +For a real cloud-S3 setup, see [object_store.md](object_store.md). diff --git a/docs/reading_writing.md b/docs/reading_writing.md new file mode 100644 index 00000000..876331dd --- /dev/null +++ b/docs/reading_writing.md @@ -0,0 +1,39 @@ +# Reading and Writing ColdFront Tables + +The same SQL works against the relation name in either mode, as the following +examples show: + +```sql +-- Reads - pg_duckdb handles the iceberg side; PG handles the heap side +SELECT count(*) FROM events; +SELECT id, status, data->>'k' FROM events WHERE ts >= '2026-04-01'; + +-- Inserts, updates, and deletes all go through the coldfront C hook, +-- which rewrites the query into one PG-set-based statement (hot side) +-- + duckdb.raw_query calls (cold side). For iceberg-only mode every write +-- goes cold; for tiered mode the hook splits by ts vs the watermark. +INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{"k":1}'); +UPDATE events SET status = 'fixed' WHERE id = 123; +DELETE FROM events WHERE ts < '2025-01-01'; + +-- Bulk INSERT shapes: the source is read once, the hot rows are one +-- set-based INSERT and the cold rows are written in batches: +INSERT INTO events (ts, status, data) VALUES (...), (...), (...); +INSERT INTO events (ts, status, data) SELECT ts, status, data FROM staging; +INSERT INTO events (ts, status, data) SELECT now() + i*'1s'::interval, 'ok', '{}' + FROM generate_series(1, 1000) i; +-- A WITH clause works in both positions; a nested INSERT takes no RETURNING: +WITH moved AS (DELETE FROM staging RETURNING ts, status, data) +INSERT INTO events (ts, status, data) SELECT ts, status, data FROM moved; +WITH i AS (INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{}')) +SELECT 1; +-- COPY FROM loads the same way, one INSERT per cold_write_batch_size rows: +COPY events (ts, status, data) FROM '/path/to/events.csv' WITH (FORMAT csv); + +-- Transactions work; ROLLBACK undoes Iceberg writes too +BEGIN; + UPDATE events SET status = 'pending' WHERE id = 1; + SELECT status FROM events WHERE id = 1; -- sees 'pending' if row 1 is hot +ROLLBACK; +SELECT status FROM events WHERE id = 1; -- back to whatever it was +``` diff --git a/docs/source.md b/docs/source.md new file mode 100644 index 00000000..512cae5b --- /dev/null +++ b/docs/source.md @@ -0,0 +1,27 @@ +# Source + +This page preserves every page from the original documentation tree, +unchanged and in its original order, as a reference while the +navigation above is reorganized into the standard pgEdge structure. + +- [pgEdge ColdFront](index.md) +- [Installing and Configuring ColdFront](installation.md) +- Getting Started + - [Getting Started with ColdFront](walkthrough.md) + - [Exploring Tiered Storage](walkthrough_tiered.md) + - [Exploring Decoupled Mode](walkthrough_decoupled.md) + - [Exploring the Standalone Partitioner](walkthrough_partitioner.md) + - [Exploring Distributed Mode](walkthrough_distributed.md) + - [Configuring your Object Store](object_store.md) +- Architecture + - [pgEdge ColdFront - Architecture](architecture_guides/index.md) + - [ColdFront - Tiered Operating Mode](architecture_tiered.md) + - [Decoupled (Iceberg-Only) Operating Mode](architecture_decoupled.md) + - [Vector Storage Architecture](architecture_vectors.md) +- [Using ColdFront](using_coldfront/index.md) +- [Working with Embeddings](usage_vectors.md) +- [COMPACTOR - Cold-Tier Table Maintenance](compaction.md) +- Developer Resources + - [Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal)](formal/README.md) +- [Changelog](changelog.md) +- [License](LICENSE.md) diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css index c46afb00..93b9f038 100644 --- a/docs/stylesheets/extra.css +++ b/docs/stylesheets/extra.css @@ -15,3 +15,9 @@ --md-footer-logo-dark-mode: block; --md-footer-logo-light-mode: none; } + +/* Keep inline code (e.g. GUC names) from wrapping mid-word inside + narrow table columns; prose cells wrap normally. */ +.md-typeset table code { + white-space: nowrap; +} diff --git a/docs/supported_types.md b/docs/supported_types.md new file mode 100644 index 00000000..151e639a --- /dev/null +++ b/docs/supported_types.md @@ -0,0 +1,153 @@ +# Understanding Supported Column Types + +The following PostgreSQL column types are supported: + +`bigint` · `integer` · `smallint` · `real` · `double precision` · +`boolean` · `timestamp with time zone` · `timestamp without time zone` · +`date` · `time without time zone` · `uuid` · `text` · `varchar(N)` · +`char(N)` · `bytea` · `numeric(P,S)` (P ≤ 38) · `jsonb` / `json` · +`interval` · `vector(N)` / `halfvec(N)` (pgvector; see +[usage_vectors.md](usage_vectors.md)) · one-dimensional arrays of most of +these types (see [Arrays](#arrays)) + +Anything else (unbounded `numeric`, `xml`, `tsvector`, range/multirange types, +custom enums, composite types, pgvector's `sparsevec`, and the arrays that +[Arrays](#arrays) lists as refused at registration) is rejected when a tiered +table is registered, and when a decoupled table is created. ColdFront refuses +silent fallback to `varchar` - losing precision/identity is worse than no +support. + +Iceberg cannot hold a `numeric` `NaN`, or an `infinity` or `-infinity` in a +`date`, `timestamp` or `timestamptz` column, so ColdFront refuses those values, +as array elements too. A tiered table's hot table has a check constraint named +`coldfront_guard_` on each column of those types and on each array +column; an `INSERT` or `UPDATE` that stores one of the values fails with an +error that names the constraint. A write that goes to the cold tier, in either +mode, is refused by DuckDB. + +The archiver adds the constraints at the first archive pass that tiers a +partition of the table, and validates them once that pass's bootstrap +transaction has committed. A value written before then is accepted, and each +archive pass stops with an error that names the constraint until the row is +fixed; the next pass then validates the constraint and tiers the table. A +column that an `ALTER TABLE` adds through the DDL hook gets its constraint in +the same statement, which PostgreSQL validates against every row of the hot +table, so an `ADD COLUMN` whose default the constraint refuses, such as +`DEFAULT 'NaN'`, fails. A renamed column's constraint takes the new column +name, and a column whose name is longer than 47 bytes gets a constraint named +after a hash of the name. + +A cold `UPDATE ... FROM`, a `MERGE` whose source is a PostgreSQL table, and an +`INSERT ... SELECT` into a decoupled table read that PostgreSQL table through +DuckDB's postgres extension, which reads a `numeric` `NaN` as `0` and drops an +array's lower bound. ColdFront cannot refuse those values on that path, so +filter them out of such a source. + +`char(N)` is stored and read as `varchar`. The data round-trips losslessly: +values, comparisons, and `length()` match a hot PG table, where `length()` +already ignores `char(N)` trailing padding. The only difference is cosmetic: a +cold `char(N)` column reads back unpadded with `pg_typeof varchar`, because +pg_duckdb has no fixed-length `char` type. Use `text` or `varchar` if +blank-padded display matters. + +`json`, `jsonb` and `interval` are stored as `varchar` in Iceberg (no native +primitive), an `interval` as ` months days seconds`, +the text PostgreSQL and DuckDB both read as the same value. On read, +`interval` is view-cast back to the rich PG type; `json` +and `jsonb` come back as DuckDB's `json` (the equivalent of PG's `jsonb`), not +the rich PG `jsonb` type, because Iceberg-backed reads run entirely in DuckDB. +Queries like `data->>'key'` and `data->'key'` work, and ColdFront translates +the `::jsonb` cast, `jsonb_array_length`, `jsonb_build_object` / `jsonb_agg` +(and their `json_` twins, which DuckDB also lacks) and `date_bin` on read: the +builders become the `concat` / `to_json` / `array_agg` form both engines +evaluate identically (the result is JSON-equal to jsonb's rendering, in compact +form and in argument order), and `date_bin` becomes DuckDB's `time_bucket`, +which takes the same arguments and agrees on every fixed-width bucket. The +jsonb-only operators (`?`, `@>`, `<@`, `#>`, `#>>`) and most jsonb functions +(`jsonb_typeof`, `jsonb_extract_path`, `jsonb_extract_path_text`, `jsonb_set`, +`jsonb_path_*`, `jsonb_each`, `jsonb_object_keys`) are not supported on tiered +or decoupled data: DuckDB either lacks them or its same-named function differs +in signature or result. Reach into the document with `->`/`->>`. + +These limits apply only to reads that go through a tiered or iceberg-only view, +which pg_duckdb plans entirely in DuckDB. Ordinary (non-tiered) PostgreSQL +tables are untouched by ColdFront and keep full jsonb support, as does the hot +partition table itself. + +Bound parameters (`$1` from a prepared statement, a driver's extended protocol, +or a plpgsql variable) work in such reads. DuckDB types most placeholders from +their context (`ts > $1`); one that is a direct argument of a DuckDB function +with several overloads (`time_bucket`'s origin, so `date_bin`'s) or any +argument of a table function (`generate_series`) it cannot, so ColdFront plans +such a read from the bound values on every execution instead of caching a +generic plan. Under `plan_cache_mode = force_generic_plan` those reads cannot +run: the forced generic plan has no values and fails with +`only works with DuckDB execution`. + +A hot-only read is the exception. When a `SELECT` reads a tiered view directly +(no join, CTE, or sub-query) and its `WHERE` provably restricts to the hot +tier, ColdFront rewrites it to read the hot partition table in plain +PostgreSQL: the full jsonb operator and function set works, and the query skips +DuckDB entirely. Such a read returns `data` as native `jsonb` rather than the +view's `json`. Reads that span tiers, are cold-only, or reach the view through +a join or sub-query stay in DuckDB, where the limits above apply. + +`inet`/`cidr`/`oid` are **not supported**: pg_duckdb cannot process them +(`inet` Oid 869, `oid` Oid 26) in any Iceberg-backed query, and every +cross-tier read is planned by pg_duckdb - so no cast makes them readable. Store +IP data as `text` and `oid` values as `bigint` (you can still index/compare +them; cast on the hot side only if needed). + +### Arrays + +A one-dimensional array of a supported type tiers as an Iceberg list of that +type. The following table shows how each array type is stored and how the view +reads it back: + +| PG type | Iceberg/Parquet storage | Reads back as | +|---|---|---| +| `bigint[]`, `integer[]`, `real[]`, `double precision[]`, `boolean[]`, `date[]`, `time[]`, `timestamp[]`, `uuid[]`, `bytea[]`, `numeric(P,S)[]` | A list of the element's storage type. | The same type. | +| `smallint[]` | A list of `INTEGER`. | `integer[]`, as a `smallint` column reads as `integer`. | +| `text[]` | A list of `VARCHAR`. | `text[]`. | +| `varchar(N)[]`, `char(N)[]` | A list of `VARCHAR`. | `character varying[]`, with `char(N)` elements unpadded. | + +The following arrays are refused when a tiered table is registered and when a +decoupled table is created, with an error that names the column: + +- `timestamptz[]`, because pg_duckdb cannot read a `timestamptz[]` column of a + PostgreSQL table, which a tiered table's hot tier is; store `timestamp[]` in + UTC instead. +- `jsonb[]`, `json[]` and `interval[]`, because those types are stored as text + and read back through a cast that an array element does not get. +- arrays of `vector` or `halfvec`, because a vector is itself stored as a list, + and ColdFront stores an array as a list of scalar values. + +An Iceberg list has one dimension and numbers its elements from 1, so ColdFront +refuses an array with more than one dimension or with a lower bound other than +1, such as `'[0:2]={1,2,3}'`. The hot table's check constraint refuses such an +array when it is written. A write to the cold tier refuses it too: a tiered +`INSERT` with an error that names the column, and a bound parameter with an +error that names no column. A tiered table with a column declared with more +than one dimension, such as `integer[][]`, passes registration and is refused +at its first archive pass, because pg_duckdb reads a column by its declared +dimensions; a decoupled table refuses such a column when it is created. A +decoupled table's array type is spelled as its element type followed by `[]`, +such as `integer[]`; the spellings `integer[3]` and `integer ARRAY` are +refused. `ALTER TABLE ... ADD COLUMN` cannot add an array column to a tiered +table, because duckdb-iceberg does not add a nested column to an Iceberg table. + +A tiered `INSERT` accepts an array in any spelling, because its source runs in +PostgreSQL. A cold `UPDATE` and a decoupled `INSERT` run in DuckDB, which does +not read PostgreSQL's brace literal: a `'{a,b}'` literal reaches DuckDB as text +and fails to cast. Write the array as `ARRAY['a', 'b']`, or pass it as a bound +parameter. + +Reads through a view run in DuckDB once the table has a cold tier, and the +following array operations work there: + +- `x = ANY(arr)`, `arr @> ARRAY[...]` and `arr && ARRAY[...]`. +- subscripts and slices, which count from 1, such as `arr[1]` and `arr[1:2]`. +- `array_length(arr, 1)` and `unnest(arr)`. + +A brace literal and a bound array parameter fail in such a read, as they do for +a vector, and so does `cardinality()`, which DuckDB defines for maps only. diff --git a/docs/usage.md b/docs/usage.md deleted file mode 100644 index fcd2d9e9..00000000 --- a/docs/usage.md +++ /dev/null @@ -1,1460 +0,0 @@ -# Using ColdFront - -ColdFront offers two operating modes. Pick one per table; both can coexist in -the same database. The following table compares the two modes: - -| | Tiered (hot + cold) | Decoupled (iceberg-only) | -|---|---|---| -| Where rows live | Recent rows are hot in the PG heap, and archived rows are cold in Iceberg. | Every row is in Iceberg. | -| Setup | Create a partitioned table and let the archiver convert it on the first run that finds a partition older than `hot_period`. | One SQL call, `coldfront.create_iceberg_table(...)`, sets up the table. | -| Archiver | The archiver is required; it runs from cron and moves old partitions to cold. | The archiver is not used. | -| Best when | The workload has a recent-row OLTP part that benefits from PG indexes and transactional ergonomics. | The workload is purely analytic or append-mostly, and you want zero PG storage and stateless compute. | - -Once the table exists, **the SQL surface is identical**: `SELECT`, `INSERT`, -`UPDATE`, `DELETE` all work normally against the relation name (e.g. `events`), -except that a write touching the cold tier rejects `RETURNING`. - -## Prerequisites (Both Modes) - -The setup needs three services: PostgreSQL with the pg_duckdb and coldfront -extensions, Lakekeeper, and any S3-compatible object store (SeaweedFS, MinIO, -GCS, etc.). The one-time setup below brings them up and bootstraps them. - -## One-Time Setup - -Bring up the end-user stack (the example uses SeaweedFS, gated behind the -`local-store` compose profile; host ports are published so the `localhost` -commands below work directly). For the image build itself, see -[installation.md](installation.md): - -```bash -docker compose --profile local-store up -d --build -``` - -Then bootstrap Lakekeeper, create the warehouse, and pre-create the Iceberg -namespace: - -```bash -# 1. Bootstrap Lakekeeper -curl -X POST http://localhost:8181/management/v1/bootstrap \ - -H "Content-Type: application/json" -d '{"accept-terms-of-use":true}' - -# 2. Create warehouse (adjust endpoint/credentials for your S3 store) -curl -X POST http://localhost:8181/management/v1/warehouse \ - -H "Content-Type: application/json" -d '{ - "warehouse-name": "wh", - "storage-profile": { - "type": "s3", "bucket": "iceberg", "region": "us-east-1", - "endpoint": "http://seaweedfs:8333", "path-style-access": true, - "flavor": "s3-compat", "sts-enabled": false, - "remote-signing-enabled": false - }, - "storage-credential": { - "type": "s3", "credential-type": "access-key", - "aws-access-key-id": "admin", "aws-secret-access-key": "adminsecret" - } - }' - -# 3. Create the Iceberg namespace in the new warehouse. -# REQUIRED for decoupled (iceberg-only) mode: the Iceberg CREATE SCHEMA is -# deferred to transaction COMMIT but the CREATE TABLE POST is sent -# immediately, so coldfront.create_iceberg_table - which runs both in one -# transaction - would fail with HTTP 404 against a cold warehouse. -# Pre-creating the namespace here (its own committed REST call) makes the -# function's in-txn CREATE SCHEMA IF NOT EXISTS a no-op so the table create -# succeeds. The archiver (tiered mode) creates the namespace itself and does -# not need this. -WID=$(curl -s http://localhost:8181/management/v1/warehouse \ - | grep -oE '"warehouse-id":"[^"]+"' | head -1 | cut -d'"' -f4) -curl -X POST "http://localhost:8181/catalog/v1/$WID/namespaces" \ - -H "Content-Type: application/json" -d '{"namespace":["public"]}' -``` - -Then install the extensions and set the cold-tier credentials, once per -database: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; - -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); -``` - -The secret is stored in the `coldfront.storage_secret` table (excluded from -`pg_dump`; in a Spock mesh it replicates by value once added to the default -repset - one-time mesh setup step 4) and materialized as a DuckDB PERSISTENT -SECRET that loads at instance init. There is **no per-session setup**: the -Iceberg catalog `ice` attaches **lazily** by the coldfront C hook on the first -query that touches a tiered/decoupled view (read or write). - -Every `set_storage_secret()` argument after the secret is optional. -`p_endpoint` defaults to `NULL`, `p_region` to `'us-east-1'`, `p_url_style` to -`'path'` (the other value is `'vhost'`), and `p_use_ssl` to `false`. -`p_url_style` and `p_use_ssl` apply only when an endpoint is given. A TLS -endpoint such as GCS's needs `p_use_ssl => true`, and AWS S3 omits the endpoint -and sets `p_region` to the bucket's Region. - -The image writes the server settings ColdFront needs into `postgresql.conf` -when it initializes a new data directory. A server installed from packages or -built another way sets them itself, as the -[package](installation.md#configuring-postgresql) and -[bare-metal](installation.md#building-coldfront-on-bare-metal) sections of -installation.md show: - -- `shared_preload_libraries = 'pg_duckdb,coldfront'` loads both extensions at - server start. coldfront refuses to load any other way: `CREATE EXTENSION - coldfront` fails on a server that does not preload it, with an error that - names the setting. -- `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` name the Lakekeeper - warehouse and its catalog endpoint, which the Iceberg catalog `ice` attaches - to. Only a superuser can set them, and while either is empty the catalog does - not attach. -- `coldfront.local_pg_dsn` is the connection string DuckDB uses to read - PostgreSQL tables from this server, which a decoupled table's - `INSERT … SELECT` from a PostgreSQL table needs, as does a cold write to a - table with a clustered vector column. A tiered `INSERT` streams its cold - rows through it, and renders them row by row when it is not set. Only a - superuser can set or read it. - Set it before calling `set_storage_secret()`, which installs the DuckDB - `postgres` extension this path loads only when the setting is present. - -An application that connects as a non-superuser needs its role granted access -with `coldfront.grant_app_access()`, which takes an existing role: - -```sql -SELECT coldfront.grant_app_access('alice'); -``` - -`coldfront.grant_app_access()` derives the grants from the registry when it -runs, so run it again after you create or adopt a table, or tier one for the -first time. The call fails while `duckdb.postgres_role` is unset, because only -superusers can then run DuckDB. -[Least-Privilege Application Roles](index.md#least-privilege-application-roles) -lists what it grants. - -For a real cloud-S3 setup, see [object_store.md](object_store.md). - -## Mode 1 - Tiered (Hot + Cold) - -Create a partitioned table normally: - -```sql -CREATE TABLE events ( - id bigint GENERATED ALWAYS AS IDENTITY, - ts timestamptz NOT NULL, - status text, - data jsonb, - PRIMARY KEY (id, ts) -) PARTITION BY RANGE (ts); - -CREATE TABLE p_2026_04 PARTITION OF events - FOR VALUES FROM ('2026-04-01') TO ('2026-05-01'); -``` - -The archiver connects the way psql does, from the libpq environment (`PGHOST`, -`PGDATABASE`, `PGUSER`, `PGPASSWORD`, `PGSERVICE`) or `--dsn`, and reads every -other setting from the server: the tables from `coldfront.partition_config`, -the cold-store credential from `coldfront.storage_secret`, and the catalog from -the `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` settings. It -needs no file to run. - -A deployment YAML is a way to write that configuration into the server once, -with `import`: - -```yaml -s3: - endpoint: "seaweedfs:8333" - access_key: "admin" - secret_key: "adminsecret" -archiver: - tables: - - source_table: events - partition_period: monthly - hot_period: "1 month" -``` - -The `import` command writes the file into the server: - -```bash -./bin/archiver import --config config.yaml -``` - -`import` writes the `s3:` or `azure:` stanza through `set_storage_secret` or -`set_storage_secret_azure`, then upserts each table into -`coldfront.partition_config`, validated as `register` validates one table, so -the file and the server agree when it returns. A file may also name -`iceberg.warehouse` and `iceberg.lakekeeper_endpoint`; the import checks those -against the server settings, which live in `postgresql.conf`, before it writes -anything, and refuses the file if they differ. From then on the server is the -configuration: change it with `set`, `register`, `set_storage_secret` or -`postgresql.conf`. A YAML passed to any later run is checked, value by value, -against what the server holds, and any difference is refused with an error -that says where configuration lives. A file that agrees, such as the output of -`export`, runs. `s3.region` defaults to `us-east-1`; -[Storage Backends](#storage-backends) describes the other `s3:` and `azure:` -keys, and a deployment on [vended credentials](#vended-credentials) has no -stanza to import. - -Register a single table in `coldfront.partition_config` without a file: - -```bash -./bin/archiver register --table events \ - --period monthly --hot-period "1 month" -# optional: --retention "5 years" DROPs cold data past that age -# (must exceed --hot-period; omit = keep forever) -``` - -Run the archiver (typically via cron): - -```bash -./bin/archiver -``` - -`--version` prints the version that `make build` stamps from `git describe` -(the release tag, or the commit it was built from) and exits; a plain -`go build` prints `unknown`. The partitioner and compactor accept the same -flag. - -The archiver also accepts `--debug-export-delay`, a test-only Go duration -(such as `5s`) that holds each partition's capture window open between the -bulk export and the replay, so that a test can race concurrent writes into it. -Leave it unset in production. - -The first run that finds a partition older than `hot_period` renames `events` -→ `_events`, creates the unified view `events`, and registers it; until then -the archiver only premakes partitions, and the table stays a plain partitioned -table. From then on every cycle (1) tiers partitions older than `hot_period` -from hot PG to cold Iceberg and advances the watermark, and (2) if -`retention_period` is set, drops cold Iceberg rows older than it. The data -lifecycle is **hot → `hot_period` → cold → `retention_period` → gone**; -omit `retention_period` to keep cold data forever. - -### Inbound Foreign Keys - -A tiered table cannot be the target of an enforced foreign key from another -table over its archivable range. Archiving physically removes the rows from -PostgreSQL (export to Iceberg, then `DETACH` + `DROP` the partition), and -PostgreSQL cannot enforce a foreign key against the cold tier. So the `DETACH` -is refused (SQLSTATE 23503) and the archiver fails fast, naming the blocking -constraint. - -This is inherent, not an edge case: referencing rows rarely have a hot-only -lifecycle. They usually outlive the partition they point at, so the foreign key -still pins those rows in PostgreSQL by the time the partition ages into the -cold tier. - -The foreign key is fundamentally incompatible with archiving the rows it -references, so before archiving such a table, drop it: - -```sql -ALTER TABLE event_logs DROP CONSTRAINT event_logs_events_fkey; -``` - -Better still, do not point an enforced foreign key at a tiered table over its -archivable range in the first place. - -### 2-Level (LIST → RANGE) Tiered Tables - -A table partitioned `LIST (region) → RANGE (ts)` can be tiered too - the same -`sub_partition` block as the standalone partition manager (Mode 3), so a -partition-manager-managed table can be "upgraded" to tiered by pointing the -archiver at it: - -```yaml - - source_table: regional - partition_column: ts # the RANGE (time) column - required for 2-level - partition_period: monthly - hot_period: 1 month - # retention_period: 5 years # optional cold expiry (region-agnostic, by ts) - sub_partition: - values_source: "SELECT region FROM regions" -``` - -One Iceberg table holds every region (region is an ordinary column); the -archiver tiers leaves a whole `ts` period at a time across **all** regions -before advancing the shared hot/cold watermark, so a period only becomes cold -once it is cold for every region. `id` mode is not supported in tiered mode -(the cold tier is time-keyed). For why the ordering works that way, see the -[Two-level tiering](architecture_tiered.md#two-level-list-range-tiering) -section of the Tiered Mode page. - -## Mode 2 - Decoupled (Iceberg-Only) - -A single call creates a decoupled table: - -```sql -SELECT coldfront.create_iceberg_table( - p_schema => 'public', - p_table => 'events', - p_columns => '[ - {"name":"id", "type":"bigint"}, - {"name":"ts", "type":"timestamptz"}, - {"name":"status", "type":"text"}, - {"name":"data", "type":"jsonb"} - ]'::jsonb, - p_partition_cols => '{month(ts)}' -); -``` - -`p_partition_cols` partitions the Iceberg table, here by the month of `ts`: -each month's rows land in their own data files, and a query with a time filter -skips the months outside it before reading anything. Each element is one term -as DuckDB's `PARTITIONED BY` takes it: a column name, `year(col)`, `month(col)` -or `day(col)` on a timestamp or date column, `hour(col)` on a timestamp column, -`bucket(N, col)` or `truncate(W, col)`; `'{month(ts), region}'` partitions by -both. The argument is a PostgreSQL array literal, so a term that contains a -comma is double-quoted inside it, the comma being the array's delimiter: -`'{"bucket(16, id)"}'`. A column name that itself needs double quotes has them -backslashed inside such an element: `'{"month(\"Event Time\")"}'`. Leave the -argument out for an unpartitioned table. - -That single statement provisions: - -- `ice.public.events` on the attached Iceberg catalog. -- a PG-side wrapper view `public.events` with proper PG-typed columns. - -- a `coldfront.tiered_views` registry row, so that the coldfront C hook - intercepts every `INSERT`, `UPDATE`, `DELETE`, and `MERGE` on the view and - rewrites each one to a single `duckdb.raw_query(...)` against - `ice.public.events`. - - -In a mesh one node provisions: Spock's `ddl_sql` repset replicates the -`CREATE VIEW`, and the `default` repset replicates the name-keyed registry row, -which enables the write hook on every peer. Each node also needs the one-time -setup call `coldfront.ensure_replicated()` - see the one-time mesh setup -below. - -### Changing a Decoupled Table's Columns - -An `ALTER TABLE` on the wrapper view adds, drops, renames or widens a column -of the Iceberg table, and the view follows, with its owner and table-level -grants kept; a column-level grant on the view is not kept. Only the view's -owner, or a role holding its privileges, can change its columns, and that role -needs the cold access `grant_app_access` gives, since the Iceberg change runs -as that role. The following statements change the columns of the table created -above: - -```sql -ALTER TABLE public.events ADD COLUMN qty integer; -ALTER TABLE public.events ALTER COLUMN qty TYPE bigint; -ALTER TABLE public.events RENAME COLUMN qty TO quantity; -ALTER TABLE public.events DROP COLUMN quantity; -``` - -A column takes the same type map as at creation, though an array column cannot -be added, because duckdb-iceberg does not add a nested column to an Iceberg -table. A type change is limited to the widening Iceberg accepts: `integer` to -`bigint`, `real` to `double precision`, `date` to `timestamp`, and a wider -`numeric` precision. An added column takes a name and a type only: a default, -a constraint, a collation, or a storage or compression option is refused, as -is a `USING` or `COLLATE` clause on a type change, and a column change shares -an `ALTER TABLE` only with other column changes. A vector column is neither -added nor changed, a table adopted read-only refuses every column change, and -an object built on the view, such as another view, makes a column change -fail. In a mesh, the change runs on one node: the Iceberg table is shared, and -the rebuilt view and registry row replicate. - -### Adopting a Table That Already Exists in the Catalog - -A table another engine wrote needs no provisioning, only a wrapper view and a -registry row. `coldfront.adopt_iceberg_table()` reads the schema from the -catalog and builds both, so the call names the relation and nothing else: - -```sql -SELECT coldfront.adopt_iceberg_table( - p_schema => 'public', - p_table => 'orders', - p_namespace => 'lake' -); -``` - -`p_schema` is the PostgreSQL schema the wrapper view goes in, and it must -exist. `p_namespace` is the Iceberg namespace the table lives in, which need -not exist as a PostgreSQL schema; it defaults to `p_schema` when omitted. The -view takes the table's name. - -Adoption is read-only by default, so reading someone else's lake table cannot -become writing it by accident. Passing `p_writable => true` enables the same -`INSERT`, `UPDATE`, `DELETE` and `MERGE` rewrite a created table gets: - -```sql -SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', - p_writable => true); -``` - -Because Iceberg records no PostgreSQL type, an adopted column reads as whatever -its storage type maps to; a `jsonb` column comes back as `text`. Pass `p_types` -to restore one, provided the override maps to the type the catalog already -stores: - -```sql -SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', - p_writable => true, - p_types => '{"meta":"jsonb"}'::jsonb); -``` - -Adoption binds the name once. A second call under a registered name is refused -whatever its arguments; to enable writes or change an override, release the -table with `coldfront.release_iceberg_table()` and adopt it again. A -non-superuser deployment runs `coldfront.grant_app_access()` after adopting, -because the grant reads the registry at call time. - -In a mesh one node adopts. The wrapper view replicates through Spock's -`ddl_sql` repset and the registry row through the `default` repset (see the -one-time mesh setup below), so every peer reads and writes the table under the -same name, and a peer's own adopt is refused as already registered. Adoption -takes the table's bakery claim around that check and the registry write, so two -nodes adopting the same table at the same moment are serialized: the second -waits, then reads the first's row and refuses, rather than both registering the -one table. A writer outside ColdFront is outside the bakery and can still -collide at Lakekeeper. - -One relation can be registered per Iceberg table. A namespace that needs -quoting, such as `Lake-EU`, works. - -## Mode 3 - Standalone Partition Manager (No Cold Tier) - -**You do not need Iceberg at all.** If automated PostgreSQL partition -maintenance is all you want - declarative time- or id-based RANGE partitioning -with a premade forward window and automatic age-out of old partitions - -ColdFront's `partitioner` binary is the whole product: stock PostgreSQL (or a -Spock mesh), no cold tier, no DuckDB, no Iceberg, nothing to preload. Each -invocation makes one reconcile pass per managed table: premake the forward -window, ensure the partition covering *now* exists, and detach-then-drop -partitions past retention (`DETACH ... CONCURRENTLY`, never a bare `DROP` of -attached data). Build the partitioner with `make build` and run it from cron: - -```bash -go build -o bin/partitioner ./cmd/partitioner # or: make build -./bin/partitioner -``` - -A database set up this way can gain the cold tier later: -`CREATE EXTENSION coldfront` adopts the `coldfront.partition_config` the -partitioner created, registrations included. - -The partitioner connects from the libpq environment or `--dsn`, like psql, and -reads its tables from `coldfront.partition_config`; it needs no file. A -partition-only deployment has no cold tier: a table with a `hot_period` -belongs to the archiver, and the partitioner leaves it alone. - -Register each managed table in `coldfront.partition_config`: - -```bash -./bin/partitioner register --table events \ - --period monthly --retention "12 months" \ - --strategy drop # drop (DETACH+DROP, destroy; default) - # | detach (DETACH only, keep as a - # standalone table - data preserved) -``` - -### Operating It - -Schedule one pass per period or more often - a cron line, or a systemd -`oneshot` service plus timer (systemd then reports a failed pass as a failed -unit, so alerting is free): - -```text -17 * * * * postgres PGDATABASE=mydb /usr/local/bin/partitioner >> /var/log/coldfront-partitioner.log 2>&1 -``` - -Keep the following operational behavior in mind when scheduling it: - -- The exit code is `0` when every table reconciled (a self-healed *behind* - condition still exits `0`) and non-zero when at least one table failed - (`N table(s) failed`), each failure logged with a `[table]` prefix (the bare - table name) followed by the schema-qualified error. Alert on a non-zero exit - code. -- If a table already has a *past* partition but none covers *now* at the start - of a pass (a lagging cron - live inserts had no home), the pass heals the - table (creates the current partition), logs a `WARNING (self-healed)` line, - and still exits `0`. Monitor for that warning in the log rather than via the - exit code, and widen `future_partitions` or run more often. A fresh table - (only just-premade future partitions) is **not** behind: its first reconcile - succeeds cleanly. -- With the default `expiration_strategy: drop`, expiry is - `DETACH CONCURRENTLY` + `DROP TABLE` - the data is **gone**, so back up - before shrinking `retention_period`. Set `expiration_strategy: detach` to - instead leave the expired partition as a standalone table (detached from the - parent, data preserved) and reclaim it yourself. A partition is expired only - once its *entire* range is older than `now − retention_period`, computed - with calendar-accurate PostgreSQL interval arithmetic (a real month, leap - years correct). `detach` is partition-only - the tiered archiver always drops - after exporting to cold. - -### Primary Keys on Time-Partitioned Tables (id Mode) - -PostgreSQL requires a unique index to include the partition key, so a plain -`PRIMARY KEY (id)` is impossible on a table partitioned by a separate `ts` -column. Partition instead by RANGE on a **time-ordered id** and the partition -key *is* the key - a single-column `PRIMARY KEY (id)`: - -```yaml - - source_table: events - partition_column: id - partition_period: monthly - retention_period: 12 months - part_mode: id - id_scheme: snowflake # snowflake | uuidv7 -``` - -`uuidv7` reads the RFC 9562 leading-millisecond timestamp; `snowflake` decodes -the pgEdge Snowflake extension's layout. Either way the manager computes the -month/day partition bounds as id values, so id-order equals time-order and the -forward/retention schedule is unchanged. - -### Two-Level (LIST → RANGE) Sub-Partitioning - -For a table partitioned by `LIST (region)` whose children are themselves -`RANGE`-partitioned by time, add a `sub_partition` block. `values_source` is a -query returning the current level-1 values; the manager provisions and -maintains a RANGE sub-tree under each, creating sub-trees for newly appearing -values automatically: - -```yaml - - source_table: events - partition_column: ts # the level-2 RANGE column - partition_period: monthly - retention_period: 12 months - sub_partition: - values_source: "SELECT code FROM regions" -``` - -Each level-1 child is named `
_`, with the value lowercased and -every character outside `a-z`, `0-9` and `_` replaced by `_`. That name must -fit in 50 bytes, which leaves room for the daily leaf suffix within -PostgreSQL's 63-byte identifier limit. A pass that meets a longer name fails, -for tiered two-level tables as well. The partitioner and the archiver also -fail a pass in which two values map to the same name. - -## Dropping an Iceberg Table (Both Modes) - -`coldfront.drop_iceberg_table()` removes the Iceberg table backing a registered -relation. The function is the only sanctioned way to do so: a plain -`DROP TABLE` or `DROP VIEW` on a registered relation stays blocked, because it -would leave the cold tier behind with nothing pointing at it. - -The call takes the schema, the table, and an explicit purge decision: - -```sql --- drop the catalog entry and delete the stored objects -SELECT coldfront.drop_iceberg_table('public', 'events', true); - --- drop the catalog entry and leave the objects in the object store -SELECT coldfront.drop_iceberg_table('public', 'events', false); -``` - -What remains afterwards depends on the mode, and the call reports which path it -took: - -- in decoupled mode the Iceberg table is the whole relation, so nothing remains - in PostgreSQL. -- in tiered mode the Iceberg table is the cold tier, so the cold tier is - removed and the hot table returns under the relation's own name, as an - ordinary partitioned table holding the data that had not yet aged out. - -In tiered mode the call also deletes the table's `partition_config` row, which -stops the archiver from tiering it again on the next run. - -There is no default for the purge argument, because the two outcomes are -irreversible in opposite directions. Passing `true` deletes the Parquet and -metadata objects, which for the cold tier are the only copy of that data. -Passing `false` keeps those objects but removes the catalog entry that -ColdFront would need to reach them again, so nothing reclaims them afterwards; -use it when another system is taking ownership of the files. - -ColdFront has no guard of its own against dropping the wrong table. The -preventive control lives in Lakekeeper: table protection has to be enabled -manually, per table, and a protected table cannot be dropped at all, whether or -not purge is requested. This function cannot override a hold. - -Deletion is not instantaneous. The catalog entry disappears with the call, and -Lakekeeper's own background queue removes the objects shortly afterwards, so a -listing taken immediately after the call can still show them. - -### Handing an Adopted Table Back - -An adopted table is released rather than dropped, because ColdFront does not -own it. `coldfront.release_iceberg_table()` removes the wrapper view, the -registry row and the relation's vector configuration, and performs no Iceberg -I/O, so the table keeps every row and stays in the catalog: - -```sql -SELECT coldfront.release_iceberg_table('public', 'orders'); -``` - -Release refuses a tiered registration, because removing one would leave its -cold rows unreachable while the hot table returned under the relation's name. -`drop_iceberg_table()` refuses a relation adopted read-only, for the mirrored -reason: read access gives no authority to destroy. - -## Managing Partitioned Tables (CLI) - -ColdFront's configuration lives in the server. **Per-table lifecycle** lives in -`coldfront.partition_config`, a name-keyed table that replicates by value -across a Spock mesh (like `tiered_views`/`archive_watermark`), so every node -reads identical config - no per-node file syncing. The cold-store credential -lives in `coldfront.storage_secret` and the catalog in the -`coldfront.warehouse` and `coldfront.lakekeeper_endpoint` settings. The -**connection** alone comes from outside, from the libpq environment or -`--dsn`, as for psql. Manage the lifecycle table with the CLI below (both -`partitioner` and `archiver` expose these subcommands; with no subcommand they -do their normal reconcile/archive run). - -`register` is the primary way to manage tables: one command adds or adopts one -table, validated on the spot. `import` and `export` are bulk helpers for -(re)configuring a machine - write a deployment YAML into a fresh server, or -dump the live config to git and replay it on another node - not the day-to-day -path. A YAML passed to anything but `import` is checked against the server and -refused if it disagrees in any value. `register` and `import` run the full -validation. `set` checks every value of the row that it would leave; it re-runs -the rest of the validation only when it changes the partition period or column, -`hot_period`, `retention_period` or the sub-partition source. - -The data lifecycle is **hot PG → `hot_period` → cold Iceberg → -`retention_period` → dropped** (tiered) or **hot PG → `retention_period` -→ dropped** (partition-only). Setting `hot_period` makes a table tiered; -omitting it makes it partition-only. - -### What Registration Refuses - -`register` and `import` run the same validation, so a table that one rejects -cannot be added by the other; `set` re-runs it when it changes a validated -field. Registration fails when: - -- a setting has a value that the table's `CHECK` constraints refuse. The - refusal names the flag, or on `import` the key, and the values that the - setting accepts; `set` checks the row that its change would leave. - Registration refuses: - - - a `--period` other than `monthly` or `daily`. - - a `--premake` below 1. - - a `--part-mode` other than `timestamp` or `id`. - - an `--id-scheme` other than `uuidv7` or `snowflake`, or one without - `--part-mode id`. - - `--part-mode id` without an `--id-scheme`. - - a `--strategy` other than `drop` or `detach`. - - `--part-mode id` or `--strategy detach` with a `--hot-period`. - - a `--sub-values-source` without a `--column`. - - a table with neither a `--hot-period` nor a `--retention`. - -- any relation in the partition tree is `UNLOGGED`, because that data is - truncated after a crash and replicates nowhere, so archiving it would - preserve rows PostgreSQL never promised to keep. -- the table has a `DEFAULT` partition. Rows the `DEFAULT` partition catches - have no time bounds, so they can never be tiered or expired, and PostgreSQL - refuses `DETACH PARTITION ... CONCURRENTLY` for every partition of a table - that has one, which is how partitions are expired. The partition's mere - existence is enough; it does not have to hold any rows. Move any rows the - `DEFAULT` partition holds into real partitions and detach it. -- the name differs only by case from an already-registered table. PostgreSQL - keeps `public."Events"` and `public.events` apart, but DuckDB matches - identifiers case-insensitively even when they are quoted, so both names would - resolve to one Iceberg table and overwrite each other. -- the name starts with an underscore, which is reserved for the tiered hot - table. -- the name leaves no room for the generated partition suffix: 53 bytes for - monthly and 50 for daily, within PostgreSQL's 63-byte identifier limit. -- a tiered table has a column whose type has no Iceberg equivalent (see - [Supported Column Types](#supported-column-types)). Every column goes through - the same type map the cold tier itself uses, so the answer comes back at the - prompt rather than hours later from cron, and the refusal names the column. - Partition-only tables are exempt: nothing about them reaches Iceberg, so - their column types are PostgreSQL's business alone. -- a tiered table's name, or the name of one of its columns, is one DuckDB - parses as a keyword, such as `by`, `at` or `show` (see [Caveats](#caveats)). - The archiver checks the column names again before it tiers or expires data; - it refuses a column added after registration before any table or data reaches - Iceberg. - -Registration validates the table as it is at that moment, not continuously. -Adding a `DEFAULT` partition to an already-registered table is therefore not -caught then; the next archiver run fails while reading that table's partition -bounds. - -Other unusual bounds are supported rather than refused. Partitions open at -either end (`MINVALUE`, `MAXVALUE`, `infinity`) are handled, as is PostgreSQL's -full timestamp range, from 4713 BC through year 294276. - -`hot_period` and `retention_period` are native PostgreSQL `interval`s - use any -interval syntax (`1 month`, `90 days`, `1 year 2 mons`, `5 years`). Expiry -boundaries are computed with calendar-accurate interval arithmetic -(`now() - period`: real months, leap years), and `retention_period` must exceed -`hot_period` (validated at `register`/`set` time). - -The following table describes the CLI subcommands: - -| Command | Purpose | -|---|---| -| `register` | Adds or adopts a table, and validates that the PRIMARY KEY covers the partition key. | -| `list` | Shows the managed tables and their lifecycle. | -| `set` | Changes fields, or disables or enables a table with `--disable`/`--enable`. | -| `remove` | Stops managing a table; the table itself is left intact. | -| `import` | Writes a deployment YAML into the server once: its `archiver.tables` as rows, validated as `register` validates each table, and its `s3:` or `azure:` stanza as the storage secret. | -| `export` | Dumps the active (enabled) config to YAML or SQL, as a git-reviewable backup to replay on another node. | - -The following examples register, inspect, and change managed tables: - -```bash -# Partition-only: keep 3 future partitions, drop those older than 12 months. -partitioner register --table events --period monthly --retention "12 months" - -# Partition-only, but DETACH (preserve) expired partitions instead of dropping them. -partitioner register --table events --period monthly \ - --retention "12 months" --strategy detach - -# Tiered: tier to cold Iceberg after 1 month, then drop cold data after 5 years. -archiver register --table events --period monthly \ - --hot-period "1 month" --retention "5 years" - -# id mode - a real single-column PRIMARY KEY (id) on a snowflake-keyed table. -partitioner register --table events --period monthly \ - --column id --part-mode id --id-scheme snowflake --retention "1 year" - -# 2-level LIST(region) → RANGE(ts), tiered; region values come from a table. -archiver register --table regional --period monthly --column ts \ - --hot-period "1 month" --sub-values-source "SELECT region FROM regions" - -partitioner list # what's managed -partitioner set --table events --retention "24 months" -partitioner set --table events --disable # pause (keeps the row) -partitioner remove --table events # unregister, keep the table -partitioner import --config deploy.yaml # write a deployment YAML into the server once -partitioner export > managed.yaml # active config, git-reviewable (--format sql for INSERTs) -``` - -Run `partitioner` (or `archiver`) with `help` or `--help` for the command -overview, and with no arguments for its normal run; every subcommand has a -detailed `--help` with worked examples. The write commands accept -`--print-sql` (emit the SQL without running it - review/commit it; `import` -prints its table `INSERT`s); `register` and `import` also accept `--dry-run`. -`set --enable`/`--disable` (mutually exclusive) pause/resume a table without -removing it; a disabled table is skipped by reconcile and omitted from -`export`. Per table, only the cadence and a lifecycle boundary (`hot_period` or -`retention_period`) are required; `partition_column` is auto-detected from -`pg_catalog` for flat tables (required for 2-level). `register` checks the -lifecycle rules before it writes the row, and the table's `CHECK` constraints -enforce them again at write time. - -The subcommands also take the following connection and table flags: - -- `--dsn` takes a PostgreSQL connection string on every subcommand. Without - it the connection comes from the libpq environment, as for psql. -- `--config` names a deployment YAML. On `import` it is required, since it - names the file to write into the server. On any other subcommand the file is - checked against the server, value by value, and refused if it disagrees; its - `postgres.dsn` connects when `--dsn` is unset. -- `--schema` (default `public`) names the table's schema on `register`, `set` - and `remove`. -- `--premake` (default `3`) sets on `register` how many future partitions are - kept ahead of now, stored as `future_partitions`. - -`set` changes only the fields whose flags are passed: `--period`, `--column`, -`--premake`, `--hot-period`, `--retention`, `--sub-values-source` and -`--strategy`, each taking the same value as on `register`. An empty value -clears the field, so `--hot-period ""` makes a tiered table partition-only and -hands it to the partitioner. A change that leaves neither a `hot_period` nor a -`retention_period` is refused. - -For managing tables, the archiver and partitioner read the managed set only -from `coldfront.partition_config`. `register` adds one table; `import` adds -every table in a YAML `archiver.tables` list. Both write through the same -validation (the PK must cover the partition key; `retention` must exceed -`hot_period`), so an imported table is checked exactly as a registered one. -`export` dumps the active rows back to YAML or SQL for git. - -The archiver manages only the rows that have a `hot_period`, and the -partitioner only the rows without one, so each table belongs to one binary. A -run that finds none of its own rows exits non-zero with -`no tables in coldfront.partition_config`, naming the `register` and `import` -commands that add one. - -In an `archiver.tables` entry, `source_schema` names the table's schema and -defaults to `public`. `source_table` also accepts the `schema.table` form, and -an explicit `source_schema` takes precedence over that prefix. -`future_partitions` defaults to `3`, as `--premake` does on `register`. - -## Storage Backends - -Configure **exactly one** of the following cold-store backends, through -`set_storage_secret` or through the `s3:` or `azure:` stanza of a deployment -YAML, which `import` writes through the same setters. - -### S3 - -The S3 backend works with any S3-compatible store (SeaweedFS, MinIO). Set -`endpoint`, `use_ssl: true` for a TLS endpoint, and `url_style: path` (default) -or `vhost`. `region` defaults to `us-east-1`. - -### Virtual-Hosted Cloud S3 - -For virtual-hosted cloud S3 (AWS S3 is the canonical one), **omit `endpoint`** -(and the `endpoint` arg to `set_storage_secret`) so DuckDB uses the native -per-Region virtual-hosted + HTTPS addressing; set `region` to your bucket's -Region. DuckDB derives the endpoint from `region`, and a bucket in a Region -launched after 2019-03-20 (e.g. `ap-south-2`) rejects requests sent to another -Region's endpoint with HTTP 400. The Lakekeeper warehouse profile must be a -virtual-hosted `s3` profile (`flavor: aws`, `path-style-access: false`, no -custom endpoint); the full walkthrough is [object_store.md](object_store.md). - -### Google Cloud Storage - -Google Cloud Storage is *not a separate backend*: use `s3:` pointed at GCS's -S3-interoperability endpoint with an -[HMAC key pair](https://cloud.google.com/storage/docs/authentication/hmackeys) -(`endpoint: storage.googleapis.com`, `use_ssl: true`, -`access_key`/`secret_key` = the HMAC id/secret). Lakekeeper's warehouse uses an -`s3` profile (`flavor: s3-compat`, `path-style`) at the same endpoint. -ColdFront's test suite covers this setup end to end, reading and writing -Iceberg through the interoperability endpoint. Lakekeeper's native `gcs` -profile is service-account only and is **not** used. - -### Azure ADLS Gen2 - -Azure ADLS Gen2 is a supported cold-store backend; the access key rides inside -`connection_string`. Set the credential with `set_storage_secret_azure()` -instead of `set_storage_secret()` - it takes a CONFIG-provider connection -string. The storage-account access key rides inside `AccountKey=…`; the -DuckDB azure secret has no separate account-key parameter, so shared-key auth -lives entirely in the connection string: - -```sql -SELECT coldfront.set_storage_secret_azure( - 'DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net'); -``` - -`set_storage_secret_azure()` writes the same `coldfront.storage_secret` row -(replicated, `pg_dump`-excluded) and materializes a `TYPE azure` PERSISTENT -SECRET. The Azure cold tier is subject to the soft-delete / change-feed -restriction in [Caveats](index.md#caveats). - -## Vended Credentials - -Vended credentials let a deployment run with no object-store credential stored -in the database, in a DuckDB secret file, or in an archiver config; this suits -compliance environments that forbid persisting long-term keys. Lakekeeper -issues a short-lived, per-table credential (an S3 STS access key, secret, and -session token) at read and write time, and ColdFront uses it directly. The -long-term credential lives only in the Lakekeeper warehouse. - -Enable vended credentials with a single call that stores no credential: - -```sql -SELECT coldfront.set_storage_secret_vended(); -``` - -This writes a `coldfront.storage_secret` row marked as vended, so nothing is -materialized as a DuckDB secret; `ensure_attached()` then attaches the catalog -with credential vending turned on. The archiver and the compactor read the -same row, so a vended deployment has no `s3:` or `azure:` stanza to import and -no credential in any file. - -Vended mode targets the two clouds that issue scoped credentials: AWS S3 (STS) -and Azure ADLS Gen2 (SAS). The `set_storage_secret_vended()` call above enables -AWS S3; the same call with `'azure'` enables Azure through the identical path: - -```sql -SELECT coldfront.set_storage_secret_vended('azure'); -``` - -Vending requires a Lakekeeper warehouse configured to vend credentials: - -- On AWS S3, the warehouse uses `flavor: aws` with `sts-enabled: true`, an - `assume-role-arn` for a bucket-scoped IAM role, and an `external-id` on the - warehouse credential that the role's trust policy also requires. The full - warehouse and IAM-role setup is in - [object_store.md](object_store.md#3b-create-the-s3-warehouse). -- On Azure ADLS Gen2, the warehouse is an `adls` warehouse with `sas-enabled` - (on by default). Lakekeeper vends a per-container SAS token; the warehouse - credential can be a `shared-access-key`, `client-credentials`, or - `azure-system-identity`. - -Google Cloud Storage over the S3-interoperability endpoint has no STS to issue -short-lived credentials, so GCS stays on static HMAC credentials. - -The compactor runs fully under vended credentials: compaction, snapshot expiry, -and orphan-file reclaim all use the vended per-table credentials. - -Switching a running deployment between static and vended credentials changes -the attach mode, which is fixed per PostgreSQL backend at attach time; open a -new session (or restart the archiver and compactor, which are short-lived -processes) so the new mode takes effect. - -## Reading + Writing (Identical for Both Modes) - -The same SQL works against the relation name in either mode, as the following -examples show: - -```sql --- Reads - pg_duckdb handles the iceberg side; PG handles the heap side -SELECT count(*) FROM events; -SELECT id, status, data->>'k' FROM events WHERE ts >= '2026-04-01'; - --- Inserts, updates, and deletes all go through the coldfront C hook, --- which rewrites the query into one PG-set-based statement (hot side) --- + duckdb.raw_query calls (cold side). For iceberg-only mode every write --- goes cold; for tiered mode the hook splits by ts vs the watermark. -INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{"k":1}'); -UPDATE events SET status = 'fixed' WHERE id = 123; -DELETE FROM events WHERE ts < '2025-01-01'; - --- Bulk INSERT shapes: the source is read once, the hot rows are one --- set-based INSERT and the cold rows are written in batches: -INSERT INTO events (ts, status, data) VALUES (...), (...), (...); -INSERT INTO events (ts, status, data) SELECT ts, status, data FROM staging; -INSERT INTO events (ts, status, data) SELECT now() + i*'1s'::interval, 'ok', '{}' - FROM generate_series(1, 1000) i; --- A WITH clause works in both positions; a nested INSERT takes no RETURNING: -WITH moved AS (DELETE FROM staging RETURNING ts, status, data) -INSERT INTO events (ts, status, data) SELECT ts, status, data FROM moved; -WITH i AS (INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{}')) -SELECT 1; --- COPY FROM loads the same way, one INSERT per cold_write_batch_size rows: -COPY events (ts, status, data) FROM '/path/to/events.csv' WITH (FORMAT csv); - --- Transactions work; ROLLBACK undoes Iceberg writes too -BEGIN; - UPDATE events SET status = 'pending' WHERE id = 1; - SELECT status FROM events WHERE id = 1; -- sees 'pending' if row 1 is hot -ROLLBACK; -SELECT status FROM events WHERE id = 1; -- back to whatever it was -``` - -## Supported Column Types - -The following PostgreSQL column types are supported: - -`bigint` · `integer` · `smallint` · `real` · `double precision` · -`boolean` · `timestamp with time zone` · `timestamp without time zone` · -`date` · `time without time zone` · `uuid` · `text` · `varchar(N)` · -`char(N)` · `bytea` · `numeric(P,S)` (P ≤ 38) · `jsonb` / `json` · -`interval` · `vector(N)` / `halfvec(N)` (pgvector; see -[usage_vectors.md](usage_vectors.md)) · one-dimensional arrays of most of -these types (see [Arrays](#arrays)) - -Anything else (unbounded `numeric`, `xml`, `tsvector`, range/multirange types, -custom enums, composite types, pgvector's `sparsevec`, and the arrays that -[Arrays](#arrays) lists as refused at registration) is rejected when a tiered -table is registered, and when a decoupled table is created. ColdFront refuses -silent fallback to `varchar` - losing precision/identity is worse than no -support. - -Iceberg cannot hold a `numeric` `NaN`, or an `infinity` or `-infinity` in a -`date`, `timestamp` or `timestamptz` column, so ColdFront refuses those values, -as array elements too. A tiered table's hot table has a check constraint named -`coldfront_guard_` on each column of those types and on each array -column; an `INSERT` or `UPDATE` that stores one of the values fails with an -error that names the constraint. A write that goes to the cold tier, in either -mode, is refused by DuckDB. - -The archiver adds the constraints at the first archive pass that tiers a -partition of the table, and validates them once that pass's bootstrap -transaction has committed. A value written before then is accepted, and each -archive pass stops with an error that names the constraint until the row is -fixed; the next pass then validates the constraint and tiers the table. A -column that an `ALTER TABLE` adds through the DDL hook gets its constraint in -the same statement, which PostgreSQL validates against every row of the hot -table, so an `ADD COLUMN` whose default the constraint refuses, such as -`DEFAULT 'NaN'`, fails. A renamed column's constraint takes the new column -name, and a column whose name is longer than 47 bytes gets a constraint named -after a hash of the name. - -A cold `UPDATE ... FROM`, a `MERGE` whose source is a PostgreSQL table, and an -`INSERT ... SELECT` into a decoupled table read that PostgreSQL table through -DuckDB's postgres extension, which reads a `numeric` `NaN` as `0` and drops an -array's lower bound. ColdFront cannot refuse those values on that path, so -filter them out of such a source. - -`char(N)` is stored and read as `varchar`. The data round-trips losslessly: -values, comparisons, and `length()` match a hot PG table, where `length()` -already ignores `char(N)` trailing padding. The only difference is cosmetic: a -cold `char(N)` column reads back unpadded with `pg_typeof varchar`, because -pg_duckdb has no fixed-length `char` type. Use `text` or `varchar` if -blank-padded display matters. - -`json`, `jsonb` and `interval` are stored as `varchar` in Iceberg (no native -primitive), an `interval` as ` months days seconds`, -the text PostgreSQL and DuckDB both read as the same value. On read, `interval` is view-cast back to the rich PG type; `json` -and `jsonb` come back as DuckDB's `json` (the equivalent of PG's `jsonb`), not -the rich PG `jsonb` type, because Iceberg-backed reads run entirely in DuckDB. -Queries like `data->>'key'` and `data->'key'` work, and ColdFront translates -the `::jsonb` cast, `jsonb_array_length`, `jsonb_build_object` / `jsonb_agg` -(and their `json_` twins, which DuckDB also lacks) and `date_bin` on read: the -builders become the `concat` / `to_json` / `array_agg` form both engines -evaluate identically (the result is JSON-equal to jsonb's rendering, in compact -form and in argument order), and `date_bin` becomes DuckDB's `time_bucket`, -which takes the same arguments and agrees on every fixed-width bucket. The -jsonb-only operators (`?`, `@>`, `<@`, `#>`, `#>>`) and most jsonb functions -(`jsonb_typeof`, `jsonb_extract_path`, `jsonb_extract_path_text`, `jsonb_set`, -`jsonb_path_*`, `jsonb_each`, `jsonb_object_keys`) are not supported on tiered -or decoupled data: DuckDB either lacks them or its same-named function differs -in signature or result. Reach into the document with `->`/`->>`. - -These limits apply only to reads that go through a tiered or iceberg-only view, -which pg_duckdb plans entirely in DuckDB. Ordinary (non-tiered) PostgreSQL -tables are untouched by ColdFront and keep full jsonb support, as does the hot -partition table itself. - -Bound parameters (`$1` from a prepared statement, a driver's extended protocol, -or a plpgsql variable) work in such reads. DuckDB types most placeholders from -their context (`ts > $1`); one that is a direct argument of a DuckDB function -with several overloads (`time_bucket`'s origin, so `date_bin`'s) or any -argument of a table function (`generate_series`) it cannot, so ColdFront plans -such a read from the bound values on every execution instead of caching a -generic plan. Under `plan_cache_mode = force_generic_plan` those reads cannot -run: the forced generic plan has no values and fails with -`only works with DuckDB execution`. - -A hot-only read is the exception. When a `SELECT` reads a tiered view directly -(no join, CTE, or sub-query) and its `WHERE` provably restricts to the hot -tier, ColdFront rewrites it to read the hot partition table in plain -PostgreSQL: the full jsonb operator and function set works, and the query skips -DuckDB entirely. Such a read returns `data` as native `jsonb` rather than the -view's `json`. Reads that span tiers, are cold-only, or reach the view through -a join or sub-query stay in DuckDB, where the limits above apply. - -`inet`/`cidr`/`oid` are **not supported**: pg_duckdb cannot process them -(`inet` Oid 869, `oid` Oid 26) in any Iceberg-backed query, and every -cross-tier read is planned by pg_duckdb - so no cast makes them readable. Store -IP data as `text` and `oid` values as `bigint` (you can still index/compare -them; cast on the hot side only if needed). - -### Arrays - -A one-dimensional array of a supported type tiers as an Iceberg list of that -type. The following table shows how each array type is stored and how the view -reads it back: - -| PG type | Iceberg/Parquet storage | Reads back as | -|---|---|---| -| `bigint[]`, `integer[]`, `real[]`, `double precision[]`, `boolean[]`, `date[]`, `time[]`, `timestamp[]`, `uuid[]`, `bytea[]`, `numeric(P,S)[]` | A list of the element's storage type. | The same type. | -| `smallint[]` | A list of `INTEGER`. | `integer[]`, as a `smallint` column reads as `integer`. | -| `text[]` | A list of `VARCHAR`. | `text[]`. | -| `varchar(N)[]`, `char(N)[]` | A list of `VARCHAR`. | `character varying[]`, with `char(N)` elements unpadded. | - -The following arrays are refused when a tiered table is registered and when a -decoupled table is created, with an error that names the column: - -- `timestamptz[]`, because pg_duckdb cannot read a `timestamptz[]` column of a - PostgreSQL table, which a tiered table's hot tier is; store `timestamp[]` in - UTC instead. -- `jsonb[]`, `json[]` and `interval[]`, because those types are stored as text - and read back through a cast that an array element does not get. -- arrays of `vector` or `halfvec`, because a vector is itself stored as a list, - and ColdFront stores an array as a list of scalar values. - -An Iceberg list has one dimension and numbers its elements from 1, so ColdFront -refuses an array with more than one dimension or with a lower bound other than -1, such as `'[0:2]={1,2,3}'`. The hot table's check constraint refuses such an -array when it is written. A write to the cold tier refuses it too: a tiered -`INSERT` with an error that names the column, and a bound parameter with an -error that names no column. A tiered table with a column declared with more -than one dimension, such as `integer[][]`, passes registration and is refused -at its first archive pass, because pg_duckdb reads a column by its declared -dimensions; a decoupled table refuses such a column when it is created. A -decoupled table's array type is spelled as its element type followed by `[]`, -such as `integer[]`; the spellings `integer[3]` and `integer ARRAY` are -refused. `ALTER TABLE ... ADD COLUMN` cannot add an array column to a tiered -table, because duckdb-iceberg does not add a nested column to an Iceberg table. - -A tiered `INSERT` accepts an array in any spelling, because its source runs in -PostgreSQL. A cold `UPDATE` and a decoupled `INSERT` run in DuckDB, which does -not read PostgreSQL's brace literal: a `'{a,b}'` literal reaches DuckDB as text -and fails to cast. Write the array as `ARRAY['a', 'b']`, or pass it as a bound -parameter. - -Reads through a view run in DuckDB once the table has a cold tier, and the -following array operations work there: - -- `x = ANY(arr)`, `arr @> ARRAY[...]` and `arr && ARRAY[...]`. -- subscripts and slices, which count from 1, such as `arr[1]` and `arr[1:2]`. -- `array_length(arr, 1)` and `unnest(arr)`. - -A brace literal and a bound array parameter fail in such a read, as they do for -a vector, and so does `cardinality()`, which DuckDB defines for maps only. - -## Caveats - -Keep the following caveats in mind when running either mode: - -- `jsonb` reads come back as `json`. The `->`/`->>` operators work, and - ColdFront translates the `::jsonb` cast, `jsonb_array_length`, - `jsonb_build_object` / `jsonb_agg` and `date_bin`; other jsonb operators and - functions are unsupported on cold or cross-tier reads. A hot-only read of the - view runs in PostgreSQL with full jsonb (see - [Supported Column Types](#supported-column-types)). -- Cross-tier isolation differs from PG-native isolation. The cold tier keeps - one Iceberg snapshot for the whole transaction, even at READ COMMITTED, so a - commit by another session after the transaction's first cold read stays - invisible until the transaction ends. The hot tier follows PostgreSQL's own - isolation level, so at READ COMMITTED a later statement can see new hot rows - but not new cold ones. Within one transaction, a read of a tiered or - decoupled table sees the transaction's own writes on both tiers. The held - snapshot is also why the compactor's `--expire-older-than` must stay longer - than the longest transaction that reads the table (see - [compaction.md](compaction.md)). -- One transaction block cannot hold both a PostgreSQL write and a cold-tier - write unless `duckdb.unsafe_allow_mixed_transactions` is on. pg_duckdb - refuses the pair ("Writing to DuckDB and Postgres tables in the same - transaction block is not supported"), at the PostgreSQL write when it comes - second and at `COMMIT` when it came first. A cold-tier write here is a - tiered `UPDATE` or `DELETE` whose `WHERE` bounds the cold tier, or any write - to a decoupled table. The statements that write both tiers themselves (a - dual-tier `UPDATE` or `DELETE`, a tiered `INSERT` with cold rows, a - cross-tier move) set the parameter `LOCAL` for the rest of their - transaction, so the block they ran in accepts later writes of either kind. - You can set it yourself, `SET LOCAL duckdb.unsafe_allow_mixed_transactions = - on`, at your own risk: pg_duckdb commits the Iceberg snapshot at - `PRE_COMMIT`, so a backend crash between that and the PostgreSQL commit - record keeps the cold write and loses the PostgreSQL writes. -- In decoupled mode, pg_duckdb commits the Iceberg snapshot at PRE_COMMIT, so a - backend crash after that but before the PG commit record leaves the Iceberg - write committed and the PG side lost. A crash after the Parquet upload but - before the commit POST leaves unreferenced objects, which the compactor's - orphan pass reclaims. -- In decoupled mode, concurrent writes from multiple PG nodes are serialized - PG-side by the bakery protocol: every iceberg-only `INSERT` goes through - `coldfront._exec_iceberg_with_claim`, which holds a globally-ordered - Snowflake ticket and waits for its turn before committing to Lakekeeper. - There are no 409 conflicts and no app-level retries. The protocol is - Lamport-1978 mutex with the Ricart-Agrawala (1981) deferred-reply - optimization; claims and acks replicate as Spock rows and it stays safe under - Spock's asymmetric apply (modeled in - [docs/formal/Bakery.tla](https://github.com/pgEdge/ColdFront/blob/main/docs/formal/Bakery.tla)). - The bakery requires the `snowflake` extension, the `coldfront.loopback_dsn` - GUC (the node's loopback, a unix-socket DSN), and a one-time - `SELECT coldfront.ensure_replicated()` call on every node after Spock mesh - setup; see - [architecture_decoupled.md](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). - Sync-rep is **not** required. The throughput ceiling is Lakekeeper's commit - rate, not the writer count. -- For direct table access, `_events` is the hot heap (tiered mode only). - `ice.public.` is the Iceberg table - only addressable via - `iceberg_scan(...)` or `duckdb.raw_query('… ice.… …')`, never via - PG-native 3-part names. -- A tiered `INSERT` streams its cold rows: the cold projection runs in - PostgreSQL, in a second session the DuckDB `postgres` extension opens from - `coldfront.local_pg_dsn`, and its rows go straight into the Iceberg writer - in one pass. An omitted IDENTITY column takes `nextval()` on the hot table's - sequence, so cold ids share it with the hot side, and an omitted column with - a DEFAULT takes it. The hot rows are one set-based `INSERT`. The second - session reads the source at the statement's own snapshot, so both tiers see - one state of the tables whatever another session commits meanwhile, and a - REPEATABLE READ transaction reads its snapshot on both tiers. A source the - stream cannot read the same way twice (a volatile or stable function such as - `now()`, a temporary table, a `WITH` entry, a table the transaction has - written, a SERIALIZABLE transaction, or a second table-reading `INSERT` in - one READ COMMITTED transaction) goes through - `coldfront._cold_sink` instead, which renders each row in plpgsql and writes - Iceberg in batches of `coldfront.cold_write_batch_size` rows, as does a - server without `coldfront.local_pg_dsn`. The statement's `cold_rows` column - is `NULL` when the cold rows streamed. A value that reaches the second - session or DuckDB as text is rendered in a form every reader takes as the - same value, whatever the caller's `DateStyle`, `IntervalStyle` or - `extra_float_digits`. -- `COPY FROM` reads the rows with PostgreSQL's `COPY` reader and writes - them through that same `INSERT` path, `coldfront.cold_write_batch_size` rows - per `INSERT`. The format options are the reader's, and a supplied value for a - `GENERATED ALWAYS` identity column is kept, as `COPY` into a table keeps it. - `COPY ... WHERE` and the `FREEZE`, `ON_ERROR`, `REJECT_LIMIT` and `DEFAULT` - options are refused. -- An `INSERT`, `UPDATE` or `DELETE` nested in a `WITH` entry goes through the - same rewrite as a top-level one. With a watermark an `INSERT`'s row may go - cold, so `RETURNING` on it is refused, as on a top-level `INSERT`; a hot - `UPDATE` or `DELETE` keeps `RETURNING`, a cold or dual-tier one cannot - return rows or read another `WITH` entry (its DuckDB half does not see - them), and a cross-tier move cannot be a `WITH` entry. A statement may write - a tiered view once, and the nested write may not have a `WITH` clause of - its own. On a decoupled view the source runs in DuckDB, so a `WITH` entry - that modifies data is refused (DuckDB 1.5 has no data-modifying `WITH`), and - a nested `INSERT` may not read another `WITH` entry. A statement that holds - a nested write and also reads a tiered view runs in DuckDB as a whole, and - pg_duckdb refuses it ("DuckDB does not support modifying CTEs"); read the - hot table, or split the statement. -- `MERGE INTO `, on PostgreSQL 17 and later (16 allows `MERGE` on tables - alone), runs on the tier its `ON` condition bounds the partition column to - (`AND t.ts >= ''` for the hot tier, `AND t.ts < ''` for the - cold tier): a hot `MERGE` runs in PostgreSQL against the hot table - and keeps `RETURNING`, a cold one in DuckDB against the Iceberg table. Each - tier sees its own rows alone, so a `MERGE` that bounds neither tier is - refused, and an `INSERT` action's row must belong to the statement's tier: a - hot `MERGE` refuses a row below the cutoff, a cold one a row at or after it; - insert such rows with `INSERT`, which splits them. A cold `INSERT` action - must give the identity column a value (`OVERRIDING SYSTEM VALUE` for a - `GENERATED ALWAYS` one), a cold `MERGE` cannot return rows and runs one - `UPDATE` or `DELETE` action per statement (duckdb-iceberg's limit), and a - `WHEN NOT MATCHED BY SOURCE` action must bound the same tier in its own - condition. A `MERGE` that sets the partition column is refused (run the - change as an `UPDATE`), as is one on a table with a clustered vector column - and one whose source reads the view. A `MERGE` nested in a `WITH` entry - takes the same path as a nested `UPDATE` or `DELETE`. On a decoupled view - every `MERGE` runs in DuckDB. -- `TRUNCATE` on a registered relation, or on the hot table behind a tiered one, - fails with an error, because the cold rows in Iceberg would stay visible - through the view. -- A table or column with a cold tier cannot have a name that PostgreSQL leaves - unquoted but DuckDB parses as a keyword. With DuckDB 1.5.4 those names are - `anti`, `asof`, `at`, `by`, `describe`, `glob`, `lambda`, `pivot`, - `pivot_longer`, `pivot_wider`, `positional`, `qualify`, `semi`, `show`, - `summarize`, `unpack` and `unpivot`. pg_duckdb passes a table or column name - to DuckDB quoted the way PostgreSQL quotes it, so such a name reaches DuckDB - bare even when the query quotes it. Every query pg_duckdb runs against the - table or column then fails with a DuckDB `Parser Error` - ([duckdb/pg_duckdb#1019](https://github.com/duckdb/pg_duckdb/issues/1019)). - ColdFront refuses such a column name wherever a column gets a cold tier: - tiered registration, an archive pass that tiers or expires data, - `coldfront.create_iceberg_table()`, `coldfront.adopt_iceberg_table()`, and an - `ADD COLUMN` or `RENAME COLUMN` on a table that already has a cold tier. - ColdFront refuses such a table name at tiered registration, in - `coldfront.create_iceberg_table()` and `coldfront.adopt_iceberg_table()`, and - in a `RENAME` of the view or the hot table of a table that already has a cold - tier. Partition-only registration, without a hot period, accepts a table with - such a name or column. ColdFront does not refuse a DuckDB keyword that the - DuckDB parser accepts as a name, such as `columns`. - -## Distributed Setup (3-Node Mesh, Decoupled Mode) - -For multi-writer iceberg workloads, run ColdFront on N PG nodes in a Spock mesh -against the same Lakekeeper + S3. The bakery serializes commits PG-side so -writers never collide at the catalog. - -### Per-Node `postgresql.conf` - -The configuration below applies to each node in the mesh: - -```ini -wal_level = logical -shared_preload_libraries = 'snowflake,spock,pg_duckdb,coldfront' - -# The bakery rules a peer dead when its walsender's last reply is older than -# coldfront.peer_alive_window_ms (default 10 s), and an idle peer replies only -# to the walsender's keepalive, every wal_sender_timeout/2. The claim refuses -# to run unless wal_sender_timeout is positive and below twice the window. -wal_sender_timeout = 15s - -# Spock prereq + DDL replication. The ddl_sql repset (and with it the -# one-node provisioning promise for wrapper views) only works with DDL -# replication on; allow_ddl_from_functions covers DDL issued inside -# coldfront's plpgsql helpers. -track_commit_timestamp = on -spock.enable_ddl_replication = on -spock.allow_ddl_from_functions = on -spock.include_ddl_repset = on - -# Only on a server that has this setting (PostgreSQL 16.15, 17.11 and 18.6 in -# pgEdge's builds): -# it lists the libraries allowed as logical decoding output plugins, and its -# default leaves out Spock's, so no subscription can create its slot. A server -# without the setting refuses to start with this line in place. -output_plugin_libraries = 'pgoutput, test_decoding, spock_output' - -# Per-node - any distinct integer 1..1023 (must be unique per node; the value -# is otherwise arbitrary - the bakery matches acks by spock node name, not by id). -snowflake.node = 1 - -# DSN of the node's loopback, which runs the bakery's autonomous claim -# INSERT/DELETE. It must name a unix socket: every onboarded role can read -# this setting, so it must never need a password. The claim session only -# touches the coldfront.claims/claim_acks heap tables, so it never attaches -# the Iceberg catalog (the lazy catalog-attach hook fires only on a tiered -# view). application_name=coldfront_loopback marks the session as bakery -# traffic. Only a superuser can set it. -coldfront.loopback_dsn = 'host=/tmp dbname=coldfront user=coldfront application_name=coldfront_loopback' - -coldfront.warehouse = 'wh' -coldfront.lakekeeper_endpoint = 'http://lakekeeper:8181/catalog' - -# Upload the Parquet outside the bakery claim and serialize only the catalog -# commit. Set both, and only where duckdb-iceberg includes ColdFront's patch (the -# ColdFront image does). With either off, the upload is serialized too. -coldfront.iceberg_async_parquet = on -coldfront.iceberg_bakery_patch = on -``` - -The bakery has no peer-ack timeout. R-A's only failure mode is a dead peer -(would wait forever), and a liveness check inside the wait loop closes it: -the bakery treats a peer whose `pg_stat_replication.reply_time` is older than -`coldfront.peer_alive_window_ms` as already acked, whatever the walsender's -state (a peer that has just reconnected is catching up and still alive). The -peer's apply worker sends that reply after it applies, every -`spock.feedback_frequency` messages, and in answer to the walsender's -keepalive; `wal_receiver_status_interval` plays no part, since Spock's apply -worker does not read it. The window and `wal_sender_timeout` go together; set -them as [Timeouts](#timeouts) below describes. An alive peer that has -not acked is either deferring legitimately (R-A's defer rule) or about to -ack, and waiting for it is correct, not a failure. - -A claim whose owner is gone (a hard backend crash) is reaped without operator -action: by that node's next cold write, to any table, by a peer's arriving -claim, or by the waiting peer's poke, a no-op `UPDATE` of its own claim row -about once a second for as long as it waits. That poke is the only replication -traffic the bakery generates while a writer waits, and there is none when -nothing waits. See [architecture_decoupled.md](architecture_decoupled.md), -*Orphan reaping*. - -Sync-rep (`synchronous_standby_names`) is **not required** by the bakery - the -R-A ack barrier is what serializes iceberg commits. You can still enable -sync-rep cluster-wide if you want stronger durability for non-bakery writes, -but it plays no part in iceberg-commit serialization. - -Run this one-time mesh setup in this order on every node, because step 3 calls -`spock.repset_add_table`, which requires the local Spock node to already exist: - -```sql --- 1. Extensions, in dependency order. snowflake is a bakery prereq --- (claim tickets come from snowflake.nextval). -CREATE EXTENSION IF NOT EXISTS snowflake; -CREATE EXTENSION IF NOT EXISTS spock; -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; - --- 2. Spock node + full-mesh subscriptions (each node has N-1 subs). Name each --- subscription sub__from_ (on n1: sub_n1_from_n2): the --- bakery's peer-liveness check looks the walsender up by that name, and a peer --- whose subscription is named any other way counts as dead and is treated as --- already acked. -SELECT spock.node_create('n', 'host= user=coldfront dbname=coldfront port=5432'); - --- on n1 (n2 and n3 symmetric): -SELECT spock.sub_create('sub_n1_from_n2', 'host= user=coldfront dbname=coldfront port=5432', - ARRAY['default','default_insert_only','ddl_sql'], - false, false, '{}', '0', false); -SELECT spock.sub_create('sub_n1_from_n3', 'host= user=coldfront dbname=coldfront port=5432', - ARRAY['default','default_insert_only','ddl_sql'], - false, false, '{}', '0', false); - --- 3. **Required** on every node, after its subscriptions exist and before --- the first cold write, storage secret or table registration: put every --- ColdFront table that replicates by value in this node's replication sets. -SELECT coldfront.ensure_replicated(); -``` - -The subscriptions set -`synchronize_structure := false, synchronize_data := false` because the tables -already exist on every node from the coldfront extension, so no initial copy is -needed. - -### Timeouts - -The following table lists the two settings that decide how long a cold write -waits for a peer; set them together, on every node: - -| Setting | Recommended | Where | What it does | -|---|---|---|---| -| `wal_sender_timeout` | `15s` | `postgresql.conf` on every node; the image sets it in mesh mode | The node's walsender asks each peer for a reply after half of it (7.5 s) and drops a peer that has not replied for the whole of it. | -| `coldfront.peer_alive_window_ms` | `10000` (the default) | `postgresql.conf`, superuser-only | The bakery treats a peer whose last reply is older than this as dead and stops waiting for its ack. | - -The rule: half of `wal_sender_timeout` must stay below the window, with a -round trip to spare; the claim enforces the first part. An idle peer replies -only when the walsender asks, so the -PostgreSQL default of 60 s against a 10 s window would rule a live idle peer -dead; a claim refuses to run with such settings and names both. With the -recommended values a writer waits at most 10 s for a peer that is really -dead, and the walsender disconnects a peer silent for 15 s, which then -reconnects on its own. - -On a slow or lossy WAN raise both together and keep the rule, for example -`wal_sender_timeout = 30s` with `coldfront.peer_alive_window_ms = 20000`. -Both take effect on a reload (`ALTER SYSTEM SET ...;` then -`SELECT pg_reload_conf();`), no restart: walsenders re-read the timeout, and -each claim reads the window once. - -### What `coldfront.ensure_replicated()` Does - -The call is the one ColdFront-specific step of the mesh setup. It adds eight -tables to the node's replication sets (`claim_acks` to `default_insert_only`, -the rest to `default`), each keyed by name, so a row is identical on every node -and replicates by value: - -| Table | What replicates, and why a peer needs it | -|---|---| -| `coldfront.claims`, `coldfront.claim_acks` | The bakery's tickets and acknowledgements. A peer acknowledges an originator's claim by inserting into `claim_acks` on its own node, and the ack reaches the originator only if the table is in the peer's set. `claim_acks` goes in the insert-only set, because each node deletes only the ack copies it holds. | -| `coldfront.tiered_views` | The registry. A peer's hook recognizes a view by its row; without it the peer's writes through the view fail in PostgreSQL ("cannot insert into view") and its reads cannot attach the cold tier. | -| `coldfront.archive_watermark` | The hot/cold cutoff that routes each tiered write. | -| `coldfront.storage_secret` | The cold-store credential, so one `set_storage_secret` call reaches every node. | -| `coldfront.partition_config` | The per-table lifecycle. The archiver and the partitioner add this table themselves as well, because the partitioner runs without the extension. | -| `coldfront.vector_config`, `coldfront.vector_centroids` | The cluster routing state, so every node resolves a vector to the same cluster. | - -Two tables stay local by design and are not added: `coldfront.deferred_acks` -(each node's queue of the acks it owes) and `coldfront._dummy_dml_target`. - -The call runs on every node because a replication set is a property of the -provider: each node sends the rows of the tables in its own set, and a node -that skipped the step keeps its acks, registry rows and secret to itself. It -cannot run at `CREATE EXTENSION` time, because `spock.repset_add_table` needs -the local Spock node, and nothing runs it later on a node's behalf: -provisioning a table, setting the secret and writing cold rows all assume it -has run. It is idempotent, so running it again is harmless, and a node added to -the mesh later runs the same three steps. - -A skipped step shows up late and away from the node that skipped it. If a peer -never ran it, the originator of every cold write waits at the ack barrier for -an ack that never arrives. If the node that registers a table never ran it, -the view reaches each peer as replicated DDL but the registry row does not, so -the peer's writes through the view fail with "cannot insert into view". A -secret set on such a node stays on that node. - -Verify the step on every node. The query lists the eight tables: - -```sql -SELECT c.relname - FROM spock.replication_set rs - JOIN spock.replication_set_table rst ON rst.set_id = rs.set_id - JOIN pg_class c ON c.oid = rst.set_reloid - WHERE rs.set_name = 'default' AND c.relnamespace = 'coldfront'::regnamespace - ORDER BY 1; -``` - -Then verify the mesh before benching - insert a sentinel claim on each node -and read it back from every other node. All N×(N-1) directions must show the -row before traffic starts. `ci/journey.sh` `story_mesh_substrate` is a -copyable reference: it checks the membership above on every node and then the -sentinels. - -## Tuning Knobs - -The following GUCs adjust write behavior and execution; tune them as needed: - -- `coldfront.allow_mixed_writes` (bool, default `on`) controls what happens for - tiered-mode `UPDATE`/`DELETE` whose WHERE cannot be proven to target one - tier. `on` emits a dual-tier CTE; `off` rejects with an error and a hint. The - setting is not relevant in decoupled mode (every write is single-tier by - definition). -- `coldfront.cold_write_batch_size` (int, default `10000`, minimum `1`) sets - how many cold rows a tiered `INSERT` gathers (see [Caveats](#caveats)) before - it writes them to Iceberg as one `INSERT`. A larger value writes fewer, - larger Parquet files, and the remainder always flushes, so a small write - stays one file. -- `coldfront.vector_probe` (bool, default `on`) sets whether a recognized - similarity search reads only the clusters nearest its query vector. `off` - gives an exact scan of the whole corpus. The setting affects only a table - with a trained vector column ([usage_vectors.md](usage_vectors.md)). -- `coldfront.vector_nprobe` (int, default `0`) sets how many clusters such a - search reads, overriding the column's own `nprobe`. `0` uses the configured - value; at or above the column's `nlist` the search is exhaustive. -- Benchmark `duckdb.force_execution` (default off) before you turn it on: on a - mixed workload it helps `count(distinct)` and similar but regresses index - lookups, top-K with PK ordering, and JSON access. -- `duckdb.temporary_directory` sets where DuckDB spills. Each backend gets its - own subdirectory there, named after its process id, so concurrent spills - cannot collide; one left by a departed backend is reclaimed. See - [architecture.md](architecture.md#duckdb-spill-files-are-not-namespaced-per-instance). -- `duckdb.max_temp_directory_size` is a cap **per connection**, not a cluster - total, and when unset the cap is 90% of free space per session. For a total - budget, divide it by the concurrent sessions or give the temp path its own - filesystem or quota. - -## Next Steps - -To go further with ColdFront, consult the following guides: - -- The [Architecture](architecture.md) overview describes the tiered - architecture, the watermark, the archiver, transparent `UPDATE`/`DELETE`, and - concurrency. -- The [Decoupled Mode](architecture_decoupled.md) deep dive describes - decoupled-mode internals, the ACID model, and distributed scaling. -- The [Embeddings](usage_vectors.md) guide covers storing and searching - embeddings through the pgvector interface. -- The [Compaction](compaction.md) guide covers cold-tier maintenance: - compaction, snapshot expiry, and orphan-file removal. diff --git a/docs/usage_decoupled.md b/docs/usage_decoupled.md new file mode 100644 index 00000000..ff96809d --- /dev/null +++ b/docs/usage_decoupled.md @@ -0,0 +1,205 @@ +# Using ColdFront in Decoupled Mode + +Before you begin, configure PostgreSQL, Lakekeeper, and your object store by +following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](one_time_setup.md), +and [Configuring your Object Store](object_store.md); every row of a +decoupled table lives in the cold tier those guides set up, from the first +row. This page walks through creating a decoupled table with +`coldfront.create_iceberg_table()`, changing its columns, and adopting a +table that another engine already wrote into the catalog. + +A single call creates a decoupled table: + +```sql +SELECT coldfront.create_iceberg_table( + p_schema => 'public', + p_table => 'events', + p_columns => '[ + {"name":"id", "type":"bigint"}, + {"name":"ts", "type":"timestamptz"}, + {"name":"status", "type":"text"}, + {"name":"data", "type":"jsonb"} + ]'::jsonb, + p_partition_cols => '{month(ts)}' +); +``` + +`p_partition_cols` partitions the Iceberg table, here by the month of `ts`: +each month's rows land in their own data files, and a query with a time filter +skips the months outside it before reading anything. Each element is one term +as DuckDB's `PARTITIONED BY` takes it: a column name, `year(col)`, `month(col)` +or `day(col)` on a timestamp or date column, `hour(col)` on a timestamp column, +`bucket(N, col)` or `truncate(W, col)`; `'{month(ts), region}'` partitions by +both. The argument is a PostgreSQL array literal, so a term that contains a +comma is double-quoted inside it, the comma being the array's delimiter: +`'{"bucket(16, id)"}'`. A column name that itself needs double quotes has them +backslashed inside such an element: `'{"month(\"Event Time\")"}'`. Leave the +argument out for an unpartitioned table. + +That single statement provisions: + +- `ice.public.events` on the attached Iceberg catalog. +- a PG-side wrapper view `public.events` with proper PG-typed columns. + +- a `coldfront.tiered_views` registry row, so that the coldfront C hook + intercepts every `INSERT`, `UPDATE`, `DELETE`, and `MERGE` on the view and + rewrites each one to a single `duckdb.raw_query(...)` against + `ice.public.events`. + + +In a mesh one node provisions: Spock's `ddl_sql` repset replicates the +`CREATE VIEW`, and the `default` repset replicates the name-keyed registry row, +which enables the write hook on every peer. Each node also needs the +one-time setup call `coldfront.ensure_replicated()` - see +[Distributed Setup](distributed_setup.md). + +## Changing a Decoupled Table's Columns + +An `ALTER TABLE` on the wrapper view adds, drops, renames or widens a column +of the Iceberg table, and the view follows, with its owner and table-level +grants kept; a column-level grant on the view is not kept. Only the view's +owner, or a role holding its privileges, can change its columns, and that role +needs the cold access `grant_app_access` gives, since the Iceberg change runs +as that role. The following statements change the columns of the table created +above: + +```sql +ALTER TABLE public.events ADD COLUMN qty integer; +ALTER TABLE public.events ALTER COLUMN qty TYPE bigint; +ALTER TABLE public.events RENAME COLUMN qty TO quantity; +ALTER TABLE public.events DROP COLUMN quantity; +``` + +A column takes the same type map as at creation, though an array column cannot +be added, because duckdb-iceberg does not add a nested column to an Iceberg +table. A type change is limited to the widening Iceberg accepts: `integer` to +`bigint`, `real` to `double precision`, `date` to `timestamp`, and a wider +`numeric` precision. An added column takes a name and a type only: a default, +a constraint, a collation, or a storage or compression option is refused, as +is a `USING` or `COLLATE` clause on a type change, and a column change shares +an `ALTER TABLE` only with other column changes. A vector column is neither +added nor changed, a table adopted read-only refuses every column change, and +an object built on the view, such as another view, makes a column change +fail. In a mesh, the change runs on one node: the Iceberg table is shared, and +the rebuilt view and registry row replicate. + +## Adopting a Table That Already Exists in the Catalog + +A table another engine wrote needs no provisioning, only a wrapper view and a +registry row. `coldfront.adopt_iceberg_table()` reads the schema from the +catalog and builds both, so the call names the relation and nothing else: + +```sql +SELECT coldfront.adopt_iceberg_table( + p_schema => 'public', + p_table => 'orders', + p_namespace => 'lake' +); +``` + +`p_schema` is the PostgreSQL schema the wrapper view goes in, and it must +exist. `p_namespace` is the Iceberg namespace the table lives in, which need +not exist as a PostgreSQL schema; it defaults to `p_schema` when omitted. The +view takes the table's name. + +Adoption is read-only by default, so reading someone else's lake table cannot +become writing it by accident. Passing `p_writable => true` enables the same +`INSERT`, `UPDATE`, `DELETE` and `MERGE` rewrite a created table gets: + +```sql +SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', + p_writable => true); +``` + +Because Iceberg records no PostgreSQL type, an adopted column reads as whatever +its storage type maps to; a `jsonb` column comes back as `text`. Pass `p_types` +to restore one, provided the override maps to the type the catalog already +stores: + +```sql +SELECT coldfront.adopt_iceberg_table('public', 'orders', 'lake', + p_writable => true, + p_types => '{"meta":"jsonb"}'::jsonb); +``` + +Adoption binds the name once. A second call under a registered name is refused +whatever its arguments; to enable writes or change an override, release the +table with `coldfront.release_iceberg_table()` and adopt it again. A +non-superuser deployment runs `coldfront.grant_app_access()` after adopting, +because the grant reads the registry at call time. + +In a mesh one node adopts. The wrapper view replicates through Spock's +`ddl_sql` repset and the registry row through the `default` repset (see +[Distributed Setup](distributed_setup.md)), +so every peer reads and writes the table under the same name, and a peer's own +adopt is refused as already registered. Adoption takes the table's bakery +claim around that check and the registry write, so two nodes adopting the +same table at the same moment are serialized: the second waits, then reads +the first's row and refuses, rather than both registering the one table. A +writer outside ColdFront is outside the bakery and can still collide at +Lakekeeper. + +One relation can be registered per Iceberg table. A namespace that needs +quoting, such as `Lake-EU`, works. + +## Dropping an Iceberg Table (Both Modes) + +`coldfront.drop_iceberg_table()` removes the Iceberg table backing a registered +relation. The function is the only sanctioned way to do so: a plain +`DROP TABLE` or `DROP VIEW` on a registered relation stays blocked, because it +would leave the cold tier behind with nothing pointing at it. + +The call takes the schema, the table, and an explicit purge decision: + +```sql +-- drop the catalog entry and delete the stored objects +SELECT coldfront.drop_iceberg_table('public', 'events', true); + +-- drop the catalog entry and leave the objects in the object store +SELECT coldfront.drop_iceberg_table('public', 'events', false); +``` + +What remains afterwards depends on the mode, and the call reports which path it +took: + +- in decoupled mode the Iceberg table is the whole relation, so nothing remains + in PostgreSQL. +- in [tiered mode](usage_tiered.md) the Iceberg table is the cold tier, so the + cold tier is removed and the hot table returns under the relation's own + name, as an ordinary partitioned table holding the data that had not yet + aged out. + +In tiered mode the call also deletes the table's `partition_config` row, which +stops the archiver from tiering it again on the next run. + +There is no default for the purge argument, because the two outcomes are +irreversible in opposite directions. Passing `true` deletes the Parquet and +metadata objects, which for the cold tier are the only copy of that data. +Passing `false` keeps those objects but removes the catalog entry that +ColdFront would need to reach them again, so nothing reclaims them afterwards; +use it when another system is taking ownership of the files. + +ColdFront has no guard of its own against dropping the wrong table. The +preventive control lives in Lakekeeper: table protection has to be enabled +manually, per table, and a protected table cannot be dropped at all, whether or +not purge is requested. This function cannot override a hold. + +Deletion is not instantaneous. The catalog entry disappears with the call, and +Lakekeeper's own background queue removes the objects shortly afterwards, so a +listing taken immediately after the call can still show them. + +### Handing an Adopted Table Back + +An adopted table is released rather than dropped, because ColdFront does not +own it. `coldfront.release_iceberg_table()` removes the wrapper view, the +registry row and the relation's vector configuration, and performs no Iceberg +I/O, so the table keeps every row and stays in the catalog: + +```sql +SELECT coldfront.release_iceberg_table('public', 'orders'); +``` + +Release refuses a tiered registration, because removing one would leave its +cold rows unreachable while the hot table returned under the relation's name. +`drop_iceberg_table()` refuses a relation adopted read-only, for the mirrored +reason: read access gives no authority to destroy. diff --git a/docs/usage_partitioner.md b/docs/usage_partitioner.md new file mode 100644 index 00000000..60d01de1 --- /dev/null +++ b/docs/usage_partitioner.md @@ -0,0 +1,312 @@ +# Using ColdFront in Standalone Partitioned Mode + +This mode needs only a working PostgreSQL instance, optionally in a Spock +mesh; it uses no Lakekeeper, no object store, no pg_duckdb, and no +coldfront extension. A database set up this way can gain the cold tier +later by following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](one_time_setup.md), +and [Configuring your Object Store](object_store.md). This page walks +through building and registering the `partitioner` binary, operating it +day to day, giving a time-partitioned table a real primary key, and +two-level sub-partitioning. + +**You do not need Iceberg at all.** If automated PostgreSQL partition +maintenance is all you want - declarative time- or id-based RANGE partitioning +with a premade forward window and automatic age-out of old partitions - +ColdFront's `partitioner` binary is the whole product: stock PostgreSQL (or a +Spock mesh), no cold tier, no DuckDB, no Iceberg, nothing to preload. Each +invocation makes one reconcile pass per managed table: premake the forward +window, ensure the partition covering *now* exists, and detach-then-drop +partitions past retention (`DETACH ... CONCURRENTLY`, never a bare `DROP` of +attached data). Build the partitioner with `make build` and run it from cron: + +```bash +go build -o bin/partitioner ./cmd/partitioner # or: make build +./bin/partitioner +``` + +A database set up this way can gain the cold tier later: +`CREATE EXTENSION coldfront` adopts the `coldfront.partition_config` the +partitioner created, registrations included. + +The partitioner connects from the libpq environment or `--dsn`, like psql, and +reads its tables from `coldfront.partition_config`; it needs no file. A +partition-only deployment has no cold tier: a table with a `hot_period` +belongs to the archiver, and the partitioner leaves it alone. + +Register each managed table in `coldfront.partition_config`: + +```bash +./bin/partitioner register --table events \ + --period monthly --retention "12 months" \ + --strategy drop # drop (DETACH+DROP, destroy; default) + # | detach (DETACH only, keep as a + # standalone table - data preserved) +``` + +## Operating the Partitioner + +Schedule one pass per period or more often - a cron line, or a systemd +`oneshot` service plus timer (systemd then reports a failed pass as a failed +unit, so alerting is free): + +```text +17 * * * * postgres PGDATABASE=mydb /usr/local/bin/partitioner >> /var/log/coldfront-partitioner.log 2>&1 +``` + +Keep the following operational behavior in mind when scheduling it: + +- The exit code is `0` when every table reconciled (a self-healed *behind* + condition still exits `0`) and non-zero when at least one table failed + (`N table(s) failed`), each failure logged with a `[table]` prefix (the bare + table name) followed by the schema-qualified error. Alert on a non-zero exit + code. +- If a table already has a *past* partition but none covers *now* at the start + of a pass (a lagging cron - live inserts had no home), the pass heals the + table (creates the current partition), logs a `WARNING (self-healed)` line, + and still exits `0`. Monitor for that warning in the log rather than via the + exit code, and widen `future_partitions` or run more often. A fresh table + (only just-premade future partitions) is **not** behind: its first reconcile + succeeds cleanly. +- With the default `expiration_strategy: drop`, expiry is + `DETACH CONCURRENTLY` + `DROP TABLE` - the data is **gone**, so back up + before shrinking `retention_period`. Set `expiration_strategy: detach` to + instead leave the expired partition as a standalone table (detached from the + parent, data preserved) and reclaim it yourself. A partition is expired only + once its *entire* range is older than `now − retention_period`, computed + with calendar-accurate PostgreSQL interval arithmetic (a real month, leap + years correct). `detach` is partition-only - the tiered archiver always drops + after exporting to cold. + +## Primary Keys on Time-Partitioned Tables (id Mode) + +PostgreSQL requires a unique index to include the partition key, so a plain +`PRIMARY KEY (id)` is impossible on a table partitioned by a separate `ts` +column. Partition instead by RANGE on a **time-ordered id** and the partition +key *is* the key - a single-column `PRIMARY KEY (id)`: + +```yaml + - source_table: events + partition_column: id + partition_period: monthly + retention_period: 12 months + part_mode: id + id_scheme: snowflake # snowflake | uuidv7 +``` + +`uuidv7` reads the RFC 9562 leading-millisecond timestamp; `snowflake` decodes +the pgEdge Snowflake extension's layout. Either way the manager computes the +month/day partition bounds as id values, so id-order equals time-order and the +forward/retention schedule is unchanged. + +## Two-Level (LIST → RANGE) Sub-Partitioning + +For a table partitioned by `LIST (region)` whose children are themselves +`RANGE`-partitioned by time, add a `sub_partition` block. `values_source` is a +query returning the current level-1 values; the manager provisions and +maintains a RANGE sub-tree under each, creating sub-trees for newly appearing +values automatically: + +```yaml + - source_table: events + partition_column: ts # the level-2 RANGE column + partition_period: monthly + retention_period: 12 months + sub_partition: + values_source: "SELECT code FROM regions" +``` + +Each level-1 child is named `
_`, with the value lowercased and +every character outside `a-z`, `0-9` and `_` replaced by `_`. That name must +fit in 50 bytes, which leaves room for the daily leaf suffix within +PostgreSQL's 63-byte identifier limit. A pass that meets a longer name fails, +for tiered two-level tables as well. The partitioner and the archiver also +fail a pass in which two values map to the same name. + +## Managing Partitioned Tables (CLI) + +ColdFront's configuration lives in the server. **Per-table lifecycle** lives in +`coldfront.partition_config`, a name-keyed table that replicates by value +across a Spock mesh (like `tiered_views`/`archive_watermark`), so every node +reads identical config - no per-node file syncing. The cold-store credential +lives in `coldfront.storage_secret` and the catalog in the +`coldfront.warehouse` and `coldfront.lakekeeper_endpoint` settings. The +**connection** alone comes from outside, from the libpq environment or +`--dsn`, as for psql. Manage the lifecycle table with the CLI below (both +`partitioner` and `archiver` expose these subcommands; with no subcommand they +do their normal reconcile/archive run). + +`register` is the primary way to manage tables: one command adds or adopts one +table, validated on the spot. `import` and `export` are bulk helpers for +(re)configuring a machine - write a deployment YAML into a fresh server, or +dump the live config to git and replay it on another node - not the day-to-day +path. A YAML passed to anything but `import` is checked against the server and +refused if it disagrees in any value. +`register` and `import` run the full +validation. `set` checks every value of the row that it would leave; it re-runs +the rest of the validation only when it changes the partition period or column, +`hot_period`, `retention_period` or the sub-partition source. + +The data lifecycle is **hot PG → `hot_period` → cold Iceberg → +`retention_period` → dropped** (tiered) or **hot PG → `retention_period` +→ dropped** (partition-only). Setting `hot_period` makes a table tiered; +omitting it makes it partition-only. + +### What Registration Refuses + +`register` and `import` run the same validation, so a table that one rejects +cannot be added by the other; `set` re-runs it when it changes a validated +field. Registration fails when: + +- a setting has a value that the table's `CHECK` constraints refuse. The + refusal names the flag, or on `import` the key, and the values that the + setting accepts; `set` checks the row that its change would leave. + Registration refuses: + + - a `--period` other than `monthly` or `daily`. + - a `--premake` below 1. + - a `--part-mode` other than `timestamp` or `id`. + - an `--id-scheme` other than `uuidv7` or `snowflake`, or one without + `--part-mode id`. + - `--part-mode id` without an `--id-scheme`. + - a `--strategy` other than `drop` or `detach`. + - `--part-mode id` or `--strategy detach` with a `--hot-period`. + - a `--sub-values-source` without a `--column`. + - a table with neither a `--hot-period` nor a `--retention`. + +- any relation in the partition tree is `UNLOGGED`, because that data is + truncated after a crash and replicates nowhere, so archiving it would + preserve rows PostgreSQL never promised to keep. +- the table has a `DEFAULT` partition. Rows the `DEFAULT` partition catches + have no time bounds, so they can never be tiered or expired, and PostgreSQL + refuses `DETACH PARTITION ... CONCURRENTLY` for every partition of a table + that has one, which is how partitions are expired. The partition's mere + existence is enough; it does not have to hold any rows. Move any rows the + `DEFAULT` partition holds into real partitions and detach it. +- the name differs only by case from an already-registered table. PostgreSQL + keeps `public."Events"` and `public.events` apart, but DuckDB matches + identifiers case-insensitively even when they are quoted, so both names would + resolve to one Iceberg table and overwrite each other. +- the name starts with an underscore, which is reserved for the tiered hot + table. +- the name leaves no room for the generated partition suffix: 53 bytes for + monthly and 50 for daily, within PostgreSQL's 63-byte identifier limit. +- a tiered table has a column whose type has no Iceberg equivalent (see + [Supported Column Types](supported_types.md)). Every column + goes through the same type map the cold tier itself uses, so the answer + comes back at the prompt rather than hours later from cron, and the + refusal names the column. Partition-only tables are exempt: nothing about + them reaches Iceberg, so their column types are PostgreSQL's business + alone. +- a tiered table's name, or the name of one of its columns, is one DuckDB + parses as a keyword, such as `by`, `at` or `show` (see [Caveats](caveats.md)). + The archiver checks the column names again before it tiers or expires data; + it refuses a column added after registration before any table or data reaches + Iceberg. + +Registration validates the table as it is at that moment, not continuously. +Adding a `DEFAULT` partition to an already-registered table is therefore not +caught then; the next archiver run fails while reading that table's partition +bounds. + +Other unusual bounds are supported rather than refused. Partitions open at +either end (`MINVALUE`, `MAXVALUE`, `infinity`) are handled, as is PostgreSQL's +full timestamp range, from 4713 BC through year 294276. + +`hot_period` and `retention_period` are native PostgreSQL `interval`s - use any +interval syntax (`1 month`, `90 days`, `1 year 2 mons`, `5 years`). Expiry +boundaries are computed with calendar-accurate interval arithmetic +(`now() - period`: real months, leap years), and `retention_period` must exceed +`hot_period` (validated at `register`/`set` time). + +The following table describes the CLI subcommands: + +| Command | Purpose | +|---|---| +| `register` | Adds or adopts a table, and validates that the PRIMARY KEY covers the partition key. | +| `list` | Shows the managed tables and their lifecycle. | +| `set` | Changes fields, or disables or enables a table with `--disable`/`--enable`. | +| `remove` | Stops managing a table; the table itself is left intact. | +| `import` | Writes a deployment YAML into the server once: its `archiver.tables` as rows, validated as `register` validates each table, and its `s3:` or `azure:` stanza as the storage secret. | +| `export` | Dumps the active (enabled) config to YAML or SQL, as a git-reviewable backup to replay on another node. | + +The following examples register, inspect, and change managed tables: + +```bash +# Partition-only: keep 3 future partitions, drop those older than 12 months. +partitioner register --table events --period monthly --retention "12 months" + +# Partition-only, but DETACH (preserve) expired partitions instead of dropping them. +partitioner register --table events --period monthly \ + --retention "12 months" --strategy detach + +# Tiered: tier to cold Iceberg after 1 month, then drop cold data after 5 years. +archiver register --table events --period monthly \ + --hot-period "1 month" --retention "5 years" + +# id mode - a real single-column PRIMARY KEY (id) on a snowflake-keyed table. +partitioner register --table events --period monthly \ + --column id --part-mode id --id-scheme snowflake --retention "1 year" + +# 2-level LIST(region) → RANGE(ts), tiered; region values come from a table. +archiver register --table regional --period monthly --column ts \ + --hot-period "1 month" --sub-values-source "SELECT region FROM regions" + +partitioner list # what's managed +partitioner set --table events --retention "24 months" +partitioner set --table events --disable # pause (keeps the row) +partitioner remove --table events # unregister, keep the table +partitioner import --config deploy.yaml # write a deployment YAML into the server once +partitioner export > managed.yaml # active config, git-reviewable (--format sql for INSERTs) +``` + +Run `partitioner` (or `archiver`) with `help` or `--help` for the command +overview, and with no arguments for its normal run; every subcommand has a +detailed `--help` with worked examples. The write commands accept +`--print-sql` (emit the SQL without running it - review/commit it; `import` +prints its table `INSERT`s); `register` and `import` also accept `--dry-run`. +`set --enable`/`--disable` (mutually exclusive) pause/resume a table without +removing it; a disabled table is skipped by reconcile and omitted from +`export`. Per table, only the cadence and a lifecycle boundary (`hot_period` or +`retention_period`) are required; `partition_column` is auto-detected from +`pg_catalog` for flat tables (required for 2-level). `register` checks the +lifecycle rules before it writes the row, and the table's `CHECK` constraints +enforce them again at write time. + +The subcommands also take the following connection and table flags: + +- `--dsn` takes a PostgreSQL connection string on every subcommand. Without + it the connection comes from the libpq environment, as for psql. +- `--config` names a deployment YAML. On `import` it is required, since it + names the file to write into the server. On any other subcommand the file is + checked against the server, value by value, and refused if it disagrees; its + `postgres.dsn` connects when `--dsn` is unset. +- `--schema` (default `public`) names the table's schema on `register`, `set` + and `remove`. +- `--premake` (default `3`) sets on `register` how many future partitions are + kept ahead of now, stored as `future_partitions`. + +`set` changes only the fields whose flags are passed: `--period`, `--column`, +`--premake`, `--hot-period`, `--retention`, `--sub-values-source` and +`--strategy`, each taking the same value as on `register`. An empty value +clears the field, so `--hot-period ""` makes a tiered table partition-only and +hands it to the partitioner. A change that leaves neither a `hot_period` nor a +`retention_period` is refused. + +For managing tables, the archiver and partitioner read the managed set only +from `coldfront.partition_config`. `register` adds one table; `import` adds +every table in a YAML `archiver.tables` list. Both write through the same +validation (the PK must cover the partition key; `retention` must exceed +`hot_period`), so an imported table is checked exactly as a registered one. +`export` dumps the active rows back to YAML or SQL for git. + +The archiver manages only the rows that have a `hot_period`, and the +partitioner only the rows without one, so each table belongs to one binary. A +run that finds none of its own rows exits non-zero with +`no tables in coldfront.partition_config`, naming the `register` and `import` +commands that add one. + +In an `archiver.tables` entry, `source_schema` names the table's schema and +defaults to `public`. `source_table` also accepts the `schema.table` form, and +an explicit `source_schema` takes precedence over that prefix. +`future_partitions` defaults to `3`, as `--premake` does on `register`. + diff --git a/docs/usage_tiered.md b/docs/usage_tiered.md new file mode 100644 index 00000000..9758557d --- /dev/null +++ b/docs/usage_tiered.md @@ -0,0 +1,134 @@ +# Using ColdFront in Tiered Mode + +Before you begin, configure PostgreSQL, Lakekeeper, and your object store by +following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](one_time_setup.md), +and [Configuring your Object Store](object_store.md); tiered mode archives +aging partitions to the cold tier those guides set up. This page walks +through creating a tiered table, registering it with the archiver, running +the archiver, and the lifecycle that follows, then details two restrictions +tiered tables carry: inbound foreign keys and 2-level tiered tables. + +Create a partitioned table normally: + +```sql +CREATE TABLE events ( + id bigint GENERATED ALWAYS AS IDENTITY, + ts timestamptz NOT NULL, + status text, + data jsonb, + PRIMARY KEY (id, ts) +) PARTITION BY RANGE (ts); + +CREATE TABLE p_2026_04 PARTITION OF events + FOR VALUES FROM ('2026-04-01') TO ('2026-05-01'); +``` + +Then tell the archiver about the table, either with a deployment YAML: + +```yaml +s3: + endpoint: "seaweedfs:8333" + access_key: "admin" + secret_key: "adminsecret" +archiver: + tables: + - source_table: events + partition_period: monthly + hot_period: "1 month" +``` + +```bash +./bin/archiver import --config config.yaml +``` + +or by registering the table directly: + +```bash +./bin/archiver register --table events \ + --period monthly --hot-period "1 month" +# optional: --retention "5 years" DROPs cold data past that age +# (must exceed --hot-period; omit = keep forever) +``` + +See [Writing ColdFront's Configuration](configuration.md#writing-coldfronts-configuration) +for how `register` and `import` write this, and what the server checks, and +[Managing Partitioned Tables (CLI)](usage_partitioner.md#managing-partitioned-tables-cli) +for `list`, `set`, `remove`, and `export`, which the archiver shares with the +partitioner. + +Run the archiver (typically via cron): + +```bash +./bin/archiver +``` + +`--version` prints the version that `make build` stamps from `git describe` +(the release tag, or the commit it was built from) and exits; a plain +`go build` prints `unknown`. The partitioner and compactor accept the same +flag. + +The archiver also accepts `--debug-export-delay`, a test-only Go duration +(such as `5s`) that holds each partition's capture window open between the +bulk export and the replay, so that a test can race concurrent writes into it. +Leave it unset in production. + +The first run that finds a partition older than `hot_period` renames `events` +→ `_events`, creates the unified view `events`, and registers it; until then +the archiver only premakes partitions, and the table stays a plain partitioned +table. From then on every cycle (1) tiers partitions older than `hot_period` +from hot PG to cold Iceberg and advances the watermark, and (2) if +`retention_period` is set, drops cold Iceberg rows older than it. The data +lifecycle is **hot → `hot_period` → cold → `retention_period` → gone**; +omit `retention_period` to keep cold data forever. + +To remove a tiered table's cold tier entirely rather than waiting out its +retention period, see [Dropping an Iceberg Table](usage_decoupled.md#dropping-an-iceberg-table-both-modes). + +## Inbound Foreign Keys + +A tiered table cannot be the target of an enforced foreign key from another +table over its archivable range. Archiving physically removes the rows from +PostgreSQL (export to Iceberg, then `DETACH` + `DROP` the partition), and +PostgreSQL cannot enforce a foreign key against the cold tier. So the `DETACH` +is refused (SQLSTATE 23503) and the archiver fails fast, naming the blocking +constraint. + +This is inherent, not an edge case: referencing rows rarely have a hot-only +lifecycle. They usually outlive the partition they point at, so the foreign key +still pins those rows in PostgreSQL by the time the partition ages into the +cold tier. + +The foreign key is fundamentally incompatible with archiving the rows it +references, so before archiving such a table, drop it: + +```sql +ALTER TABLE event_logs DROP CONSTRAINT event_logs_events_fkey; +``` + +Better still, do not point an enforced foreign key at a tiered table over its +archivable range in the first place. + +## 2-Level (LIST → RANGE) Tiered Tables + +A table partitioned `LIST (region) → RANGE (ts)` can be tiered too - the same +`sub_partition` block as [Using ColdFront in Standalone Partitioned Mode](usage_partitioner.md), +so a partition-manager-managed table can be "upgraded" to tiered by pointing +the archiver at it: + +```yaml + - source_table: regional + partition_column: ts # the RANGE (time) column - required for 2-level + partition_period: monthly + hot_period: 1 month + # retention_period: 5 years # optional cold expiry (region-agnostic, by ts) + sub_partition: + values_source: "SELECT region FROM regions" +``` + +One Iceberg table holds every region (region is an ordinary column); the +archiver tiers leaves a whole `ts` period at a time across **all** regions +before advancing the shared hot/cold watermark, so a period only becomes cold +once it is cold for every region. `id` mode is not supported in tiered mode +(the cold tier is time-keyed). For why the ordering works that way, see the +[Two-level tiering](architecture_tiered.md#two-level-list-range-tiering) +section of the Tiered Mode page. diff --git a/docs/usage_vectors.md b/docs/usage_vectors.md index df63b51a..eb90d48e 100644 --- a/docs/usage_vectors.md +++ b/docs/usage_vectors.md @@ -34,7 +34,7 @@ You do not need pgvector in a database that has none. A table that already exists in the catalog can be adopted instead, through `coldfront.adopt_iceberg_table()` (see -[Adopting a Table That Already Exists in the Catalog](usage.md#adopting-a-table-that-already-exists-in-the-catalog)). +[Adopting a Table That Already Exists in the Catalog](usage_decoupled.md#adopting-a-table-that-already-exists-in-the-catalog)). Adoption registers a clustered vector column for each Iceberg column named `_cf_vec_list_`, which is where ColdFront keeps each row's cluster. No `coldfront.vector_config` row and no centroids come back with an adopted @@ -432,5 +432,5 @@ To go further with ColdFront, consult the following documents: state, cluster assignment, and probe rewrite behind this guide. - The [Compaction](compaction.md) guide covers the maintenance that keeps clustered tables in cluster order. -- The [Using ColdFront](usage.md) guide covers the table setup both modes +- The [Using ColdFront](using_coldfront/index.md) guide covers the table setup both modes share. diff --git a/docs/using_coldfront/index.md b/docs/using_coldfront/index.md new file mode 100644 index 00000000..352cb71e --- /dev/null +++ b/docs/using_coldfront/index.md @@ -0,0 +1,32 @@ +# Using ColdFront + +ColdFront offers three operating modes. Pick one per table; tables in +different modes can coexist in the same database. The following table +compares the two modes that use the cold Iceberg tier: + +| | Tiered (hot + cold) | Decoupled (iceberg-only) | +|---|---|---| +| Where rows live | Recent rows are hot in the PG heap, and archived rows are cold in Iceberg. | Every row is in Iceberg. | +| Setup | Create a partitioned table and let the archiver convert it on the first run that finds a partition older than `hot_period`. | One SQL call, `coldfront.create_iceberg_table(...)`, sets up the table. | +| Archiver | The archiver is required; it runs from cron and moves old partitions to cold. | The archiver is not used. | +| Best when | The workload has a recent-row OLTP part that benefits from PG indexes and transactional ergonomics. | The workload is purely analytic or append-mostly, and you want zero PG storage and stateless compute. | + +Once a tiered or decoupled table exists, **the SQL surface is identical**: +`SELECT`, `INSERT`, `UPDATE`, `DELETE` all work normally against the relation +name (e.g. `events`), except that a write touching the cold tier rejects +`RETURNING`. + +Follow the guide for the mode you want: + +- [Using ColdFront in Tiered Mode](../usage_tiered.md) archives aging + partitions to Iceberg while recent rows stay in the PostgreSQL heap. +- [Using ColdFront in Decoupled Mode](../usage_decoupled.md) keeps every row + in Iceberg from the start. +- [Using ColdFront in Standalone Partitioned Mode](../usage_partitioner.md) + manages plain PostgreSQL partitions with no cold tier at all. + +Operational topics shared across modes - caveats, vended credentials, +reading and writing, supported types, and distributed (mesh) setup - are +documented under Administration, starting with +[Caveats](../caveats.md). + diff --git a/docs/vended_credentials.md b/docs/vended_credentials.md new file mode 100644 index 00000000..97919482 --- /dev/null +++ b/docs/vended_credentials.md @@ -0,0 +1,51 @@ +# Using Vended Credentials + +Vended credentials let a deployment run with no object-store credential stored +in the database, in a DuckDB secret file, or in an archiver config; this suits +compliance environments that forbid persisting long-term keys. Lakekeeper +issues a short-lived, per-table credential (an S3 STS access key, secret, and +session token) at read and write time, and ColdFront uses it directly. The +long-term credential lives only in the Lakekeeper warehouse. + +Enable vended credentials with a single call that stores no credential: + +```sql +SELECT coldfront.set_storage_secret_vended(); +``` + +This writes a `coldfront.storage_secret` row marked as vended, so nothing is +materialized as a DuckDB secret; `ensure_attached()` then attaches the catalog +with credential vending turned on. The archiver and the compactor read the +same row, so a vended deployment has no `s3:` or `azure:` stanza to import and +no credential in any file. + +Vended mode targets the two clouds that issue scoped credentials: AWS S3 (STS) +and Azure ADLS Gen2 (SAS). The `set_storage_secret_vended()` call above enables +AWS S3; the same call with `'azure'` enables Azure through the identical path: + +```sql +SELECT coldfront.set_storage_secret_vended('azure'); +``` + +Vending requires a Lakekeeper warehouse configured to vend credentials: + +- On AWS S3, the warehouse uses `flavor: aws` with `sts-enabled: true`, an + `assume-role-arn` for a bucket-scoped IAM role, and an `external-id` on the + warehouse credential that the role's trust policy also requires. The full + warehouse and IAM-role setup is in + [object_store.md](object_store.md#configuring-lakekeeper). +- On Azure ADLS Gen2, the warehouse is an `adls` warehouse with `sas-enabled` + (on by default). Lakekeeper vends a per-container SAS token; the warehouse + credential can be a `shared-access-key`, `client-credentials`, or + `azure-system-identity`. + +Google Cloud Storage over the S3-interoperability endpoint has no STS to issue +short-lived credentials, so GCS stays on static HMAC credentials. + +The compactor runs fully under vended credentials: compaction, snapshot expiry, +and orphan-file reclaim all use the vended per-table credentials. + +Switching a running deployment between static and vended credentials changes +the attach mode, which is fixed per PostgreSQL backend at attach time; open a +new session (or restart the archiver and compactor, which are short-lived +processes) so the new mode takes effect. diff --git a/docs/walkthrough.md b/docs/walkthrough.md index 2b4cba88..55370c03 100644 --- a/docs/walkthrough.md +++ b/docs/walkthrough.md @@ -165,10 +165,8 @@ store: | GCS (HMAC) | `SELECT coldfront.set_storage_secret(p_key_id => '', p_secret => '', p_endpoint => 'storage.googleapis.com', p_region => 'us-east-1', p_url_style => 'path', p_use_ssl => true);` | | Azure ADLS Gen2 | `SELECT coldfront.set_storage_secret_azure('AccountName=;AccountKey=;EndpointSuffix=core.windows.net');` | -The [Configuring your Object Store](object_store.md) guide shows the matching -warehouse JSON for AWS S3; for GCS and Azure, see the -[Storage Backends](usage.md#storage-backends) section of the Using ColdFront -guide. +The [Configuring your Object Store](object_store.md) guide shows the +matching Lakekeeper warehouse JSON for each of these backends. ## Tearing Down the Stack diff --git a/docs/walkthrough_decoupled.md b/docs/walkthrough_decoupled.md index 6cec6a5e..b10e0338 100644 --- a/docs/walkthrough_decoupled.md +++ b/docs/walkthrough_decoupled.md @@ -178,10 +178,10 @@ To go further with ColdFront, consult the following guides: - The [Partitioner Demo](walkthrough_partitioner.md) manages PostgreSQL range partitions without any cold tier. -- The [Using ColdFront](usage.md) guide covers both modes in depth, including +- The [Using ColdFront](using_coldfront/index.md) guide covers both modes in depth, including the full one-time setup, supported column types, the partition manager CLI, and tuning options. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. - The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_distributed.md b/docs/walkthrough_distributed.md index 75c0521e..8b9bb570 100644 --- a/docs/walkthrough_distributed.md +++ b/docs/walkthrough_distributed.md @@ -223,10 +223,10 @@ number and no fork. To go further with ColdFront, consult the following guides: -- The [Using ColdFront](usage.md) guide covers both modes in depth, including +- The [Using ColdFront](using_coldfront/index.md) guide covers both modes in depth, including the full one-time setup, supported column types, the partition manager CLI, and tuning options. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. - The [Compaction](compaction.md) guide covers cold-tier maintenance: compaction, snapshot expiry, and orphan-file removal. diff --git a/docs/walkthrough_partitioner.md b/docs/walkthrough_partitioner.md index 992a74e7..1390ef8d 100644 --- a/docs/walkthrough_partitioner.md +++ b/docs/walkthrough_partitioner.md @@ -66,7 +66,7 @@ WHERE inhparent = 'part_demo'::regclass; A table registered with `--strategy detach` keeps its expired partitions as standalone tables instead of dropping them. The -[Managing Partitioned Tables (CLI)](usage.md#managing-partitioned-tables-cli) +[Managing Partitioned Tables (CLI)](usage_partitioner.md#managing-partitioned-tables-cli) section describes the partition CLI and shows more `register` examples. ## Next Steps @@ -75,7 +75,7 @@ To go further with ColdFront, consult the following guides: - The [Distributed Demo](walkthrough_distributed.md) points two PostgreSQL nodes at one shared lake. -- The [Using ColdFront](usage.md) guide covers both modes in depth, including +- The [Using ColdFront](using_coldfront/index.md) guide covers both modes in depth, including the full one-time setup, supported column types, the partition manager CLI, and tuning options. - The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section diff --git a/docs/walkthrough_tiered.md b/docs/walkthrough_tiered.md index 62bc626b..9c50eaaf 100644 --- a/docs/walkthrough_tiered.md +++ b/docs/walkthrough_tiered.md @@ -595,9 +595,9 @@ To go further with ColdFront, consult the following guides: - The [Decoupled Mode Demo](walkthrough_decoupled.md) stores a table in Iceberg from the first row and adopts a table that another engine wrote. -- The [Configuring your Object Store](object_store.md) guide takes you from an - empty bucket to a working cold tier on AWS S3. -- The [Architecture](architecture.md) overview explains the shared mechanics +- The [Configuring your Object Store](object_store.md) + guide takes you from an empty bucket to a working cold tier on AWS S3. +- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. - The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/mkdocs.yml b/mkdocs.yml index 162a029a..be529344 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ theme: - navigation.tracking - navigation.prune - navigation.top + - navigation.indexes - toc.follow palette: @@ -57,27 +58,45 @@ repo_url: https://github.com/pgEdge/ColdFront nav: - pgEdge ColdFront: index.md - - Installing and Configuring ColdFront: installation.md - - Getting Started: - - Getting Started with ColdFront: walkthrough.md + + - Architecture Guides: + - architecture_guides/index.md + - Tiered Operating Mode: architecture_tiered.md + - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md + - Vector Storage Architecture: architecture_vectors.md + + - Getting Started/Quick Start: walkthrough.md + - Installing ColdFront: installation.md + - Configuring ColdFront: + - configuration.md + - Configuring Lakekeeper: one_time_setup.md + - Configuring your Object Store: object_store.md + + - Exploring ColdFront: - Exploring Tiered Storage: walkthrough_tiered.md - Exploring Decoupled Mode: walkthrough_decoupled.md - Exploring the Standalone Partitioner: walkthrough_partitioner.md - Exploring Distributed Mode: walkthrough_distributed.md - - Configuring your Object Store: object_store.md - - Architecture: - - pgEdge ColdFront - Architecture: architecture.md - - ColdFront - Tiered Operating Mode: architecture_tiered.md - - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md - - Vector Storage Architecture: architecture_vectors.md - - - Using ColdFront: usage.md - - Working with Embeddings: usage_vectors.md - - COMPACTOR - Cold-Tier Table Maintenance: compaction.md + - Using ColdFront: + - using_coldfront/index.md + - Using ColdFront in Tiered Mode: usage_tiered.md + - Using ColdFront in Decoupled Mode: usage_decoupled.md + - Using ColdFront in Standalone Partitioned Mode: usage_partitioner.md + - Working with Embeddings: usage_vectors.md + + - Administration: + - Understanding ColdFront Caveats: caveats.md + - Using Vended Credentials: vended_credentials.md + - Reading and Writing ColdFront Tables: reading_writing.md + - Understanding Supported Column Types: supported_types.md + - Setting Up a Distributed Deployment: distributed_setup.md + - Compacting the Cold Tier: compaction.md - Developer Resources: - Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal): formal/README.md - - Changelog: changelog.md + - Release Notes - Changelog: changelog.md - License: LICENSE.md + + - Source: source.md From bb257b379af6b70ac04eb0f61226b9eea1c98da3 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 13 Oct 2026 10:39:11 -0400 Subject: [PATCH 02/10] docs: add Connecting to ColdFront page, polish Exploring ColdFront and index.md Moves the Least-Privilege Application Roles content into a new Connecting to ColdFront guide, renames and restructures the Getting Started walkthrough into Exploring ColdFront - Getting Started, folds the Caveats pointer into a new Resources/Exploring ColdFront with Walkthroughs section on the landing page, and polishes wording across installation.md, configuration.md, and one_time_setup.md. --- README.md | 99 ++------------ docs/caveats.md | 5 + docs/configuration.md | 6 +- docs/connecting.md | 45 +++++++ .../index.md} | 88 +++++++----- docs/index.md | 125 +++--------------- docs/installation.md | 41 +++--- docs/object_store.md | 2 +- docs/one_time_setup.md | 7 +- docs/source.md | 2 +- docs/walkthrough_decoupled.md | 4 +- docs/walkthrough_distributed.md | 4 +- docs/walkthrough_partitioner.md | 4 +- docs/walkthrough_tiered.md | 6 +- mkdocs.yml | 4 +- 15 files changed, 178 insertions(+), 264 deletions(-) create mode 100644 docs/connecting.md rename docs/{walkthrough.md => exploring_coldfront/index.md} (68%) diff --git a/README.md b/README.md index 810369e3..78d09347 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The ColdFront documentation consists of the following guides: - [Introduction](docs/index.md) - Getting Started - - [Getting Started with ColdFront](docs/walkthrough.md) + - [Exploring ColdFront](docs/exploring_coldfront/index.md) - [Exploring Tiered Storage](docs/walkthrough_tiered.md) - [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) - [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) @@ -103,7 +103,7 @@ tier, so the application sees one relation: ## Installation -If you are new to ColdFront, run the [guided walkthrough](docs/walkthrough.md) +If you are new to ColdFront, run the [guided walkthrough](docs/exploring_coldfront/index.md) to see all three modes in action with copy-pasteable commands. ColdFront is open source under the PostgreSQL License and runs on PostgreSQL @@ -112,7 +112,8 @@ build with Spock, which the Docker image is based on, for distributed mode. The full build workflow lives in the **[Installation guide](docs/installation.md)**: build the thin ColdFront layer on top of the published DuckDB 1.5.x base image (or build the base yourself), -or install bare-metal. Then continue with the Quickstart below. +or install bare-metal. Then continue with the +[walkthrough](docs/exploring_coldfront/index.md). ### Setting Up on Cloud S3? @@ -138,55 +139,13 @@ For every setting, see the [Configuring PostgreSQL](docs/configuration.md#configuring-postgresql) section of the Configuring ColdFront guide. -## Quickstart - -Build the image (see the [Installation](docs/installation.md) guide) and bring -up the stack: - -```bash -docker compose --profile local-store up -d --build -``` - -Bootstrap Lakekeeper and create a warehouse (see the one-time setup in the -[Using ColdFront](docs/using_coldfront/index.md) guide), then create a table in psql: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); - --- Decoupled (iceberg-only) table, stored entirely in Iceberg on S3: -SELECT coldfront.create_iceberg_table('public', 'events', - '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, - '{month(ts)}'); -INSERT INTO events VALUES (1, now(), 'hello'); -SELECT count(*) FROM events; -``` - -ColdFront can adopt a table that already exists in the Iceberg catalog rather -than creating it: `coldfront.adopt_iceberg_table()` reads its schema from the -catalog and gives it the same wrapper view and registry row, read-only unless -you pass `p_writable => true`. `coldfront.release_iceberg_table()` hands the -table back with the Iceberg table untouched. See -[Adopting a Table That Already Exists in the Catalog](docs/usage_decoupled.md#adopting-a-table-that-already-exists-in-the-catalog). - -To remove a table again, `coldfront.drop_iceberg_table()` unregisters it and -drops the Iceberg table, deleting the stored objects only when asked to. See -[Dropping an Iceberg Table](docs/usage_decoupled.md#dropping-an-iceberg-table-both-modes). - -For compliance environments that cannot store an object-store credential, -`coldfront.set_storage_secret_vended()` runs with no credential in the -database: Lakekeeper issues short-lived per-table credentials at access time. -See [Vended Credentials](docs/vended_credentials.md). - ## Using ColdFront -The Quickstart walks through a decoupled table from start to finish. [Using ColdFront in Tiered Mode](docs/usage_tiered.md), [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md), and [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) document each mode in depth; the [Using ColdFront](docs/using_coldfront/index.md) guide -helps you pick one, and the [walkthrough](docs/walkthrough.md) demos run +helps you pick one, and the [walkthrough](docs/exploring_coldfront/index.md) demos run each mode on a sample table. Operational topics shared across modes - caveats, vended credentials, reading and writing, supported types, and distributed (mesh) setup - are documented under Administration. @@ -198,7 +157,7 @@ documents: | Doc | Contents | |---|---| -| [Walkthrough](docs/walkthrough.md) | Sets up the demo stack and runs ColdFront hands-on. | +| [Walkthrough](docs/exploring_coldfront/index.md) | Sets up the demo stack and runs ColdFront hands-on. | | [Tiered storage demo](docs/walkthrough_tiered.md) | Adds ColdFront to an existing database and moves its cold data to object storage. | | [Decoupled mode demo](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | | [Partitioner demo](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | @@ -208,7 +167,7 @@ documents: | [Tiered mode usage](docs/usage_tiered.md) | Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. | | [Decoupled mode usage](docs/usage_decoupled.md) | Documents tables that live entirely in Iceberg from the first row, and dropping or releasing a registered table. | | [Standalone partitioner usage](docs/usage_partitioner.md) | Documents managing PostgreSQL partitions with no cold tier at all, and the partition CLI shared with tiered mode. | -| [Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other sharp edges shared between tiered and decoupled tables. | +| [Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other edge cases shared between tiered and decoupled tables. | | [Vended Credentials](docs/vended_credentials.md) | Documents running with no stored object-store credential, for AWS S3 and Azure ADLS Gen2. | | [Reading and Writing](docs/reading_writing.md) | Documents the SQL surface shared between tiered and decoupled tables. | | [Supported Column Types](docs/supported_types.md) | Documents the PostgreSQL column types, including arrays, that a cold tier supports. | @@ -217,53 +176,23 @@ documents: | [Configuring ColdFront](docs/configuration.md) | Documents configuring PostgreSQL and ColdFront's own configuration. | | [Configuring Lakekeeper](docs/one_time_setup.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | | [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running end to end, for any of four supported backends: S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. | +| [Connecting to ColdFront](docs/connecting.md) | Documents onboarding an application role with least privilege. | | [Compaction](docs/compaction.md) | Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | | [Architecture](docs/architecture_guides/index.md) | Describes the shared architecture and core mechanics. | | [Architecture: tiered](docs/architecture_tiered.md) | Describes tiered mode (hot PG plus cold Iceberg) in depth. | | [Architecture: decoupled](docs/architecture_decoupled.md) | Describes decoupled (iceberg-only) mode in depth. | | [Architecture: vectors](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | -## Least-Privilege Application Roles - -Application roles need no superuser and no server-file access, yet they read -and write the cold tier through the same transparent view. Onboarding an -application role is a single call: - -```sql -SELECT coldfront.grant_app_access('alice'); -``` +## Connecting to ColdFront -grant_app_access grants only the minimum the cold path needs: membership in -duckdb.postgres_role, SET on the duckdb.unsafe_allow_execution_inside_functions -parameter, schema USAGE, `SELECT` on the registry and the watermark table, DML -on the dual-write anchor table, DML on every registered view and the hot table -behind it, and USAGE and `SELECT` on the hot table's sequences. Those objects -are derived from the registry, not hardcoded, and the call also grants EXECUTE -on a fixed allow-list of runtime cold-path functions. The call is idempotent -and is not executable by PUBLIC, so an application role can never self-grant. -The role is never granted pg_read_server_files or pg_write_server_files, so it -has no host-file access. `CREATE ROLE` and `GRANT` both replicate over Spock, -so you onboard a role once on any node and it propagates across the mesh. - -The Docker image sets `duckdb.postgres_role` to `coldfront_duckdb` and creates -that role when it initializes a new data directory. To name a different role, -set the container's `COLDFRONT_DUCKDB_ROLE` environment variable before that -first start. An empty value keeps pg_duckdb's default, under which only -superusers run DuckDB, so grant_app_access then fails with an error. - -For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, -the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the -`duckdb.postgres_role` default that the image sets up, and how least privilege -holds across a Spock mesh - see -[Architecture: non-superuser app roles](docs/architecture_guides/index.md#non-superuser-app-roles-least-privilege). +See [Connecting to ColdFront](docs/connecting.md) for onboarding an +application role with least privilege. ## Caveats -Iceberg on Azure ADLS Gen2 requires Blob soft-delete, container soft-delete, -and change feed (blob events) to be OFF on the storage account. Lakekeeper -warehouse creation otherwise fails with HTTP 409 ("This endpoint does not -support BlobStorageEvents or SoftDelete"). Disable those features on the -storage account before using it as a cold tier. +See [Understanding ColdFront Caveats](docs/caveats.md) for ColdFront +behaviors you should be aware of, including a storage-account setting +Azure ADLS Gen2 requires before you use it as a cold tier. ## Project Structure diff --git a/docs/caveats.md b/docs/caveats.md index 9461725a..5f02dc0f 100644 --- a/docs/caveats.md +++ b/docs/caveats.md @@ -136,3 +136,8 @@ Keep the following caveats in mind when running either mode: tier. Partition-only registration, without a hot period, accepts a table with such a name or column. ColdFront does not refuse a DuckDB keyword that the DuckDB parser accepts as a name, such as `columns`. +- Iceberg on Azure ADLS Gen2 requires Blob soft-delete, container soft-delete, + and change feed (blob events) to be OFF on the storage account. Lakekeeper + warehouse creation otherwise fails with HTTP 409 ("This endpoint does not + support BlobStorageEvents or SoftDelete"). Disable those features on the + storage account before using it as a cold tier. diff --git a/docs/configuration.md b/docs/configuration.md index 0dc22f37..498d66e9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -3,14 +3,14 @@ After installing ColdFront packages or building from source, several components still need configuring before the cold tier comes online: -- PostgreSQL and ColdFront's own configuration, detailed in this +- PostgreSQL and ColdFront's configuration is detailed in this guide, regardless of which installation path you followed. -- your object store, documented in - [Configuring your Object Store](object_store.md). - installing, configuring, and bootstrapping Lakekeeper, and the one-time per-database setup that ties everything together, documented in the [Configuring Lakekeeper](one_time_setup.md) guide. +- your object store, documented in + [Configuring your Object Store](object_store.md). ## Configuring PostgreSQL diff --git a/docs/connecting.md b/docs/connecting.md new file mode 100644 index 00000000..866c5aac --- /dev/null +++ b/docs/connecting.md @@ -0,0 +1,45 @@ +# Connecting to ColdFront + +An application connects to ColdFront as an ordinary PostgreSQL client; the +only ColdFront-specific step is onboarding its role using the principle of +least privilege. + +## Least-Privilege Application Roles + +Application roles need no superuser and no server-file access, yet they read +and write the cold tier through the same transparent view. Onboarding an +application role requires a single call: + +```sql +SELECT coldfront.grant_app_access('alice'); +``` + +grant_app_access grants only the minimum the cold path needs: + +- membership in duckdb.postgres_role. +- SET on the duckdb.unsafe_allow_execution_inside_functions parameter. +- schema USAGE. +- `SELECT` on the registry and the watermark table. +- DML on the dual-write anchor table. +- DML on every registered view and the hot table behind it. +- USAGE and `SELECT` on the hot table's sequences. + +The tables and views above are derived from the registry, not hardcoded; +the call also grants EXECUTE on a fixed allow-list of runtime cold-path +functions. The call is idempotent +and is not executable by PUBLIC, so an application role can never self-grant. +The role is never granted pg_read_server_files or pg_write_server_files, so it +has no host-file access. `CREATE ROLE` and `GRANT` both replicate over Spock, +so you onboard a role once on any node and it propagates across the mesh. + +The Docker image sets `duckdb.postgres_role` to `coldfront_duckdb` and creates +that role when it initializes a new data directory. To name a different role, +set the container's `COLDFRONT_DUCKDB_ROLE` environment variable before that +first start. An empty value keeps pg_duckdb's default, under which only +superusers run DuckDB, so grant_app_access then fails with an error. + +For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, +the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the +`duckdb.postgres_role` default that the image sets up, and how least privilege +holds across a Spock mesh - see +[Architecture: non-superuser app roles](architecture_guides/index.md#non-superuser-app-roles-least-privilege). diff --git a/docs/walkthrough.md b/docs/exploring_coldfront/index.md similarity index 68% rename from docs/walkthrough.md rename to docs/exploring_coldfront/index.md index 55370c03..f7efd4bc 100644 --- a/docs/walkthrough.md +++ b/docs/exploring_coldfront/index.md @@ -1,43 +1,63 @@ --- -cwd: ../ +cwd: ../../ --- -# Getting Started with ColdFront +# Exploring ColdFront - Getting Started -If you are new to ColdFront, run the guided walkthrough. +This walkthrough is the foundation step for the mode-specific guides that +follow: -The walkthrough is a self-contained, step-by-step tour of ColdFront's three -operating modes - tiered storage (hot PostgreSQL + cold Iceberg), decoupled -mode (Iceberg-only from the first row), and the standalone partitioner - plus a -distributed demo that runs two nodes over one shared lake. Every command on -this page is real: click it to run in Codespaces, or copy it into a local -shell. +- [Exploring Tiered Storage](../walkthrough_tiered.md) +- [Exploring Decoupled Mode](../walkthrough_decoupled.md) +- [Exploring the Standalone Partitioner](../walkthrough_partitioner.md) +- [Exploring Distributed Mode](../walkthrough_distributed.md) + +It is a self-contained, step-by-step tour of ColdFront's three operating +modes, using Docker containers deployed with the Compose files in the +ColdFront repository. The walkthrough demonstrates: + +- tiered storage, which keeps recent data in PostgreSQL and archives older + data to cold Iceberg. +- decoupled mode, which stores a table entirely in Iceberg from the first + row. +- the standalone partitioner, which manages PostgreSQL partitions with no + cold tier at all. +- a distributed demo that runs two nodes over one shared lake. > **Beta software** - ColdFront is beta software under active development. Do > not use it in production. Interfaces, on-disk formats, and behavior may > change without notice, and data loss is possible. -## Running in GitHub Codespaces +!!! tip "Running in GitHub Codespaces" + + [Open this repo in a Codespace](https://github.com/codespaces/new?repo=pgEdge/coldfront) + and select `Get Started` to deploy this walkthrough in a Codespace + environment. Docker, psql, and the + [Runme extension](https://marketplace.visualstudio.com/items?itemName=stateful.runme) + are preinstalled, and turn each code block below into a runnable cell. + Click **Run** on each cell as you read. The first cell builds the Docker + images (two to five minutes); each subsequent step is quicker. -[Open this repo in a Codespace](https://github.com/codespaces/new?repo=pgEdge/coldfront) -and everything is preinstalled - Docker, psql, and the -[Runme extension](https://marketplace.visualstudio.com/items?itemName=stateful.runme) -that turns each code block below into a runnable cell. Click **Run** on each -cell as you read. The first cell builds the Docker images (two to five -minutes); everything after it is quick. +If you prefer a terminal, run +[`examples/walkthrough/guide.sh`](https://github.com/pgEdge/ColdFront/blob/main/examples/walkthrough/guide.sh). +It brings up the stack, then presents the same four demos as an +interactive menu - tiered, decoupled, partitioner, and distributed - +plus options to reset the demo tables or tear the stack down. -If you prefer a terminal, `bash examples/walkthrough/guide.sh` runs the same -demos as an interactive guide. +## Walking Through on your own System -## Running on Your Own Machine +Before performing the walkthrough, confirm that your system meets the +following prerequisites: -You need Docker 24+ with Compose V2 (the `docker compose` plugin, not the -legacy `docker-compose` binary), roughly 3 GB of free disk inside Docker's -virtual disk (the images alone are about 2.5 GB), and `curl`, `bash`, and -`psql` on the host. The interactive guide also needs `ss` on Linux or `lsof` on -macOS, which it uses to find free ports. +- Docker 24+ with Compose V2 (the `docker compose` plugin, not the legacy + `docker-compose` binary). +- roughly 3 GB of free disk inside Docker's virtual disk (the images alone + are about 2.5 GB). +- `curl`, `bash`, and `psql` reside on the host. +- `ss` on Linux or `lsof` on macOS, which the interactive guide uses to find + free ports. -The fastest path is the one-liner, which downloads the walkthrough files and -launches the interactive guide: +The following curl command downloads the walkthrough files and launches the +interactive guide: ```bash {"ignore":"true"} curl -fsSL \ @@ -150,7 +170,7 @@ curl -sf -X POST \ The walkthrough uses SeaweedFS by default. To use a cloud store instead, replace the warehouse JSON above, the `set_storage_secret` call in -[Step 5](walkthrough_tiered.md#pointing-coldfront-at-the-object-store) of the +[Step 5](../walkthrough_tiered.md#pointing-coldfront-at-the-object-store) of the tiered demo, and the `s3:` block in `examples/walkthrough/config/archiver.yaml`, which `import` writes into the server as the storage secret. @@ -165,7 +185,7 @@ store: | GCS (HMAC) | `SELECT coldfront.set_storage_secret(p_key_id => '', p_secret => '', p_endpoint => 'storage.googleapis.com', p_region => 'us-east-1', p_url_style => 'path', p_use_ssl => true);` | | Azure ADLS Gen2 | `SELECT coldfront.set_storage_secret_azure('AccountName=;AccountKey=;EndpointSuffix=core.windows.net');` | -The [Configuring your Object Store](object_store.md) guide shows the +The [Configuring your Object Store](../object_store.md) guide shows the matching Lakekeeper warehouse JSON for each of these backends. ## Tearing Down the Stack @@ -182,7 +202,7 @@ The `-v` flag removes the named volumes: `pgdata` (PostgreSQL), `lkdata` (the Lakekeeper catalog) and `s3data` (the object store). Omit the flag to keep the data for a later session. -If you ran the [distributed demo](walkthrough_distributed.md), tear down its +If you ran the [distributed demo](../walkthrough_distributed.md), tear down its separate mesh stack too: ```bash @@ -198,13 +218,13 @@ switches back to the single-node stack. ## Next Steps -With the stack running, continue with the following guides: +With the stack running, continue with the following walkthroughs: -- The [Tiered Storage Demo](walkthrough_tiered.md) adds ColdFront to an +- The [Tiered Storage Demo](../walkthrough_tiered.md) adds ColdFront to an existing database and moves its cold data to object storage. -- The [Decoupled Mode Demo](walkthrough_decoupled.md) stores a table in Iceberg +- The [Decoupled Mode Demo](../walkthrough_decoupled.md) stores a table in Iceberg from the first row and adopts a table that another engine wrote. -- The [Partitioner Demo](walkthrough_partitioner.md) manages PostgreSQL range +- The [Partitioner Demo](../walkthrough_partitioner.md) manages PostgreSQL range partitions without any cold tier. -- The [Distributed Demo](walkthrough_distributed.md) points two PostgreSQL +- The [Distributed Demo](../walkthrough_distributed.md) points two PostgreSQL nodes at one shared lake. diff --git a/docs/index.md b/docs/index.md index a9e6653a..5a440550 100644 --- a/docs/index.md +++ b/docs/index.md @@ -75,95 +75,29 @@ tier, so the application sees one relation: cron job: moves partitions out of the hot window: hot -> cold ``` -## Quickstart +!!! tip "New to ColdFront?" -Build the image (see the [Installation](installation.md) guide) and bring up -the stack: + If you are new to ColdFront, review the guided + [walkthroughs](exploring_coldfront/index.md) to see all three modes in action + with copy-pasteable commands. -```bash -docker compose --profile local-store up -d --build -``` - -Bootstrap Lakekeeper and create a warehouse (see the one-time setup in the -[Using ColdFront](using_coldfront/index.md) guide), then create a table in psql: - -```sql -CREATE EXTENSION IF NOT EXISTS pg_duckdb; -CREATE EXTENSION IF NOT EXISTS coldfront; -SELECT coldfront.set_storage_secret('admin', 'adminsecret', 'seaweedfs:8333'); - --- Decoupled (iceberg-only) table, stored entirely in Iceberg on S3: -SELECT coldfront.create_iceberg_table('public', 'events', - '[{"name":"id","type":"bigint"},{"name":"ts","type":"timestamptz"},{"name":"note","type":"text"}]'::jsonb, - '{month(ts)}'); -INSERT INTO events VALUES (1, now(), 'hello'); -SELECT count(*) FROM events; -``` - -ColdFront can adopt a table that already exists in the Iceberg catalog rather -than creating it: `coldfront.adopt_iceberg_table()` reads its schema from the -catalog and gives it the same wrapper view and registry row, read-only unless -you pass `p_writable => true`. `coldfront.release_iceberg_table()` hands the -table back with the Iceberg table untouched. See -[Adopting a Table That Already Exists in the Catalog](usage_decoupled.md#adopting-a-table-that-already-exists-in-the-catalog). +## Exploring ColdFront with Walkthroughs -To remove a table again, `coldfront.drop_iceberg_table()` unregisters it and -drops the Iceberg table, deleting the stored objects only when asked to. See -[Dropping an Iceberg Table](usage_decoupled.md#dropping-an-iceberg-table-both-modes). +Use the walkthroughs in the [Exploring ColdFront](exploring_coldfront/index.md) +section to learn more about ColdFront. Perform the steps on the +[Exploring ColdFront - Getting Started](exploring_coldfront/index.md) page +first, which brings up the stack; then continue to the detailed guide for +the mode you want to try: -For compliance environments that cannot store an object-store credential, -`coldfront.set_storage_secret_vended()` runs with no credential in the -database: Lakekeeper issues short-lived per-table credentials at access time. -See [Vended Credentials](vended_credentials.md). - -## Least-Privilege Application Roles - -Application roles need no superuser and no server-file access, yet they read -and write the cold tier through the same transparent view. Onboarding an -application role is a single call: - -```sql -SELECT coldfront.grant_app_access('alice'); -``` - -grant_app_access grants only the minimum the cold path needs: membership in -duckdb.postgres_role, SET on the duckdb.unsafe_allow_execution_inside_functions -parameter, schema USAGE, `SELECT` on the registry and the watermark table, DML -on the dual-write anchor table, DML on every registered view and the hot table -behind it, and USAGE and `SELECT` on the hot table's sequences. Those objects -are derived from the registry, not hardcoded, and the call also grants EXECUTE -on a fixed allow-list of runtime cold-path functions. The call is idempotent -and is not executable by PUBLIC, so an application role can never self-grant. -The role is never granted pg_read_server_files or pg_write_server_files, so it -has no host-file access. `CREATE ROLE` and `GRANT` both replicate over Spock, -so you onboard a role once on any node and it propagates across the mesh. - -The Docker image sets `duckdb.postgres_role` to `coldfront_duckdb` and creates -that role when it initializes a new data directory. To name a different role, -set the container's `COLDFRONT_DUCKDB_ROLE` environment variable before that -first start. An empty value keeps pg_duckdb's default, under which only -superusers run DuckDB, so grant_app_access then fails with an error. - -For how the non-superuser path works - the `SECURITY DEFINER` attach helpers, -the `PGC_SUSET` / `GUC_SUPERUSER_ONLY` config hardening, the -`duckdb.postgres_role` default that the image sets up, and how least privilege -holds across a Spock mesh - see -[Architecture: non-superuser app roles](architecture_guides/index.md#non-superuser-app-roles-least-privilege). - -## Caveats - -Iceberg on Azure ADLS Gen2 requires Blob soft-delete, container soft-delete, -and change feed (blob events) to be OFF on the storage account. Lakekeeper -warehouse creation otherwise fails with HTTP 409 ("This endpoint does not -support BlobStorageEvents or SoftDelete"). Disable those features on the -storage account before using it as a cold tier. +- [Exploring Tiered Storage](walkthrough_tiered.md) +- [Exploring Decoupled Mode](walkthrough_decoupled.md) +- [Exploring the Standalone Partitioner](walkthrough_partitioner.md) +- [Exploring Distributed Mode](walkthrough_distributed.md) ## Next Steps -If you are new to ColdFront, run the [guided walkthrough](walkthrough.md) to -see all three modes in action with copy-pasteable commands. - -To go further with ColdFront, consult the following guides: +For details about installing and configuring ColdFront, consult the +following guides: - The [Installation](installation.md) guide details installing ColdFront from a package or building it from source. @@ -172,33 +106,6 @@ To go further with ColdFront, consult the following guides: - The [Configuring Lakekeeper](one_time_setup.md) guide installs, configures, and bootstraps it, then installs the extensions each database needs. -- The [Using ColdFront](using_coldfront/index.md) guide compares the three operating modes - and links to each mode's guide. -- The [Using ColdFront in Tiered Mode](usage_tiered.md) guide documents - archiving aging partitions to Iceberg. -- The [Using ColdFront in Decoupled Mode](usage_decoupled.md) guide - documents tables that live entirely in Iceberg, and dropping or - releasing a registered table. -- The [Using ColdFront in Standalone Partitioned Mode](usage_partitioner.md) - guide documents managing PostgreSQL partitions with no cold tier, and - the partition CLI shared with tiered mode. -- The [Caveats](caveats.md) guide documents cross-tier isolation, mixed - transactions, and other sharp edges shared between tiered and decoupled - tables. -- The [Vended Credentials](vended_credentials.md) guide documents running - with no stored object-store credential. -- The [Reading and Writing](reading_writing.md) guide documents the SQL - surface shared between tiered and decoupled tables. -- The [Supported Column Types](supported_types.md) guide documents the - PostgreSQL column types, including arrays, that a cold tier supports. -- The [Distributed Setup](distributed_setup.md) guide documents running a - multi-writer decoupled-mode Spock mesh against one shared lake. -- The [Embeddings](usage_vectors.md) guide documents storing and searching - embeddings through the pgvector interface. - The [Configuring your Object Store](object_store.md) guide takes you from an empty bucket or container to a working cold tier, for any of four supported backends. -- The [Compaction](compaction.md) guide documents cold-tier maintenance: - compaction, snapshot expiry, and orphan-file removal. -- The [Architecture](architecture_guides/index.md) overview explains the shared mechanics - and links to the per-mode deep dives. diff --git a/docs/installation.md b/docs/installation.md index e4bd8e9a..4e51173c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,9 +1,9 @@ # Installing ColdFront -This guide details installing ColdFront from the published package or -building it from source. See [Configuring ColdFront](configuration.md) -for setting up PostgreSQL, Lakekeeper, and ColdFront itself once -installed. +This guide details installing ColdFront with a pgEdge package or +building it from source. After installing ColdFront, see +[Configuring ColdFront](configuration.md) for detailed information about +setting up PostgreSQL, Lakekeeper, and ColdFront itself. ## Using a Package to Install ColdFront @@ -16,7 +16,7 @@ the steps in the Enterprise Postgres documentation: - For Debian and Ubuntu, see [Configuring the Repository on Debian and Ubuntu](https://docs.pgedge.com/enterprise/debian/configure-repo/). -The documentation details the platform prerequisites, disabling any PGDG +The documentation details the platform requirements, disabling the PGDG repository, and creating the pgEdge repository. With the repository in place, install the ColdFront extension package for @@ -57,14 +57,20 @@ Docker on top of the published base image or on bare metal. ColdFront runs on a **DuckDB 1.5.x** stack: PostgreSQL + pg_duckdb (DuckDB 1.5.4) and a **patched** duckdb-iceberg that includes ColdFront's five -patches - the bakery-aware commit-refresh patch (the no-409 guarantee for -concurrent cold-tier writers), two strict-reader interop patches (so -apache/iceberg-go, the cold-tier compactor, can read the manifests -duckdb-iceberg writes), a port of an upstream fix that computes the time -partitions of a `timestamptz` column in UTC, and a port of another that numbers -a new table schema above the highest schema id. No released pg_duckdb tag -includes DuckDB 1.5.x yet, so the stack is built from a pinned upstream PR plus -ColdFront's patches - all from sources you can fetch. +patches: + +- the bakery-aware commit-refresh patch (the no-409 guarantee for + concurrent cold-tier writers). +- two strict-reader interop patches, so apache/iceberg-go, the cold-tier + compactor, can read the manifests duckdb-iceberg writes. +- a port of an upstream fix that computes the time partitions of a + `timestamptz` column in UTC. +- a port of another that numbers a new table schema above the highest + schema id. + +No released pg_duckdb tag includes DuckDB 1.5.x yet, so the stack is built +from a pinned upstream PR plus ColdFront's patches - all from sources you +can fetch. ### Prerequisites @@ -77,10 +83,11 @@ The following table lists the prerequisites for each build path: | The compactor and the CI gate | golangci-lint for `make compactor`; `./run-ci-local.sh` also needs Docker and mkdocs with mkdocs-material. | | Bare metal (near the end of this section) | `pg_config`, PostgreSQL server dev headers, libpq client headers and library (libpq-dev / libpq-devel), `make`, and `gcc`. | -### What the Build Produces +### Preparing the DuckDB Base Image -`docker/Dockerfile.duckdb15-base` is the recipe; it fetches the requirements, -applies ColdFront's patches, and compiles a set of components. The following +[`docker/Dockerfile.duckdb15-base`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15-base) +contains the recipe for the base image: it fetches the requirements, applies +ColdFront's patches, and compiles a set of components. The following table shows each component and its source: | Component | Source | @@ -360,7 +367,7 @@ To go further with ColdFront, consult the following guides: - The [Configuring Lakekeeper](one_time_setup.md) guide installs, configures, and bootstraps it, then installs the extensions each database needs. -- The [Walkthrough](walkthrough.md) guide runs the demo stack hands-on. +- The [Walkthrough](exploring_coldfront/index.md) guide runs the demo stack hands-on. - The [Using ColdFront](using_coldfront/index.md) guide documents both operating modes. - The [Configuring your Object Store](object_store.md) guide connects the cold tier to AWS S3. diff --git a/docs/object_store.md b/docs/object_store.md index 22f7898d..15061309 100644 --- a/docs/object_store.md +++ b/docs/object_store.md @@ -368,7 +368,7 @@ SELECT coldfront.set_storage_secret_azure( `set_storage_secret_azure()` writes the same `coldfront.storage_secret` row (replicated, `pg_dump`-excluded) and materializes a `TYPE azure` PERSISTENT SECRET. The Azure cold tier is subject to the soft-delete / change-feed -restriction in [Caveats](index.md#caveats). +restriction in [Caveats](caveats.md). A vended-credentials variant is also available: set `sas-enabled` on the warehouse (on by default) and Lakekeeper vends a per-container SAS diff --git a/docs/one_time_setup.md b/docs/one_time_setup.md index 10c88e7b..d498d588 100644 --- a/docs/one_time_setup.md +++ b/docs/one_time_setup.md @@ -1,6 +1,6 @@ # Configuring Lakekeeper -The setup needs three services configured once per deployment: +ColdFront requires three services configured once per deployment: PostgreSQL with the pg_duckdb and coldfront extensions, Lakekeeper, and any object store (SeaweedFS, MinIO, AWS S3, GCS, or Azure ADLS Gen2). This guide brings up and bootstraps Lakekeeper, then installs and @@ -14,8 +14,7 @@ Choose how you are running it below. Bring up the end-user stack first (the example uses SeaweedFS, gated behind the `local-store` compose profile; host ports are published so -the `localhost` commands below work directly). For the image build -itself, see [installation.md](installation.md): +the `localhost` commands below work directly): ```bash docker compose --profile local-store up -d --build @@ -247,7 +246,7 @@ SELECT coldfront.grant_app_access('alice'); runs, so run it again after you create or adopt a table, or tier one for the first time. The call fails while `duckdb.postgres_role` is unset, because only superusers can then run DuckDB. -[Least-Privilege Application Roles](index.md#least-privilege-application-roles) +[Connecting to ColdFront](connecting.md#least-privilege-application-roles) lists what it grants. For a real cloud-S3 setup, see [object_store.md](object_store.md). diff --git a/docs/source.md b/docs/source.md index 512cae5b..5d08b446 100644 --- a/docs/source.md +++ b/docs/source.md @@ -7,7 +7,7 @@ navigation above is reorganized into the standard pgEdge structure. - [pgEdge ColdFront](index.md) - [Installing and Configuring ColdFront](installation.md) - Getting Started - - [Getting Started with ColdFront](walkthrough.md) + - [Getting Started with ColdFront](exploring_coldfront/index.md) - [Exploring Tiered Storage](walkthrough_tiered.md) - [Exploring Decoupled Mode](walkthrough_decoupled.md) - [Exploring the Standalone Partitioner](walkthrough_partitioner.md) diff --git a/docs/walkthrough_decoupled.md b/docs/walkthrough_decoupled.md index b10e0338..e58f67e7 100644 --- a/docs/walkthrough_decoupled.md +++ b/docs/walkthrough_decoupled.md @@ -8,7 +8,7 @@ PostgreSQL holds a thin wrapper view and a registry entry. This is a fresh table - there is no migration from the tiered demo, and the two modes are independent. -Run the steps in [Setting Up the Stack](walkthrough.md#setting-up-the-stack) +Run the steps in [Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) before you start this demo. The demo also needs the extensions and storage secret from [Step 4](walkthrough_tiered.md#enabling-the-extensions) and [Step 5](walkthrough_tiered.md#pointing-coldfront-at-the-object-store) of the @@ -183,5 +183,5 @@ To go further with ColdFront, consult the following guides: and tuning options. - The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_distributed.md b/docs/walkthrough_distributed.md index 8b9bb570..187218fa 100644 --- a/docs/walkthrough_distributed.md +++ b/docs/walkthrough_distributed.md @@ -44,7 +44,7 @@ If port 5442, 5443, 8191, or 8343 is already in use on your host, set or `COLDFRONT_MESH_S3_PORT` before `up`. Next, run the `curl` block from -[Setting Up the Stack](walkthrough.md#setting-up-the-stack) with port 8191 in +[Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) with port 8191 in place of 8181. That creates the `wh` warehouse and `public` namespace on the mesh's own Lakekeeper. @@ -230,5 +230,5 @@ To go further with ColdFront, consult the following guides: and links to the per-mode deep dives. - The [Compaction](compaction.md) guide covers cold-tier maintenance: compaction, snapshot expiry, and orphan-file removal. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_partitioner.md b/docs/walkthrough_partitioner.md index 1390ef8d..a299df0c 100644 --- a/docs/walkthrough_partitioner.md +++ b/docs/walkthrough_partitioner.md @@ -8,7 +8,7 @@ tier. If all you need is automated partition maintenance on stock PostgreSQL, the partitioner is the whole product - no Iceberg, no DuckDB, no archiver cold path. -Run the steps in [Setting Up the Stack](walkthrough.md#setting-up-the-stack) +Run the steps in [Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) before you start this demo. ## Creating the Demo Table @@ -78,5 +78,5 @@ To go further with ColdFront, consult the following guides: - The [Using ColdFront](using_coldfront/index.md) guide covers both modes in depth, including the full one-time setup, supported column types, the partition manager CLI, and tuning options. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/docs/walkthrough_tiered.md b/docs/walkthrough_tiered.md index 9c50eaaf..65d9b09b 100644 --- a/docs/walkthrough_tiered.md +++ b/docs/walkthrough_tiered.md @@ -8,7 +8,7 @@ database full of data, add ColdFront to it, and let the archiver relocate the cold majority to object storage - without migrating to a new database or changing a line of application SQL. -Run the steps in [Setting Up the Stack](walkthrough.md#setting-up-the-stack) +Run the steps in [Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack) before you start this demo. The following table shows the eleven steps this demo covers: @@ -33,7 +33,7 @@ Setup starts the infrastructure: PostgreSQL (your database), plus Lakekeeper and SeaweedFS. The cold-storage side sits idle until you point ColdFront at it in [Step 5](#pointing-coldfront-at-the-object-store). Run the `docker compose up` command shown in -[Setting Up the Stack](walkthrough.md#setting-up-the-stack). Then confirm that +[Setting Up the Stack](exploring_coldfront/index.md#setting-up-the-stack). Then confirm that the `db`, `lakekeeper-db`, and `lakekeeper` services report `healthy`: ```bash @@ -599,5 +599,5 @@ To go further with ColdFront, consult the following guides: guide takes you from an empty bucket to a working cold tier on AWS S3. - The [Architecture](architecture_guides/index.md) overview explains the shared mechanics and links to the per-mode deep dives. -- The [Tearing Down the Stack](walkthrough.md#tearing-down-the-stack) section +- The [Tearing Down the Stack](exploring_coldfront/index.md#tearing-down-the-stack) section describes how to stop the stack and remove its data. diff --git a/mkdocs.yml b/mkdocs.yml index be529344..78700283 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -65,14 +65,16 @@ nav: - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md - Vector Storage Architecture: architecture_vectors.md - - Getting Started/Quick Start: walkthrough.md - Installing ColdFront: installation.md - Configuring ColdFront: - configuration.md - Configuring Lakekeeper: one_time_setup.md - Configuring your Object Store: object_store.md + - Connecting to ColdFront: connecting.md + - Exploring ColdFront: + - exploring_coldfront/index.md - Exploring Tiered Storage: walkthrough_tiered.md - Exploring Decoupled Mode: walkthrough_decoupled.md - Exploring the Standalone Partitioner: walkthrough_partitioner.md From d60757e4935cf3ba6384268e8abcc1f193db0cfb Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 13 Oct 2026 12:00:40 -0400 Subject: [PATCH 03/10] docs: reorganize config pages, tidy types and compaction, sync README - Rename the distributed setup page to Configuring a Distributed Deployment and move it under Configuring ColdFront. - Fold reading_writing.md into other pages and remove it. - Rework supported_types.md and rename it Supported Column Types. - Move the build step into the compaction intro and drop the Usage, Requirements, and Next Steps headings. - Bring the README table of contents and documentation table in line with the nav. --- README.md | 97 ++++++++++++---------- docs/architecture_decoupled.md | 7 +- docs/architecture_tiered.md | 5 +- docs/caveats.md | 7 +- docs/compaction.md | 25 ++---- docs/configuration.md | 3 +- docs/distributed_setup.md | 31 +++---- docs/reading_writing.md | 39 --------- docs/supported_types.md | 145 ++++++++++++++++++--------------- docs/usage_decoupled.md | 6 +- docs/using_coldfront/index.md | 42 +++++++++- mkdocs.yml | 32 ++++---- 12 files changed, 229 insertions(+), 210 deletions(-) delete mode 100644 docs/reading_writing.md diff --git a/README.md b/README.md index 78d09347..5cd510ba 100644 --- a/README.md +++ b/README.md @@ -7,24 +7,35 @@ The ColdFront documentation consists of the following guides: - [Introduction](docs/index.md) -- Getting Started - - [Exploring ColdFront](docs/exploring_coldfront/index.md) +- Architecture Guides + - [Architecture Overview](docs/architecture_guides/index.md) + - [Tiered Operating Mode](docs/architecture_tiered.md) + - [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) + - [Vector Storage Architecture](docs/architecture_vectors.md) +- Installing and Configuring ColdFront + - [Installing ColdFront](docs/installation.md) + - [Configuring ColdFront](docs/configuration.md) + - [Configuring Lakekeeper](docs/one_time_setup.md) + - [Configuring your Object Store](docs/object_store.md) + - [Configuring a Distributed Deployment](docs/distributed_setup.md) +- [Using ColdFront](docs/using_coldfront/index.md) + - [Using ColdFront in Tiered Mode](docs/usage_tiered.md) + - [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) + - [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) + - [Working with Embeddings](docs/usage_vectors.md) +- [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) - [Exploring Tiered Storage](docs/walkthrough_tiered.md) - [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) - [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) - [Exploring Distributed Mode](docs/walkthrough_distributed.md) - - [Building ColdFront from Source](docs/installation.md) - - [Setting Up an Object Store](docs/object_store.md) -- Architecture - - [Architecture Overview](docs/architecture_guides/index.md) - - [Tiered Mode](docs/architecture_tiered.md) - - [Decoupled Mode](docs/architecture_decoupled.md) - - [Vector Storage](docs/architecture_vectors.md) -- [Using ColdFront](docs/using_coldfront/index.md) -- [Storing and Searching Embeddings](docs/usage_vectors.md) -- [Compacting the Cold Tier](docs/compaction.md) +- [Connecting to ColdFront](docs/connecting.md) +- Administration + - [Understanding ColdFront Caveats](docs/caveats.md) + - [Using Vended Credentials](docs/vended_credentials.md) + - [Supported Column Types](docs/supported_types.md) + - [Compacting the Cold Tier](docs/compaction.md) - Developer Resources - - [Verifying the Bakery Protocol](docs/formal/README.md) + - [Formal Model - ColdFront Decoupled-Mode Bakery](docs/formal/README.md) - [Release Notes](docs/changelog.md) ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet @@ -117,8 +128,9 @@ or install bare-metal. Then continue with the ### Setting Up on Cloud S3? -Once the image is built, the **[S3 setup guide](docs/object_store.md)** takes -you from an empty bucket to a working cold tier end-to-end. +When the image is built, the +**[Configuring your Object Store](docs/object_store.md)** guide takes you from +an empty bucket to a working cold tier end-to-end. ## Configuration @@ -144,11 +156,15 @@ section of the Configuring ColdFront guide. [Using ColdFront in Tiered Mode](docs/usage_tiered.md), [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md), and [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) -document each mode in depth; the [Using ColdFront](docs/using_coldfront/index.md) guide -helps you pick one, and the [walkthrough](docs/exploring_coldfront/index.md) demos run -each mode on a sample table. Operational topics shared across modes - -caveats, vended credentials, reading and writing, supported types, and -distributed (mesh) setup - are documented under Administration. +document each mode in depth; the +[Using ColdFront](docs/using_coldfront/index.md) guide helps you pick one, and +the [walkthrough](docs/exploring_coldfront/index.md) demos run each mode on a +sample table. Operational topics shared across modes - +[caveats](docs/caveats.md), [vended credentials](docs/vended_credentials.md), +[supported column types](docs/supported_types.md), and +[compaction](docs/compaction.md) - are documented under Administration; to run +ColdFront on more than one node, see +[Configuring a Distributed Deployment](docs/distributed_setup.md). ## Documentation @@ -157,31 +173,30 @@ documents: | Doc | Contents | |---|---| -| [Walkthrough](docs/exploring_coldfront/index.md) | Sets up the demo stack and runs ColdFront hands-on. | -| [Tiered storage demo](docs/walkthrough_tiered.md) | Adds ColdFront to an existing database and moves its cold data to object storage. | -| [Decoupled mode demo](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | -| [Partitioner demo](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | -| [Distributed demo](docs/walkthrough_distributed.md) | Points two PostgreSQL nodes at one shared lake. | -| [Embeddings](docs/usage_vectors.md) | Documents storing and searching embeddings with the pgvector interface. | -| [Usage](docs/using_coldfront/index.md) | Compares the three operating modes and links to each mode's guide. | -| [Tiered mode usage](docs/usage_tiered.md) | Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. | -| [Decoupled mode usage](docs/usage_decoupled.md) | Documents tables that live entirely in Iceberg from the first row, and dropping or releasing a registered table. | -| [Standalone partitioner usage](docs/usage_partitioner.md) | Documents managing PostgreSQL partitions with no cold tier at all, and the partition CLI shared with tiered mode. | -| [Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other edge cases shared between tiered and decoupled tables. | -| [Vended Credentials](docs/vended_credentials.md) | Documents running with no stored object-store credential, for AWS S3 and Azure ADLS Gen2. | -| [Reading and Writing](docs/reading_writing.md) | Documents the SQL surface shared between tiered and decoupled tables. | -| [Supported Column Types](docs/supported_types.md) | Documents the PostgreSQL column types, including arrays, that a cold tier supports. | -| [Distributed Setup](docs/distributed_setup.md) | Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. | -| [Installation](docs/installation.md) | Documents installing from a package or building from source (Docker or bare-metal), and testing and CI. | +| [Architecture Overview](docs/architecture_guides/index.md) | Describes the shared architecture and core mechanics. | +| [Tiered Operating Mode](docs/architecture_tiered.md) | Describes tiered mode (hot PostgreSQL plus cold Iceberg) in depth. | +| [Decoupled (Iceberg-Only) Operating Mode](docs/architecture_decoupled.md) | Describes decoupled (Iceberg-only) mode in depth. | +| [Vector Storage Architecture](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | +| [Installing ColdFront](docs/installation.md) | Documents installing from a package or building from source (Docker or bare-metal), and testing and CI. | | [Configuring ColdFront](docs/configuration.md) | Documents configuring PostgreSQL and ColdFront's own configuration. | | [Configuring Lakekeeper](docs/one_time_setup.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | | [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running end to end, for any of four supported backends: S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. | +| [Configuring a Distributed Deployment](docs/distributed_setup.md) | Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. | +| [Using ColdFront](docs/using_coldfront/index.md) | Compares the three operating modes, documents the shared SQL surface, and links to each mode's guide. | +| [Using ColdFront in Tiered Mode](docs/usage_tiered.md) | Documents archiving aging partitions to Iceberg while recent rows stay in the PostgreSQL heap. | +| [Using ColdFront in Decoupled Mode](docs/usage_decoupled.md) | Documents tables that live entirely in Iceberg from the first row, and dropping or releasing a registered table. | +| [Using ColdFront in Standalone Partitioned Mode](docs/usage_partitioner.md) | Documents managing PostgreSQL partitions with no cold tier at all, and the partition CLI shared with tiered mode. | +| [Working with Embeddings](docs/usage_vectors.md) | Documents storing and searching embeddings with the pgvector interface. | +| [Exploring ColdFront Modes](docs/exploring_coldfront/index.md) | Sets up the demo stack and runs ColdFront hands-on. | +| [Exploring Tiered Storage](docs/walkthrough_tiered.md) | Adds ColdFront to an existing database and moves its cold data to object storage. | +| [Exploring Decoupled Mode](docs/walkthrough_decoupled.md) | Stores a table in Iceberg from the first row and adopts a table that another engine wrote. | +| [Exploring the Standalone Partitioner](docs/walkthrough_partitioner.md) | Manages PostgreSQL range partitions without any cold tier. | +| [Exploring Distributed Mode](docs/walkthrough_distributed.md) | Points two PostgreSQL nodes at one shared lake. | | [Connecting to ColdFront](docs/connecting.md) | Documents onboarding an application role with least privilege. | -| [Compaction](docs/compaction.md) | Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | -| [Architecture](docs/architecture_guides/index.md) | Describes the shared architecture and core mechanics. | -| [Architecture: tiered](docs/architecture_tiered.md) | Describes tiered mode (hot PG plus cold Iceberg) in depth. | -| [Architecture: decoupled](docs/architecture_decoupled.md) | Describes decoupled (iceberg-only) mode in depth. | -| [Architecture: vectors](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | +| [Understanding ColdFront Caveats](docs/caveats.md) | Documents cross-tier isolation, mixed transactions, and other edge cases shared between tiered and decoupled tables. | +| [Using Vended Credentials](docs/vended_credentials.md) | Documents running with no stored object-store credential, for AWS S3 and Azure ADLS Gen2. | +| [Supported Column Types](docs/supported_types.md) | Documents the PostgreSQL column types, including arrays, that a cold tier supports. | +| [Compacting the Cold Tier](docs/compaction.md) | Documents cold-tier table maintenance: compaction, snapshot expiry, and orphan-file removal. | ## Connecting to ColdFront diff --git a/docs/architecture_decoupled.md b/docs/architecture_decoupled.md index c225b608..7b4f6598 100644 --- a/docs/architecture_decoupled.md +++ b/docs/architecture_decoupled.md @@ -162,7 +162,8 @@ which would lose precision or identity: - custom enums, `xml`, `tsvector`/`tsquery`, range types, and multirange types. - composite types. - arrays of `timestamptz`, `jsonb`, `json`, `interval` or a vector type, and - arrays of arrays; [Arrays](supported_types.md#arrays) gives the reasons. + arrays of arrays; [Supported Array Types](supported_types.md#supported-array-types) + gives the reasons. - pgvector's `sparsevec`. - any type not enumerated above. @@ -724,7 +725,7 @@ pointing at the same Lakekeeper endpoint and S3 bucket: - For DDL, Spock's `ddl_sql` repset replicates `CREATE/ALTER/DROP` of the wrapper view, and the `default` repset replicates the `coldfront.tiered_views` registry row (see - [Distributed Setup](distributed_setup.md)), so one node provisions the + [Configuring a Distributed Deployment](distributed_setup.md)), so one node provisions the table and replication enables every peer's hook. A column `ALTER TABLE` reaches the peers the same way: the node that runs it changes the shared Iceberg table, and its rebuilt view and registry row @@ -792,7 +793,7 @@ The call is idempotent. It puts every ColdFront table that replicates by value in the node's replication sets: the two bakery tables, the registry and the watermark, the storage secret, the lifecycle config and the vector routing state (the list and the reason for each table are in the -[Distributed Setup](distributed_setup.md#what-coldfrontensure_replicated-does) guide). +[Configuring a Distributed Deployment](distributed_setup.md#what-coldfrontensure_replicated-does) guide). If it has not run on a peer, that peer's ack `INSERT`s are local-only and never replicate back to the originating writer: every claim on the originator waits at the ack barrier for an ack that never arrives. diff --git a/docs/architecture_tiered.md b/docs/architecture_tiered.md index a20c6b54..3539bd95 100644 --- a/docs/architecture_tiered.md +++ b/docs/architecture_tiered.md @@ -584,8 +584,9 @@ So alongside the bakery substrate (`coldfront.claims` / `coldfront.claim_acks`), **both `coldfront.tiered_views` and `coldfront.archive_watermark` are in the Spock replication set**: `coldfront.ensure_replicated()`, the one-time per-node mesh step described in -[distributed_setup.md](distributed_setup.md#what-coldfrontensure_replicated-does), adds them with the -rest. The archiver runs on one node and registers a table, its view and its +[Configuring a Distributed Deployment](distributed_setup.md#what-coldfrontensure_replicated-does), +adds them with the rest. The archiver runs on one node and registers a +table, its view and its registry row, in one transaction, so a peer applies them together and only gets these rows by replication; without `tiered_views` a peer cannot read the cold tier, and `INSERT`/`UPDATE`/`DELETE`/DDL-blocking stop recognizing the view. diff --git a/docs/caveats.md b/docs/caveats.md index 5f02dc0f..270dee75 100644 --- a/docs/caveats.md +++ b/docs/caveats.md @@ -1,6 +1,11 @@ # Understanding ColdFront Caveats -Keep the following caveats in mind when running either mode: +A tiered or decoupled table behaves like an ordinary PostgreSQL table, but +the hot/cold split and the cold tier's DuckDB execution show through in a +few places: SQL statements that need special handling (`jsonb` operators, +`WITH`, `MERGE`, `TRUNCATE`, `COPY`), transaction and isolation semantics +that differ from native PostgreSQL, and a few naming and infrastructure +requirements. Keep the following in mind when running either mode: - `jsonb` reads come back as `json`. The `->`/`->>` operators work, and ColdFront translates the `::jsonb` cast, `jsonb_array_length`, diff --git a/docs/compaction.md b/docs/compaction.md index 94133734..98cc2ecf 100644 --- a/docs/compaction.md +++ b/docs/compaction.md @@ -15,7 +15,12 @@ claim. [apache/iceberg-go]: https://github.com/apache/iceberg-go -## Usage +Build the compactor binary with `make compactor`, which vets, lints +(golangci-lint required), tests, and builds `./bin/compactor`: + +```bash +make compactor +``` Run the compactor against the database server, naming the table to maintain: @@ -118,21 +123,3 @@ releases it when its PostgreSQL transaction commits, so its commit cannot interleave with a cold write's commit to the same table. Snapshot maintenance is the engine's job, not the catalog's: Lakekeeper does no Iceberg snapshot or orphan maintenance. - -## Requirements - -The compactor (`cmd/compactor`, apache/iceberg-go) reads the cold-tier Iceberg -tables directly. Build the binary with `make compactor`, which vets, lints -(golangci-lint required), tests, and builds `./bin/compactor`. - -## Next Steps - -To go further with ColdFront, consult the following documents: - -- The [Using ColdFront](using_coldfront/index.md) guide covers the server configuration the - compactor shares with the archiver, and the deployment YAML `import` writes - into it. -- The [Vector Storage](architecture_vectors.md) deep dive describes the sort - key the compactor applies to clustered tables. -- The [Architecture](architecture_guides/index.md) overview describes the bakery claim each - compaction step takes. diff --git a/docs/configuration.md b/docs/configuration.md index 498d66e9..e74bd71b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -42,7 +42,8 @@ The two `coldfront.iceberg_*` settings make a node upload the Parquet files of an `INSERT` or `COPY` outside its write lock (the bakery claim on a Spock mesh, an advisory lock on a single node) and serialize only the catalog commit. The packaged duckdb-iceberg includes the patch that this ordering requires, and the -[Distributed Setup](distributed_setup.md) guide describes the mesh settings. +[Configuring a Distributed Deployment](distributed_setup.md) guide describes +the mesh settings. ColdFront also exposes the following settings, which adjust write behavior and execution; tune them as needed: diff --git a/docs/distributed_setup.md b/docs/distributed_setup.md index a91dfb09..3910c05e 100644 --- a/docs/distributed_setup.md +++ b/docs/distributed_setup.md @@ -1,8 +1,8 @@ -# Setting Up a Distributed Deployment +# Configuring a Distributed Deployment -For multi-writer iceberg workloads, run ColdFront on N PG nodes in a Spock mesh -against the same Lakekeeper + S3. The bakery serializes commits PG-side so -writers never collide at the catalog. +For multi-writer iceberg workloads, run ColdFront on PostgreSQL nodes in a +Spock mesh against the same Lakekeeper + S3. The bakery serializes commits +PostgreSQL-side so writers never collide at the catalog. ### Per-Node `postgresql.conf` @@ -57,8 +57,10 @@ coldfront.iceberg_async_parquet = on coldfront.iceberg_bakery_patch = on ``` -The bakery has no peer-ack timeout. R-A's only failure mode is a dead peer -(would wait forever), and a liveness check inside the wait loop closes it: +The bakery has no peer-ack timeout. The +[Ricart-Agrawala (R-A)](formal/README.md) deferred-reply protocol's only +failure mode is a dead peer (would wait forever), and a liveness check +inside the wait loop closes it: the bakery treats a peer whose `pg_stat_replication.reply_time` is older than `coldfront.peer_alive_window_ms` as already acked, whatever the walsender's state (a peer that has just reconnected is catching up and still alive). The @@ -79,7 +81,8 @@ nothing waits. See [architecture_decoupled.md](architecture_decoupled.md), *Orphan reaping*. Sync-rep (`synchronous_standby_names`) is **not required** by the bakery - the -R-A ack barrier is what serializes iceberg commits. You can still enable +R-A ack barrier (the wait, before a claim commits, for every live peer to +acknowledge it) is what serializes iceberg commits. You can still enable sync-rep cluster-wide if you want stronger durability for non-bakery writes, but it plays no part in iceberg-commit serialization. @@ -122,13 +125,13 @@ needed. ### Timeouts -The following table lists the two settings that decide how long a cold write -waits for a peer; set them together, on every node: +The following table lists the two `postgresql.conf` settings that decide +how long a cold write waits for a peer; set them together, on every node: -| Setting | Recommended | Where | What it does | -|---|---|---|---| -| `wal_sender_timeout` | `15s` | `postgresql.conf` on every node; the image sets it in mesh mode | The node's walsender asks each peer for a reply after half of it (7.5 s) and drops a peer that has not replied for the whole of it. | -| `coldfront.peer_alive_window_ms` | `10000` (the default) | `postgresql.conf`, superuser-only | The bakery treats a peer whose last reply is older than this as dead and stops waiting for its ack. | +| Setting | Notes | +|---|---| +| `wal_sender_timeout` | Recommended `15s`, set in `postgresql.conf` on every node (the image sets it if you are running in mesh mode). The node's walsender asks each peer for a reply after half of it (7.5 s) and drops a peer that has not replied for the whole of it. | +| `coldfront.peer_alive_window_ms` | Recommended `10000` (the default), set in `postgresql.conf`, by a superuser only. The bakery treats a peer whose last reply is older than this as dead and stops waiting for its ack. | The rule: half of `wal_sender_timeout` must stay below the window, with a round trip to spare; the claim enforces the first part. An idle peer replies @@ -191,7 +194,7 @@ SELECT c.relname ORDER BY 1; ``` -Then verify the mesh before benching - insert a sentinel claim on each node +Then verify the mesh before benchmarking - insert a sentinel claim on each node and read it back from every other node. All N×(N-1) directions must show the row before traffic starts. `ci/journey.sh` `story_mesh_substrate` is a copyable reference: it checks the membership above on every node and then the diff --git a/docs/reading_writing.md b/docs/reading_writing.md deleted file mode 100644 index 876331dd..00000000 --- a/docs/reading_writing.md +++ /dev/null @@ -1,39 +0,0 @@ -# Reading and Writing ColdFront Tables - -The same SQL works against the relation name in either mode, as the following -examples show: - -```sql --- Reads - pg_duckdb handles the iceberg side; PG handles the heap side -SELECT count(*) FROM events; -SELECT id, status, data->>'k' FROM events WHERE ts >= '2026-04-01'; - --- Inserts, updates, and deletes all go through the coldfront C hook, --- which rewrites the query into one PG-set-based statement (hot side) --- + duckdb.raw_query calls (cold side). For iceberg-only mode every write --- goes cold; for tiered mode the hook splits by ts vs the watermark. -INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{"k":1}'); -UPDATE events SET status = 'fixed' WHERE id = 123; -DELETE FROM events WHERE ts < '2025-01-01'; - --- Bulk INSERT shapes: the source is read once, the hot rows are one --- set-based INSERT and the cold rows are written in batches: -INSERT INTO events (ts, status, data) VALUES (...), (...), (...); -INSERT INTO events (ts, status, data) SELECT ts, status, data FROM staging; -INSERT INTO events (ts, status, data) SELECT now() + i*'1s'::interval, 'ok', '{}' - FROM generate_series(1, 1000) i; --- A WITH clause works in both positions; a nested INSERT takes no RETURNING: -WITH moved AS (DELETE FROM staging RETURNING ts, status, data) -INSERT INTO events (ts, status, data) SELECT ts, status, data FROM moved; -WITH i AS (INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{}')) -SELECT 1; --- COPY FROM loads the same way, one INSERT per cold_write_batch_size rows: -COPY events (ts, status, data) FROM '/path/to/events.csv' WITH (FORMAT csv); - --- Transactions work; ROLLBACK undoes Iceberg writes too -BEGIN; - UPDATE events SET status = 'pending' WHERE id = 1; - SELECT status FROM events WHERE id = 1; -- sees 'pending' if row 1 is hot -ROLLBACK; -SELECT status FROM events WHERE id = 1; -- back to whatever it was -``` diff --git a/docs/supported_types.md b/docs/supported_types.md index 151e639a..cf5fe1da 100644 --- a/docs/supported_types.md +++ b/docs/supported_types.md @@ -1,41 +1,70 @@ -# Understanding Supported Column Types +# Supported Column Types The following PostgreSQL column types are supported: -`bigint` · `integer` · `smallint` · `real` · `double precision` · -`boolean` · `timestamp with time zone` · `timestamp without time zone` · -`date` · `time without time zone` · `uuid` · `text` · `varchar(N)` · -`char(N)` · `bytea` · `numeric(P,S)` (P ≤ 38) · `jsonb` / `json` · -`interval` · `vector(N)` / `halfvec(N)` (pgvector; see -[usage_vectors.md](usage_vectors.md)) · one-dimensional arrays of most of -these types (see [Arrays](#arrays)) - -Anything else (unbounded `numeric`, `xml`, `tsvector`, range/multirange types, -custom enums, composite types, pgvector's `sparsevec`, and the arrays that -[Arrays](#arrays) lists as refused at registration) is rejected when a tiered -table is registered, and when a decoupled table is created. ColdFront refuses -silent fallback to `varchar` - losing precision/identity is worse than no -support. - -Iceberg cannot hold a `numeric` `NaN`, or an `infinity` or `-infinity` in a -`date`, `timestamp` or `timestamptz` column, so ColdFront refuses those values, -as array elements too. A tiered table's hot table has a check constraint named -`coldfront_guard_` on each column of those types and on each array -column; an `INSERT` or `UPDATE` that stores one of the values fails with an -error that names the constraint. A write that goes to the cold tier, in either -mode, is refused by DuckDB. - -The archiver adds the constraints at the first archive pass that tiers a -partition of the table, and validates them once that pass's bootstrap -transaction has committed. A value written before then is accepted, and each -archive pass stops with an error that names the constraint until the row is -fixed; the next pass then validates the constraint and tiers the table. A -column that an `ALTER TABLE` adds through the DDL hook gets its constraint in -the same statement, which PostgreSQL validates against every row of the hot -table, so an `ADD COLUMN` whose default the constraint refuses, such as -`DEFAULT 'NaN'`, fails. A renamed column's constraint takes the new column -name, and a column whose name is longer than 47 bytes gets a constraint named -after a hash of the name. +- `bigint`. +- `integer`. +- `smallint`. +- `real`. +- `double precision`. +- `boolean`. +- `timestamp with time zone`. +- `timestamp without time zone`. +- `date`. +- `time without time zone`. +- `uuid`. +- `text`. +- `varchar(N)`. +- `char(N)`. +- `bytea`. +- `numeric(P,S)` (P ≤ 38). +- `jsonb` / `json`. +- `interval`. +- `vector(N)` / `halfvec(N)` (pgvector; see [usage_vectors.md](usage_vectors.md)). +- one-dimensional arrays of most of these types (see + [Supported Array Types](#supported-array-types)). + +Other data types (unbounded `numeric`, `xml`, `tsvector`, range/multirange types, +custom enums, composite types, pgvector's `sparsevec`, `inet`, `cidr`, `oid`, +and the arrays that [Supported Array Types](#supported-array-types) lists +as refused at registration) is +rejected when a tiered table is registered, and when a decoupled table is +created. ColdFront refuses silent fallback to `varchar` - losing +precision/identity is worse than no support. `inet`/`cidr`/`oid` are +rejected because pg_duckdb cannot process them in any Iceberg-backed query, +so no cast makes them readable; store IP data as `text` and `oid` values as +`bigint` instead (you can still index/compare them; cast on the hot side +only if needed). + +A few supported types need a closer look: + +- Iceberg cannot hold a `numeric` `NaN`, or an `infinity` or `-infinity` + value in a `date`, `timestamp with time zone` or `timestamp without time + zone` column, as array elements too. ColdFront refuses those values with + a check constraint named `coldfront_guard_`; a write that goes to + the cold tier, in either mode, is refused by DuckDB. +- `char(N)` is stored and read as `varchar`. Values, comparisons, and + `length()` round-trip losslessly, but a cold read comes back unpadded + with `pg_typeof varchar`, because pg_duckdb has no fixed-length `char` + type. Use `text` or `varchar` if blank-padded display matters. +- `json` and `jsonb` are stored as `varchar` in Iceberg (no native + primitive); reads come back as DuckDB's `json`, not the rich PG `jsonb` + type, because Iceberg-backed reads run entirely in DuckDB. +- `interval` is stored as `varchar` in Iceberg, as ` months + days seconds`, the text PostgreSQL and DuckDB both read as the + same value; it is view-cast back to the rich PG type on read. + +The archiver adds the `NaN`/infinity constraints at the first archive pass +that tiers a partition of the table, and validates them once that pass's +bootstrap transaction has committed. A value written before then is accepted, +and each archive pass stops with an error that names the constraint until the +row is fixed; the next pass then validates the constraint and tiers the +table. A column that an `ALTER TABLE` adds through the DDL hook gets its +constraint in the same statement, which PostgreSQL validates against every +row of the hot table, so an `ADD COLUMN` whose default the constraint +refuses, such as `DEFAULT 'NaN'`, fails. A renamed column's constraint takes +the new column name, and a column whose name is longer than 47 bytes gets a +constraint named after a hash of the name. A cold `UPDATE ... FROM`, a `MERGE` whose source is a PostgreSQL table, and an `INSERT ... SELECT` into a decoupled table read that PostgreSQL table through @@ -43,31 +72,19 @@ DuckDB's postgres extension, which reads a `numeric` `NaN` as `0` and drops an array's lower bound. ColdFront cannot refuse those values on that path, so filter them out of such a source. -`char(N)` is stored and read as `varchar`. The data round-trips losslessly: -values, comparisons, and `length()` match a hot PG table, where `length()` -already ignores `char(N)` trailing padding. The only difference is cosmetic: a -cold `char(N)` column reads back unpadded with `pg_typeof varchar`, because -pg_duckdb has no fixed-length `char` type. Use `text` or `varchar` if -blank-padded display matters. - -`json`, `jsonb` and `interval` are stored as `varchar` in Iceberg (no native -primitive), an `interval` as ` months days seconds`, -the text PostgreSQL and DuckDB both read as the same value. On read, -`interval` is view-cast back to the rich PG type; `json` -and `jsonb` come back as DuckDB's `json` (the equivalent of PG's `jsonb`), not -the rich PG `jsonb` type, because Iceberg-backed reads run entirely in DuckDB. -Queries like `data->>'key'` and `data->'key'` work, and ColdFront translates -the `::jsonb` cast, `jsonb_array_length`, `jsonb_build_object` / `jsonb_agg` -(and their `json_` twins, which DuckDB also lacks) and `date_bin` on read: the -builders become the `concat` / `to_json` / `array_agg` form both engines -evaluate identically (the result is JSON-equal to jsonb's rendering, in compact -form and in argument order), and `date_bin` becomes DuckDB's `time_bucket`, -which takes the same arguments and agrees on every fixed-width bucket. The -jsonb-only operators (`?`, `@>`, `<@`, `#>`, `#>>`) and most jsonb functions -(`jsonb_typeof`, `jsonb_extract_path`, `jsonb_extract_path_text`, `jsonb_set`, -`jsonb_path_*`, `jsonb_each`, `jsonb_object_keys`) are not supported on tiered -or decoupled data: DuckDB either lacks them or its same-named function differs -in signature or result. Reach into the document with `->`/`->>`. +Queries like `data->>'key'` and `data->'key'` work against `jsonb`, and +ColdFront translates the `::jsonb` cast, `jsonb_array_length`, +`jsonb_build_object` / `jsonb_agg` (and their `json_` twins, which DuckDB +also lacks) and `date_bin` on read: the builders become the `concat` / +`to_json` / `array_agg` form both engines evaluate identically (the result +is JSON-equal to jsonb's rendering, in compact form and in argument order), +and `date_bin` becomes DuckDB's `time_bucket`, which takes the same +arguments and agrees on every fixed-width bucket. The jsonb-only operators +(`?`, `@>`, `<@`, `#>`, `#>>`) and most jsonb functions (`jsonb_typeof`, +`jsonb_extract_path`, `jsonb_extract_path_text`, `jsonb_set`, `jsonb_path_*`, +`jsonb_each`, `jsonb_object_keys`) are not supported on tiered or decoupled +data: DuckDB either lacks them or its same-named function differs in +signature or result. Reach into the document with `->`/`->>`. These limits apply only to reads that go through a tiered or iceberg-only view, which pg_duckdb plans entirely in DuckDB. Ordinary (non-tiered) PostgreSQL @@ -92,13 +109,7 @@ DuckDB entirely. Such a read returns `data` as native `jsonb` rather than the view's `json`. Reads that span tiers, are cold-only, or reach the view through a join or sub-query stay in DuckDB, where the limits above apply. -`inet`/`cidr`/`oid` are **not supported**: pg_duckdb cannot process them -(`inet` Oid 869, `oid` Oid 26) in any Iceberg-backed query, and every -cross-tier read is planned by pg_duckdb - so no cast makes them readable. Store -IP data as `text` and `oid` values as `bigint` (you can still index/compare -them; cast on the hot side only if needed). - -### Arrays +### Supported Array Types A one-dimensional array of a supported type tiers as an Iceberg list of that type. The following table shows how each array type is stored and how the view diff --git a/docs/usage_decoupled.md b/docs/usage_decoupled.md index ff96809d..c61378f7 100644 --- a/docs/usage_decoupled.md +++ b/docs/usage_decoupled.md @@ -51,7 +51,7 @@ In a mesh one node provisions: Spock's `ddl_sql` repset replicates the `CREATE VIEW`, and the `default` repset replicates the name-keyed registry row, which enables the write hook on every peer. Each node also needs the one-time setup call `coldfront.ensure_replicated()` - see -[Distributed Setup](distributed_setup.md). +[Configuring a Distributed Deployment](distributed_setup.md). ## Changing a Decoupled Table's Columns @@ -130,8 +130,8 @@ because the grant reads the registry at call time. In a mesh one node adopts. The wrapper view replicates through Spock's `ddl_sql` repset and the registry row through the `default` repset (see -[Distributed Setup](distributed_setup.md)), -so every peer reads and writes the table under the same name, and a peer's own +[Configuring a Distributed Deployment](distributed_setup.md)), so every peer +reads and writes the table under the same name, and a peer's own adopt is refused as already registered. Adoption takes the table's bakery claim around that check and the registry write, so two nodes adopting the same table at the same moment are serialized: the second waits, then reads diff --git a/docs/using_coldfront/index.md b/docs/using_coldfront/index.md index 352cb71e..100e961d 100644 --- a/docs/using_coldfront/index.md +++ b/docs/using_coldfront/index.md @@ -14,7 +14,42 @@ compares the two modes that use the cold Iceberg tier: Once a tiered or decoupled table exists, **the SQL surface is identical**: `SELECT`, `INSERT`, `UPDATE`, `DELETE` all work normally against the relation name (e.g. `events`), except that a write touching the cold tier rejects -`RETURNING`. +`RETURNING`, as the following examples show: + +```sql +-- Reads - pg_duckdb handles the iceberg side; PG handles the heap side +SELECT count(*) FROM events; +SELECT id, status, data->>'k' FROM events WHERE ts >= '2026-04-01'; + +-- Inserts, updates, and deletes all go through the coldfront C hook, +-- which rewrites the query into one PG-set-based statement (hot side) +-- + duckdb.raw_query calls (cold side). For iceberg-only mode every write +-- goes cold; for tiered mode the hook splits by ts vs the watermark. +INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{"k":1}'); +UPDATE events SET status = 'fixed' WHERE id = 123; +DELETE FROM events WHERE ts < '2025-01-01'; + +-- Bulk INSERT shapes: the source is read once, the hot rows are one +-- set-based INSERT and the cold rows are written in batches: +INSERT INTO events (ts, status, data) VALUES (...), (...), (...); +INSERT INTO events (ts, status, data) SELECT ts, status, data FROM staging; +INSERT INTO events (ts, status, data) SELECT now() + i*'1s'::interval, 'ok', '{}' + FROM generate_series(1, 1000) i; +-- A WITH clause works in both positions; a nested INSERT takes no RETURNING: +WITH moved AS (DELETE FROM staging RETURNING ts, status, data) +INSERT INTO events (ts, status, data) SELECT ts, status, data FROM moved; +WITH i AS (INSERT INTO events (ts, status, data) VALUES (now(), 'ok', '{}')) +SELECT 1; +-- COPY FROM loads the same way, one INSERT per cold_write_batch_size rows: +COPY events (ts, status, data) FROM '/path/to/events.csv' WITH (FORMAT csv); + +-- Transactions work; ROLLBACK undoes Iceberg writes too +BEGIN; + UPDATE events SET status = 'pending' WHERE id = 1; + SELECT status FROM events WHERE id = 1; -- sees 'pending' if row 1 is hot +ROLLBACK; +SELECT status FROM events WHERE id = 1; -- back to whatever it was +``` Follow the guide for the mode you want: @@ -26,7 +61,6 @@ Follow the guide for the mode you want: manages plain PostgreSQL partitions with no cold tier at all. Operational topics shared across modes - caveats, vended credentials, -reading and writing, supported types, and distributed (mesh) setup - are -documented under Administration, starting with -[Caveats](../caveats.md). +supported types, and distributed (mesh) setup - are documented under +Administration, starting with [Caveats](../caveats.md). diff --git a/mkdocs.yml b/mkdocs.yml index 78700283..2773355e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -65,34 +65,34 @@ nav: - Decoupled (Iceberg-Only) Operating Mode: architecture_decoupled.md - Vector Storage Architecture: architecture_vectors.md - - Installing ColdFront: installation.md - - Configuring ColdFront: - - configuration.md - - Configuring Lakekeeper: one_time_setup.md - - Configuring your Object Store: object_store.md + - Installing and Configuring ColdFront: + - Installing ColdFront: installation.md + - Configuring ColdFront: + - configuration.md + - Configuring Lakekeeper: one_time_setup.md + - Configuring your Object Store: object_store.md + - Configuring a Distributed Deployment: distributed_setup.md - - Connecting to ColdFront: connecting.md + - Using ColdFront: + - using_coldfront/index.md + - Using ColdFront in Tiered Mode: usage_tiered.md + - Using ColdFront in Decoupled Mode: usage_decoupled.md + - Using ColdFront in Standalone Partitioned Mode: usage_partitioner.md + - Working with Embeddings: usage_vectors.md - - Exploring ColdFront: + - Exploring ColdFront Modes: - exploring_coldfront/index.md - Exploring Tiered Storage: walkthrough_tiered.md - Exploring Decoupled Mode: walkthrough_decoupled.md - Exploring the Standalone Partitioner: walkthrough_partitioner.md - Exploring Distributed Mode: walkthrough_distributed.md - - Using ColdFront: - - using_coldfront/index.md - - Using ColdFront in Tiered Mode: usage_tiered.md - - Using ColdFront in Decoupled Mode: usage_decoupled.md - - Using ColdFront in Standalone Partitioned Mode: usage_partitioner.md - - Working with Embeddings: usage_vectors.md + - Connecting to ColdFront: connecting.md - Administration: - Understanding ColdFront Caveats: caveats.md - Using Vended Credentials: vended_credentials.md - - Reading and Writing ColdFront Tables: reading_writing.md - - Understanding Supported Column Types: supported_types.md - - Setting Up a Distributed Deployment: distributed_setup.md + - Supported Column Types: supported_types.md - Compacting the Cold Tier: compaction.md - Developer Resources: From 0073410aecfd6a36f3d87b4d6a4c3061e70a28ff Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Tue, 13 Oct 2026 12:00:53 -0400 Subject: [PATCH 04/10] docs: remove the temporary Source page and its nav entry --- docs/source.md | 27 --------------------------- mkdocs.yml | 2 -- 2 files changed, 29 deletions(-) delete mode 100644 docs/source.md diff --git a/docs/source.md b/docs/source.md deleted file mode 100644 index 5d08b446..00000000 --- a/docs/source.md +++ /dev/null @@ -1,27 +0,0 @@ -# Source - -This page preserves every page from the original documentation tree, -unchanged and in its original order, as a reference while the -navigation above is reorganized into the standard pgEdge structure. - -- [pgEdge ColdFront](index.md) -- [Installing and Configuring ColdFront](installation.md) -- Getting Started - - [Getting Started with ColdFront](exploring_coldfront/index.md) - - [Exploring Tiered Storage](walkthrough_tiered.md) - - [Exploring Decoupled Mode](walkthrough_decoupled.md) - - [Exploring the Standalone Partitioner](walkthrough_partitioner.md) - - [Exploring Distributed Mode](walkthrough_distributed.md) - - [Configuring your Object Store](object_store.md) -- Architecture - - [pgEdge ColdFront - Architecture](architecture_guides/index.md) - - [ColdFront - Tiered Operating Mode](architecture_tiered.md) - - [Decoupled (Iceberg-Only) Operating Mode](architecture_decoupled.md) - - [Vector Storage Architecture](architecture_vectors.md) -- [Using ColdFront](using_coldfront/index.md) -- [Working with Embeddings](usage_vectors.md) -- [COMPACTOR - Cold-Tier Table Maintenance](compaction.md) -- Developer Resources - - [Formal Model - ColdFront Decoupled-Mode Bakery (TLA+/PlusCal)](formal/README.md) -- [Changelog](changelog.md) -- [License](LICENSE.md) diff --git a/mkdocs.yml b/mkdocs.yml index 2773355e..468731bb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -100,5 +100,3 @@ nav: - Release Notes - Changelog: changelog.md - License: LICENSE.md - - - Source: source.md From 69fcd2f372ff0e95e9b5058d0042152810a37907 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 07:33:49 -0400 Subject: [PATCH 05/10] docs: format DuckDB's postgres extension name as code --- docs/architecture_decoupled.md | 2 +- docs/architecture_guides/index.md | 22 +++++++++++----------- docs/architecture_tiered.md | 2 +- docs/changelog.md | 2 +- docs/supported_types.md | 2 +- 5 files changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/architecture_decoupled.md b/docs/architecture_decoupled.md index 584ce02e..4e408dd3 100644 --- a/docs/architecture_decoupled.md +++ b/docs/architecture_decoupled.md @@ -100,7 +100,7 @@ notes: |---|---|---| | Lazy catalog ATTACH | The C hook calls `ensure_attached()` on the first query that touches a tiered view. | The ATTACH costs one round-trip on the first Iceberg query per session. | | CREATE TABLE | `SELECT duckdb.raw_query('CREATE TABLE ice.. (...)')` | The statement is DuckDB SQL in attached-catalog syntax. | -| INSERT | The C hook rewrites an `INSERT INTO ` whose source is `VALUES (…)`, `SELECT … FROM ` or `SELECT … FROM generate_series(…)` to one `SELECT coldfront._exec_iceberg_with_claim(, 'INSERT INTO ice.. …')`. Source-table references get the prefix `pglocal..
`, so DuckDB's postgres extension streams the source via libpq into the Iceberg writer. That attachment needs `coldfront.local_pg_dsn` (see [Bootstrap Sequence](#bootstrap-sequence)). A `WITH` entry that modifies data is refused, because DuckDB 1.5 has no data-modifying `WITH`. An `INSERT` nested in a `WITH` entry is rewritten in place when it reads no other entry, since DuckDB does not see them. | Each INSERT produces a single Iceberg snapshot, regardless of row count. | +| INSERT | The C hook rewrites an `INSERT INTO ` whose source is `VALUES (…)`, `SELECT … FROM ` or `SELECT … FROM generate_series(…)` to one `SELECT coldfront._exec_iceberg_with_claim(, 'INSERT INTO ice.. …')`. Source-table references get the prefix `pglocal..
`, so DuckDB's `postgres` extension streams the source via libpq into the Iceberg writer. That attachment needs `coldfront.local_pg_dsn` (see [Bootstrap Sequence](#bootstrap-sequence)). A `WITH` entry that modifies data is refused, because DuckDB 1.5 has no data-modifying `WITH`. An `INSERT` nested in a `WITH` entry is rewritten in place when it reads no other entry, since DuckDB does not see them. | Each INSERT produces a single Iceberg snapshot, regardless of row count. | | UPDATE | The hook rewrites `UPDATE SET … WHERE …` to `SELECT coldfront._exec_iceberg_with_claim(, 'UPDATE ice.. SET ... WHERE ...')`. | Iceberg applies the update as merge-on-read. | | DELETE | The hook rewrites `DELETE FROM WHERE …` to `SELECT coldfront._exec_iceberg_with_claim(, 'DELETE FROM ice.. WHERE ...')`. | Iceberg records the delete in position-delete files. | | MERGE (PostgreSQL 17 and later) | The hook rewrites `MERGE INTO USING … ON … WHEN …` to `SELECT coldfront._exec_iceberg_with_claim(, 'MERGE INTO ice.. …')`, the source's PostgreSQL tables prefixed with `pglocal..
` and each `INSERT` action's `OVERRIDING` clause dropped. Nested in a `WITH` entry it is rewritten in place when it reads no other entry, since DuckDB does not see them. | duckdb-iceberg runs one `UPDATE` or `DELETE` action per statement, and no `RETURNING`. | diff --git a/docs/architecture_guides/index.md b/docs/architecture_guides/index.md index f1cdb674..0dbc62c5 100644 --- a/docs/architecture_guides/index.md +++ b/docs/architecture_guides/index.md @@ -175,17 +175,17 @@ view, so a missing warehouse never blocks a connection opened before bootstrap. pg_duckdb force-disables DuckDB's `LocalFileSystem` for any role that lacks both `pg_read_server_files` and `pg_write_server_files` (see [Upstream requests](#pg_duckdb-non-superuser-localfilesystem-blocks-side-loaded-extensions)), -which would block the side-loaded iceberg/postgres DuckDB extensions from -loading on `ATTACH`. So `coldfront.ensure_attached()` / `ensure_pg_attached()` -are `SECURITY DEFINER` with a pinned `search_path`: the extension load + -`ATTACH` run elevated (gates key off `GetUserId()`, the effective user), and -because the DuckDB instance is per-backend the attach persists for the -session - every subsequent cold read / `_exec_iceberg_with_claim` then runs -as the **app role** over S3/httpfs, never touching `LocalFileSystem`. The app -role needs only `duckdb.postgres_role` membership, object grants, and SET on -the superuser-only `duckdb.unsafe_allow_execution_inside_functions` -parameter, which the cross-tier move needs; **no superuser, no -`pg_{read,write}_server_files`**. +which would block the side-loaded `iceberg` and `postgres` DuckDB extensions +from loading on `ATTACH`. So `coldfront.ensure_attached()` / +`ensure_pg_attached()` are `SECURITY DEFINER` with a pinned `search_path`: the +extension load + `ATTACH` run elevated (gates key off `GetUserId()`, the +effective user), and because the DuckDB instance is per-backend the attach +persists for the session - every subsequent cold read / +`_exec_iceberg_with_claim` then runs as the **app role** over S3/httpfs, never +touching `LocalFileSystem`. The app role needs only `duckdb.postgres_role` +membership, object grants, and SET on the superuser-only +`duckdb.unsafe_allow_execution_inside_functions` parameter, which the +cross-tier move needs; **no superuser, no `pg_{read,write}_server_files`**. Because the attach helpers run elevated, the deployment-config GUCs they consume (`coldfront.warehouse`, `coldfront.lakekeeper_endpoint`, diff --git a/docs/architecture_tiered.md b/docs/architecture_tiered.md index 02af4c20..385f3ef4 100644 --- a/docs/architecture_tiered.md +++ b/docs/architecture_tiered.md @@ -119,7 +119,7 @@ batches. An `INSERT` nested in a `WITH` entry is rewritten in place, and a The bootstrap also gives the hot table its guards: a check constraint, named `coldfront_guard_`, on each column whose type admits a value the cold tier cannot hold unchanged. A `numeric` column refuses `NaN`, which pg_duckdb's -scan and DuckDB's postgres extension both read as `0` and which Iceberg's +scan and DuckDB's `postgres` extension both read as `0` and which Iceberg's decimal lacks; a `date`, `timestamp` or `timestamptz` column refuses `infinity` and `-infinity`, which Iceberg lacks and duckdb-iceberg refuses to write. An array column keeps the shape of an Iceberg list, one dimension numbered from 1, diff --git a/docs/changelog.md b/docs/changelog.md index 3a1b5435..a5e4a138 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -370,7 +370,7 @@ this project adheres to data file it can apply to is left out of the rewrite. - A `numeric` `NaN` in a tiered table read back as `0` through the view once the table had a cold tier, and was archived as `0`: pg_duckdb and DuckDB's - postgres extension read `NaN` as zero, and Iceberg's decimal has no `NaN`. + `postgres` extension read `NaN` as zero, and Iceberg's decimal has no `NaN`. The hot table now refuses `NaN` when it is written, and an archive pass refuses a table that already holds one, with an error that names the column's check constraint. diff --git a/docs/supported_types.md b/docs/supported_types.md index 6f858376..c790383b 100644 --- a/docs/supported_types.md +++ b/docs/supported_types.md @@ -69,7 +69,7 @@ constraint named after a hash of the name. A cold `UPDATE ... FROM` or `DELETE ... USING`, a `MERGE` whose source is a PostgreSQL table, and an `INSERT ... SELECT` into a decoupled table read that -PostgreSQL table through DuckDB's postgres extension, which reads a `numeric` +PostgreSQL table through DuckDB's `postgres` extension, which reads a `numeric` `NaN` as `0`, also inside an array, and drops an array's lower bound. ColdFront cannot refuse those values on that path: the write stores the changed values with no error, and its conditions compare the changed values. A condition that From 1abc9146cae4cd757fd2dcae7e89d15773e307b3 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 07:44:49 -0400 Subject: [PATCH 06/10] docs: replace stale usage.md and architecture.md references Point the remaining mentions of the removed usage.md and architecture.md at the pages that now hold that content, and rebuild the docs/ section of the README project tree from the current file list. --- README.md | 14 ++++++++++---- docs/architecture_decoupled.md | 19 ++++++++++--------- docs/architecture_tiered.md | 7 ++++--- docs/configuration.md | 2 +- docs/object_store.md | 5 ++++- 5 files changed, 29 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index bbacf1a5..69d04a8a 100644 --- a/README.md +++ b/README.md @@ -250,12 +250,18 @@ pgedge-coldfront/ │ ├── entrypoint.sh │ └── seaweedfs-s3.json ← SeaweedFS S3 auth config (example) ├── docs/ ← MkDocs site (user docs; mkdocs.yml at repo root) -│ ├── index.md · walkthrough.md · installation.md +│ ├── index.md · installation.md · configuration.md +│ ├── one_time_setup.md · object_store.md · distributed_setup.md +│ ├── usage_tiered.md · usage_decoupled.md · usage_partitioner.md +│ ├── usage_vectors.md · connecting.md · caveats.md +│ ├── vended_credentials.md · supported_types.md · compaction.md │ ├── walkthrough_tiered.md · walkthrough_decoupled.md │ ├── walkthrough_partitioner.md · walkthrough_distributed.md -│ ├── object_store.md · usage.md · compaction.md -│ ├── architecture.md · architecture_tiered.md · architecture_decoupled.md -│ ├── architecture_vectors.md · usage_vectors.md · changelog.md +│ ├── architecture_tiered.md · architecture_decoupled.md +│ ├── architecture_vectors.md · changelog.md · LICENSE.md +│ ├── architecture_guides/ ← Architecture Guides landing page (index.md) +│ ├── exploring_coldfront/ ← Exploring ColdFront Modes landing page (index.md) +│ ├── using_coldfront/ ← Using ColdFront landing page (index.md) │ └── formal/ ← TLA+ model of the bakery protocol (Bakery.tla) ├── examples/walkthrough/ ← interactive walkthrough (guide.sh, its compose stack and configs) ├── docker-compose.yml ← END-USER single-node stack (ports published) diff --git a/docs/architecture_decoupled.md b/docs/architecture_decoupled.md index 4e408dd3..1658c6ba 100644 --- a/docs/architecture_decoupled.md +++ b/docs/architecture_decoupled.md @@ -6,9 +6,9 @@ archiver. PostgreSQL becomes a stateless compute front-end. Lakekeeper and the underlying object store own the storage. Decoupled mode shares the same codebase, docker stack and extension as tiered mode. The shared mechanics - pg_duckdb Iceberg I/O, the rewrite hook, the bakery protocol, the registry - -are in [architecture.md](architecture_guides/index.md); tiered mode is in -[architecture_tiered.md](architecture_tiered.md). This document covers what is -specific to decoupled mode. +are in the [Architecture Overview](architecture_guides/index.md); tiered mode +is in [Tiered Operating Mode](architecture_tiered.md). This document covers +what is specific to decoupled mode. ## What "Decoupled" Means @@ -522,8 +522,9 @@ Three properties are worth knowing: ## ACID Model -(This section summarizes material from [architecture.md](architecture_guides/index.md) -§Concurrency and §Known Limitations applied to the decoupled scenario.) +(This section summarizes material from the +[Architecture Overview](architecture_guides/index.md) §Concurrency and §Known +Limitations applied to the decoupled scenario.) The following table shows the status of each ACID property in decoupled mode: @@ -705,9 +706,9 @@ pointing at the same Lakekeeper endpoint and S3 bucket: `spock.feedback_frequency` messages, and in answer to the walsender's keepalive, which on an idle link comes every `wal_sender_timeout/2`, so the claim refuses to run unless `wal_sender_timeout` is positive and - below twice the window; usage.md gives the default and the rule. An - *alive* - peer that has not acked is either deferring (R-A's defer rule, legitimate) + below twice the window; + [Timeouts](distributed_setup.md#timeouts) gives the default and the + rule. An *alive* peer that has not acked is either deferring (R-A's defer rule, legitimate) or about to ack - either way, waiting is correct. A same-node claim is released by the C XactCallback in [extension/coldfront/src/coldfront.c](https://github.com/pgEdge/ColdFront/blob/main/extension/coldfront/src/coldfront.c) @@ -750,7 +751,7 @@ settings: wal_level = logical shared_preload_libraries = 'snowflake,spock,pg_duckdb,coldfront' -# Set together with coldfront.peer_alive_window_ms (see usage.md). +# Set together with coldfront.peer_alive_window_ms (see distributed_setup.md). wal_sender_timeout = 15s # Sync-rep is NOT required by the bakery - R-A's ack barrier replaces it. diff --git a/docs/architecture_tiered.md b/docs/architecture_tiered.md index 385f3ef4..07f3b538 100644 --- a/docs/architecture_tiered.md +++ b/docs/architecture_tiered.md @@ -6,9 +6,10 @@ archiver moves rows hot→cold on a cron. This document covers the **tiered-specific** design. The shared mechanics - pg_duckdb Iceberg I/O, the rewrite hook, the bakery protocol, the registry, DDL -handling, infrastructure - live in [architecture.md](architecture_guides/index.md); the -all-Iceberg alternative is -[architecture_decoupled.md](architecture_decoupled.md). +handling, infrastructure - live in the +[Architecture Overview](architecture_guides/index.md); the all-Iceberg +alternative is +[Decoupled (Iceberg-Only) Operating Mode](architecture_decoupled.md). ## Data Flow diff --git a/docs/configuration.md b/docs/configuration.md index e74bd71b..763e9325 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -55,7 +55,7 @@ behavior and execution; tune them as needed: | `coldfront.vector_probe` | Sets whether a recognized similarity search reads only the clusters nearest its query vector. `off` gives an exact scan of the whole corpus. Affects only a table with a trained vector column ([usage_vectors.md](usage_vectors.md)). The default is `on`. | | `coldfront.vector_nprobe` | Sets how many clusters such a search reads, overriding the column's own `nprobe`; at or above the column's `nlist` the search is exhaustive. The default is `0`, which uses the configured value. | | `duckdb.force_execution` | Benchmark before enabling: on a mixed workload it helps `count(distinct)` and similar but regresses index lookups, top-K with PK ordering, and JSON access. The default is `off`. | -| `duckdb.temporary_directory` | Sets where DuckDB spills. Each backend gets its own subdirectory there, named after its process id, so concurrent spills cannot collide; one left by a departed backend is reclaimed. See [architecture.md](architecture_guides/index.md#duckdb-spill-files-are-not-namespaced-per-instance). | +| `duckdb.temporary_directory` | Sets where DuckDB spills. Each backend gets its own subdirectory there, named after its process id, so concurrent spills cannot collide; one left by a departed backend is reclaimed. See [DuckDB: Spill Files Are Not Namespaced per Instance](architecture_guides/index.md#duckdb-spill-files-are-not-namespaced-per-instance). | | `duckdb.max_temp_directory_size` | A cap per connection, not a cluster total. For a total budget, divide it by the concurrent sessions, or give the temp path its own filesystem or quota. The default is 90% of free space per session. | ## Writing ColdFront's Configuration diff --git a/docs/object_store.md b/docs/object_store.md index ffc7fdc0..cc5f48f1 100644 --- a/docs/object_store.md +++ b/docs/object_store.md @@ -479,7 +479,10 @@ You drive tiered tables with the `archiver` binary against a small YAML config (Postgres DSN, your warehouse name, and your backend's credentials); each table's lifecycle is registered with `archiver register`. The credential and warehouse you set up above are exactly what it needs. See -[usage.md](using_coldfront/index.md) for the archiver config and the partition CLI. +[Writing ColdFront's Configuration](configuration.md#writing-coldfronts-configuration) +for the archiver configuration and +[Managing Partitioned Tables (CLI)](usage_partitioner.md#managing-partitioned-tables-cli) +for the partition CLI. --- From 7e56e95e201e12575680a550e28f153734619905 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 07:48:09 -0400 Subject: [PATCH 07/10] docs: use page titles instead of filenames as link text --- docs/architecture_guides/index.md | 21 +++++++++++---------- docs/caveats.md | 4 ++-- docs/configuration.md | 2 +- docs/distributed_setup.md | 4 ++-- docs/object_store.md | 13 +++++++------ docs/one_time_setup.md | 3 ++- docs/supported_types.md | 3 ++- docs/vended_credentials.md | 4 ++-- 8 files changed, 29 insertions(+), 25 deletions(-) diff --git a/docs/architecture_guides/index.md b/docs/architecture_guides/index.md index 0dbc62c5..3f3222f1 100644 --- a/docs/architecture_guides/index.md +++ b/docs/architecture_guides/index.md @@ -7,10 +7,11 @@ rewrites reads and writes to hit the right tier, and **all Iceberg I/O goes through `pg_duckdb` running in-process inside PostgreSQL** - no external query engine, no Go Iceberg libraries. -For mode-specific design, see [architecture_tiered.md](../architecture_tiered.md) -(hot PG + cold Iceberg) · -[architecture_decoupled.md](../architecture_decoupled.md) (all-Iceberg) · -[architecture_vectors.md](../architecture_vectors.md) (vector storage). +For mode-specific design, see +[Tiered Operating Mode](../architecture_tiered.md) (hot PostgreSQL plus cold +Iceberg), +[Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md), and +[Vector Storage Architecture](../architecture_vectors.md). ## Operating Modes and Topologies @@ -33,8 +34,8 @@ path in the hook takes, mostly through `_exec_iceberg_with_claim`. Decoupled mode always classifies as `TIER_COLD` and never reaches `emit_hot`; vanilla and mesh differ only in how that claim serializes cold writes. This document covers the shared mechanics and the tiered path; see -[architecture_decoupled.md](../architecture_decoupled.md) for the decoupled mode's -ACID model and distributed scaling story. +[Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md) for the +decoupled mode's ACID model and distributed scaling story. ### Read Target: Primary or Physical Standby @@ -106,7 +107,7 @@ How rows move through this depends on the storage mode: the tiered hot heap + archiver + `UNION ALL` data-flow is in the [Data Flow](../architecture_tiered.md#data-flow) section of the Tiered Mode page; the all-Iceberg flow is in -[architecture_decoupled.md](../architecture_decoupled.md). +[Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md). ## Core Mechanics: pg_duckdb @@ -290,7 +291,7 @@ How the hook splits a write is mode-specific: in the Tiered Mode page. - In decoupled mode, the hook always classifies `TIER_COLD`, so every write is a single-tier Iceberg write. See - [architecture_decoupled.md](../architecture_decoupled.md). + [Decoupled (Iceberg-Only) Operating Mode](../architecture_decoupled.md). ### Cold-Tier DML from Inside PL/pgSQL (Functions, DO Blocks, Triggers) @@ -359,7 +360,7 @@ The following table shows the two orderings: | Ordering | Applies to | Behavior | |---|---|---| | stock (default) | every write on stock upstream duckdb-iceberg; a `DELETE`, `UPDATE` or `MERGE` (the cross-tier move and `vector_train`'s assignment included) on every build | The writer takes the serializer first, *then* uploads parquet and commits inside it. On an unpatched binary this is what keeps a second writer from capturing a stale parent. A write that references rows the table holds needs it on every build: its position deletes name data files by path, and a compaction or another such write committing between its scan and its commit would rewrite or remove those files, after which the deletes apply to nothing and the rows come back. | -| async (both GUCs `on`) | appends (`INSERT`, `COPY`, the archiver's exports, the tiered `INSERT`'s cold sink) on the patched build (`iceberg-bakery-aware-commit-refresh-v15.patch`) | The statement uploads its parquet and queues the claim; the extension's transaction callback takes it at PRE_COMMIT, just before pg_duckdb's own PRE_COMMIT callback commits the Iceberg transaction, and releases it at COMMIT. Uploads overlap on one node and across nodes, an open transaction blocks no other writer, and only the short commit POST is serialized. A new data file conflicts with nothing committed in between. An append to a clustered table is checked at commit against the live centroid generation and fails with `serialization_failure` if a retrain replaced it (see [architecture_vectors.md](../architecture_vectors.md)). | +| async (both GUCs `on`) | appends (`INSERT`, `COPY`, the archiver's exports, the tiered `INSERT`'s cold sink) on the patched build (`iceberg-bakery-aware-commit-refresh-v15.patch`) | The statement uploads its parquet and queues the claim; the extension's transaction callback takes it at PRE_COMMIT, just before pg_duckdb's own PRE_COMMIT callback commits the Iceberg transaction, and releases it at COMMIT. Uploads overlap on one node and across nodes, an open transaction blocks no other writer, and only the short commit POST is serialized. A new data file conflicts with nothing committed in between. An append to a clustered table is checked at commit against the live centroid generation and fails with `serialization_failure` if a retrain replaced it (see [Vector Storage Architecture](../architecture_vectors.md)). | For every write but a clustered table's append, the code path and the application-visible behavior are identical, so the GUCs are a performance knob; @@ -521,7 +522,7 @@ mode (the archiver those with a `hot_period`, the partitioner those without), and exit with an error when they find none; tables enter it via `register`, or from a YAML `archiver.tables` list fed to `import`. Both expose a management CLI - `register` / `list` / `set` / `remove` / `import` / `export` - documented -in [usage_partitioner.md](../usage_partitioner.md#managing-partitioned-tables-cli). +in [Managing Partitioned Tables (CLI)](../usage_partitioner.md#managing-partitioned-tables-cli). ## Known Limitations diff --git a/docs/caveats.md b/docs/caveats.md index 58e07ebe..f43b8167 100644 --- a/docs/caveats.md +++ b/docs/caveats.md @@ -22,7 +22,7 @@ requirements. Keep the following in mind when running either mode: decoupled table sees the transaction's own writes on both tiers. The held snapshot is also why the compactor's `--expire-older-than` must stay longer than the longest transaction that reads the table (see - [compaction.md](compaction.md)). + [Compacting the Cold Tier](compaction.md)). - One transaction block cannot hold both a PostgreSQL write and a cold-tier write unless `duckdb.unsafe_allow_mixed_transactions` is on. pg_duckdb refuses the pair ("Writing to DuckDB and Postgres tables in the same @@ -55,7 +55,7 @@ requirements. Keep the following in mind when running either mode: GUC (the node's loopback, a unix-socket DSN), and a one-time `SELECT coldfront.ensure_replicated()` call on every node after Spock mesh setup; see - [architecture_decoupled.md](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). + [The Bakery Protocol](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). Sync-rep is **not** required. The throughput ceiling is Lakekeeper's commit rate, not the writer count. - For direct table access, `_events` is the hot heap (tiered mode only). diff --git a/docs/configuration.md b/docs/configuration.md index 763e9325..f4d0595c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -52,7 +52,7 @@ behavior and execution; tune them as needed: |---|---| | `coldfront.allow_mixed_writes` | Controls tiered-mode `UPDATE`/`DELETE` whose WHERE cannot be proven to target one tier. `on` emits a dual-tier CTE; `off` rejects with an error and a hint. Not relevant in decoupled mode (every write is single-tier by definition). The default is `on`. | | `coldfront.cold_write_batch_size` | Sets how many cold rows a tiered `INSERT` gathers (see [Caveats](caveats.md)) before writing them to Iceberg as one `INSERT`. A larger value writes fewer, larger Parquet files; the remainder always flushes, so a small write stays one file. The default is `10000`, with a minimum of `1`. | -| `coldfront.vector_probe` | Sets whether a recognized similarity search reads only the clusters nearest its query vector. `off` gives an exact scan of the whole corpus. Affects only a table with a trained vector column ([usage_vectors.md](usage_vectors.md)). The default is `on`. | +| `coldfront.vector_probe` | Sets whether a recognized similarity search reads only the clusters nearest its query vector. `off` gives an exact scan of the whole corpus. Affects only a table with a trained vector column (see [Working with Embeddings](usage_vectors.md)). The default is `on`. | | `coldfront.vector_nprobe` | Sets how many clusters such a search reads, overriding the column's own `nprobe`; at or above the column's `nlist` the search is exhaustive. The default is `0`, which uses the configured value. | | `duckdb.force_execution` | Benchmark before enabling: on a mixed workload it helps `count(distinct)` and similar but regresses index lookups, top-K with PK ordering, and JSON access. The default is `off`. | | `duckdb.temporary_directory` | Sets where DuckDB spills. Each backend gets its own subdirectory there, named after its process id, so concurrent spills cannot collide; one left by a departed backend is reclaimed. See [DuckDB: Spill Files Are Not Namespaced per Instance](architecture_guides/index.md#duckdb-spill-files-are-not-namespaced-per-instance). | diff --git a/docs/distributed_setup.md b/docs/distributed_setup.md index 3910c05e..159359cb 100644 --- a/docs/distributed_setup.md +++ b/docs/distributed_setup.md @@ -77,8 +77,8 @@ action: by that node's next cold write, to any table, by a peer's arriving claim, or by the waiting peer's poke, a no-op `UPDATE` of its own claim row about once a second for as long as it waits. That poke is the only replication traffic the bakery generates while a writer waits, and there is none when -nothing waits. See [architecture_decoupled.md](architecture_decoupled.md), -*Orphan reaping*. +nothing waits. See *Orphan reaping* in +[The Bakery Protocol](architecture_decoupled.md#concurrency-horizontal-scaling-the-bakery-protocol). Sync-rep (`synchronous_standby_names`) is **not required** by the bakery - the R-A ack barrier (the wait, before a claim commits, for every live peer to diff --git a/docs/object_store.md b/docs/object_store.md index cc5f48f1..50f06dfc 100644 --- a/docs/object_store.md +++ b/docs/object_store.md @@ -44,7 +44,7 @@ S3, and GCS): > credentials instead, where the warehouse issues short-lived per-table > credentials at access time. Vended credentials are available for AWS S3 > and Azure ADLS Gen2, but not for GCS or other S3-compatible stores; see -> [vended_credentials.md](vended_credentials.md). +> [Using Vended Credentials](vended_credentials.md). For AWS S3 and S3-compatible stores, grant the key permission to read, write, and list the bucket (`GetObject` / `PutObject` / `DeleteObject` / @@ -71,8 +71,8 @@ DuckDB never aborts a failed multipart upload, so add an `AbortIncompleteMultipartUpload` lifecycle rule to the bucket to delete leftover parts. -You also need the ColdFront image, built once by following -[installation.md](installation.md), which notes the registry access the +You also need the ColdFront image, built once by following the steps in +[Installing ColdFront](installation.md), which notes the registry access the base image needs. Run the commands below from the repo root. No prior ColdFront knowledge is required to perform the steps on this @@ -240,9 +240,10 @@ SELECT coldfront.set_storage_secret('AKIAEXAMPLE...', '', NULL, that shape for cloud S3. A vended-credentials variant is also available. To run ColdFront with no stored -credential ([vended_credentials.md](vended_credentials.md)), the warehouse issues -per-table temporary credentials from AWS Security Token Service (STS) instead -of handing the client a static key. Two things change from the warehouse above. +credential (see [Using Vended Credentials](vended_credentials.md)), the +warehouse issues per-table temporary credentials from AWS Security Token +Service (STS) instead of handing the client a static key. Two things change +from the warehouse above. First, create an IAM role scoped to the bucket. Lakekeeper assumes the role per table and vends the resulting short-lived key, secret, and session token to diff --git a/docs/one_time_setup.md b/docs/one_time_setup.md index d498d588..abd9166d 100644 --- a/docs/one_time_setup.md +++ b/docs/one_time_setup.md @@ -249,4 +249,5 @@ superusers can then run DuckDB. [Connecting to ColdFront](connecting.md#least-privilege-application-roles) lists what it grants. -For a real cloud-S3 setup, see [object_store.md](object_store.md). +For a real cloud-S3 setup, see +[Configuring your Object Store](object_store.md). diff --git a/docs/supported_types.md b/docs/supported_types.md index c790383b..2fa4571f 100644 --- a/docs/supported_types.md +++ b/docs/supported_types.md @@ -20,7 +20,8 @@ The following PostgreSQL column types are supported: - `numeric(P,S)` (P ≤ 38). - `jsonb` / `json`. - `interval`. -- `vector(N)` / `halfvec(N)` (pgvector; see [usage_vectors.md](usage_vectors.md)). +- `vector(N)` / `halfvec(N)` (pgvector; see + [Working with Embeddings](usage_vectors.md)). - one-dimensional arrays of most of these types (see [Supported Array Types](#supported-array-types)). diff --git a/docs/vended_credentials.md b/docs/vended_credentials.md index 97919482..43f136b5 100644 --- a/docs/vended_credentials.md +++ b/docs/vended_credentials.md @@ -32,8 +32,8 @@ Vending requires a Lakekeeper warehouse configured to vend credentials: - On AWS S3, the warehouse uses `flavor: aws` with `sts-enabled: true`, an `assume-role-arn` for a bucket-scoped IAM role, and an `external-id` on the warehouse credential that the role's trust policy also requires. The full - warehouse and IAM-role setup is in - [object_store.md](object_store.md#configuring-lakekeeper). + warehouse and IAM-role setup is in the Configuring Lakekeeper section of + [Configuring your Object Store](object_store.md#configuring-lakekeeper). - On Azure ADLS Gen2, the warehouse is an `adls` warehouse with `sas-enabled` (on by default). Lakekeeper vends a per-container SAS token; the warehouse credential can be a `shared-access-key`, `client-credentials`, or From 6be8bac9b199f2b0d33e8df358346801e5db501f Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 08:02:12 -0400 Subject: [PATCH 08/10] docs: split the opening sentence of the introduction --- docs/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index 5a440550..5f03b2ba 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,8 +1,8 @@ # pgEdge ColdFront ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet -on S3-compatible, Azure, or GCS storage), and the cold tier is both readable -and writable through the same SQL with no application changes. The application +on S3-compatible, Azure, or GCS storage). The cold tier is both readable and +writable through the same SQL, with no application changes. The application queries every table as an ordinary PostgreSQL relation, and both operating modes present the same standard SQL surface. From 51f9cfbd41cd8ce2b9c2907abaa5a3339de8a96b Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 10:15:29 -0400 Subject: [PATCH 09/10] docs: rename one_time_setup.md to configure_lakekeeper.md --- CLAUDE.md | 2 +- README.md | 8 ++++---- docs/configuration.md | 2 +- docs/{one_time_setup.md => configure_lakekeeper.md} | 0 docs/index.md | 2 +- docs/installation.md | 6 +++--- docs/usage_decoupled.md | 2 +- docs/usage_partitioner.md | 2 +- docs/usage_tiered.md | 2 +- mkdocs.yml | 2 +- 10 files changed, 14 insertions(+), 14 deletions(-) rename docs/{one_time_setup.md => configure_lakekeeper.md} (100%) diff --git a/CLAUDE.md b/CLAUDE.md index d71dab12..0eaf4df7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -112,5 +112,5 @@ Snapshots of upstream state for the next major moves. Re-verify before acting on - Three indexes point at the pages in `docs/`, each with its own focus: the `mkdocs.yml` `nav:` (everything, including changelog, license and `formal/`), the `README.md` Documentation table (one terse row per guide), and the `docs/index.md` Next Steps list (prose, user-facing, top-level guides only; the per-mode deep dives are reached through `docs/architecture_guides/index.md`, which lists them). Adding, renaming, or removing a guide under `docs/` means updating all three in the same commit; adding a deep dive means the nav, the README table, and the list in `docs/architecture_guides/index.md`. `mkdocs build --strict` passes on a page no index links, so it will not catch this: check by listing `docs/*.md` against each index. - Internal/scratch docs (bench, experimental, hardening, comparison, backlog, scale-test, E2E) are gitignored and NEVER pushed to GitHub; never reference them from published docs. - ARCHITECTURE docs are reference material, not work logs: brief, present-tense, no history ("we used to…", version churn), no rejected-alternative narration. Keep the *why* (rationale that explains the design); cut the journey. -- One home per topic, no duplication across docs: shared day-to-day usage and the partition CLI in `docs/using_coldfront/index.md`; each mode's own guide in `docs/usage_tiered.md`, `docs/usage_decoupled.md`, and `docs/usage_partitioner.md`; PostgreSQL/ColdFront config in `docs/configuration.md`; Lakekeeper install/config/bootstrap in `docs/one_time_setup.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture_guides/index.md` and `docs/architecture_tiered.md`/`docs/architecture_decoupled.md`/`docs/architecture_vectors.md`. When moving a section, fix every cross-anchor that pointed at it. +- One home per topic, no duplication across docs: shared day-to-day usage and the partition CLI in `docs/using_coldfront/index.md`; each mode's own guide in `docs/usage_tiered.md`, `docs/usage_decoupled.md`, and `docs/usage_partitioner.md`; PostgreSQL/ColdFront config in `docs/configuration.md`; Lakekeeper install/config/bootstrap in `docs/configure_lakekeeper.md`; build-from-source and testing/CI in `docs/installation.md`; storage-backend setup in `docs/object_store.md`; deep dives in `docs/architecture_guides/index.md` and `docs/architecture_tiered.md`/`docs/architecture_decoupled.md`/`docs/architecture_vectors.md`. When moving a section, fix every cross-anchor that pointed at it. - Tracked build-journal docs (`DUCKDB_1.5_PATCHED.md`, `DUCKDB_1.5_UNPATCHED.md`) are kept as history and never deleted, but they are not part of the docs site: never link them from `README.md` or `docs/`. Point build internals at the real recipe, `docker/Dockerfile.duckdb15-base`. diff --git a/README.md b/README.md index 69d04a8a..93104291 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ The ColdFront documentation consists of the following guides: - Installing and Configuring ColdFront - [Installing ColdFront](docs/installation.md) - [Configuring ColdFront](docs/configuration.md) - - [Configuring Lakekeeper](docs/one_time_setup.md) + - [Configuring Lakekeeper](docs/configure_lakekeeper.md) - [Configuring your Object Store](docs/object_store.md) - [Configuring a Distributed Deployment](docs/distributed_setup.md) - [Using ColdFront](docs/using_coldfront/index.md) @@ -147,7 +147,7 @@ settings above. The `import` command takes a deployment YAML, modeled on ColdFront has no configuration file. A YAML passed to any later run is checked against the server, and the run takes only `postgres.dsn` from the file. For every setting, see the -[Configuring Lakekeeper](docs/one_time_setup.md) guide and the +[Configuring Lakekeeper](docs/configure_lakekeeper.md) guide and the [Configuring PostgreSQL](docs/configuration.md#configuring-postgresql) section of the Configuring ColdFront guide. @@ -179,7 +179,7 @@ documents: | [Vector Storage Architecture](docs/architecture_vectors.md) | Describes vector storage internals: type mapping, routing state, cluster assignment, and layout. | | [Installing ColdFront](docs/installation.md) | Documents installing from a package or building from source (Docker or bare-metal), and testing and CI. | | [Configuring ColdFront](docs/configuration.md) | Documents configuring PostgreSQL and ColdFront's own configuration. | -| [Configuring Lakekeeper](docs/one_time_setup.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | +| [Configuring Lakekeeper](docs/configure_lakekeeper.md) | Installs, configures, and bootstraps Lakekeeper, then installs the extensions each database needs. | | [Configuring your Object Store](docs/object_store.md) | Gets ColdFront running end to end, for any of four supported backends: S3-compatible, virtual-hosted cloud S3, GCS, or Azure ADLS Gen2. | | [Configuring a Distributed Deployment](docs/distributed_setup.md) | Documents running a multi-writer decoupled-mode Spock mesh against one shared lake. | | [Using ColdFront](docs/using_coldfront/index.md) | Compares the three operating modes, documents the shared SQL surface, and links to each mode's guide. | @@ -251,7 +251,7 @@ pgedge-coldfront/ │ └── seaweedfs-s3.json ← SeaweedFS S3 auth config (example) ├── docs/ ← MkDocs site (user docs; mkdocs.yml at repo root) │ ├── index.md · installation.md · configuration.md -│ ├── one_time_setup.md · object_store.md · distributed_setup.md +│ ├── configure_lakekeeper.md · object_store.md · distributed_setup.md │ ├── usage_tiered.md · usage_decoupled.md · usage_partitioner.md │ ├── usage_vectors.md · connecting.md · caveats.md │ ├── vended_credentials.md · supported_types.md · compaction.md diff --git a/docs/configuration.md b/docs/configuration.md index f4d0595c..6e5f67cc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -7,7 +7,7 @@ components still need configuring before the cold tier comes online: guide, regardless of which installation path you followed. - installing, configuring, and bootstrapping Lakekeeper, and the one-time per-database setup that ties everything together, - documented in the [Configuring Lakekeeper](one_time_setup.md) + documented in the [Configuring Lakekeeper](configure_lakekeeper.md) guide. - your object store, documented in [Configuring your Object Store](object_store.md). diff --git a/docs/one_time_setup.md b/docs/configure_lakekeeper.md similarity index 100% rename from docs/one_time_setup.md rename to docs/configure_lakekeeper.md diff --git a/docs/index.md b/docs/index.md index 5f03b2ba..0f6d7416 100644 --- a/docs/index.md +++ b/docs/index.md @@ -103,7 +103,7 @@ following guides: from a package or building it from source. - The [Configuring ColdFront](configuration.md) guide documents PostgreSQL and ColdFront's own configuration. -- The [Configuring Lakekeeper](one_time_setup.md) guide installs, +- The [Configuring Lakekeeper](configure_lakekeeper.md) guide installs, configures, and bootstraps it, then installs the extensions each database needs. - The [Configuring your Object Store](object_store.md) guide takes you from diff --git a/docs/installation.md b/docs/installation.md index 4e51173c..aaae39ae 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -167,7 +167,7 @@ publishes it. The floating `pg` tag moves to the same image only when the base-image workflow is dispatched from `main` with `push=true` and both architectures. -Then follow the [Configuring Lakekeeper](one_time_setup.md) guide: +Then follow the [Configuring Lakekeeper](configure_lakekeeper.md) guide: bootstrap it, create a table, tier it, and verify. !!! note "pg_duckdb pin" @@ -259,7 +259,7 @@ SQL A row count of 1 read back through Iceberg confirms the full path. For a real cloud store, drop the `local-store` profile, point the warehouse at your own -bucket, and follow the [Configuring Lakekeeper](one_time_setup.md) guide +bucket, and follow the [Configuring Lakekeeper](configure_lakekeeper.md) guide for the full tier-and-verify journey. ## Building ColdFront on Bare Metal @@ -364,7 +364,7 @@ To go further with ColdFront, consult the following guides: - The [Configuring ColdFront](configuration.md) guide details PostgreSQL and ColdFront's own configuration. -- The [Configuring Lakekeeper](one_time_setup.md) guide installs, +- The [Configuring Lakekeeper](configure_lakekeeper.md) guide installs, configures, and bootstraps it, then installs the extensions each database needs. - The [Walkthrough](exploring_coldfront/index.md) guide runs the demo stack hands-on. diff --git a/docs/usage_decoupled.md b/docs/usage_decoupled.md index c61378f7..f274a5ad 100644 --- a/docs/usage_decoupled.md +++ b/docs/usage_decoupled.md @@ -1,7 +1,7 @@ # Using ColdFront in Decoupled Mode Before you begin, configure PostgreSQL, Lakekeeper, and your object store by -following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](one_time_setup.md), +following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](configure_lakekeeper.md), and [Configuring your Object Store](object_store.md); every row of a decoupled table lives in the cold tier those guides set up, from the first row. This page walks through creating a decoupled table with diff --git a/docs/usage_partitioner.md b/docs/usage_partitioner.md index d1b8718d..54ecf6aa 100644 --- a/docs/usage_partitioner.md +++ b/docs/usage_partitioner.md @@ -3,7 +3,7 @@ This mode needs only a working PostgreSQL instance, optionally in a Spock mesh; it uses no Lakekeeper, no object store, no pg_duckdb, and no coldfront extension. A database set up this way can gain the cold tier -later by following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](one_time_setup.md), +later by following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](configure_lakekeeper.md), and [Configuring your Object Store](object_store.md). This page walks through building and registering the `partitioner` binary, operating it day to day, giving a time-partitioned table a real primary key, and diff --git a/docs/usage_tiered.md b/docs/usage_tiered.md index 9758557d..fceaaf94 100644 --- a/docs/usage_tiered.md +++ b/docs/usage_tiered.md @@ -1,7 +1,7 @@ # Using ColdFront in Tiered Mode Before you begin, configure PostgreSQL, Lakekeeper, and your object store by -following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](one_time_setup.md), +following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](configure_lakekeeper.md), and [Configuring your Object Store](object_store.md); tiered mode archives aging partitions to the cold tier those guides set up. This page walks through creating a tiered table, registering it with the archiver, running diff --git a/mkdocs.yml b/mkdocs.yml index 468731bb..69ff8c8d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -69,7 +69,7 @@ nav: - Installing ColdFront: installation.md - Configuring ColdFront: - configuration.md - - Configuring Lakekeeper: one_time_setup.md + - Configuring Lakekeeper: configure_lakekeeper.md - Configuring your Object Store: object_store.md - Configuring a Distributed Deployment: distributed_setup.md From 0484f4cbcbee2ba826764a33d3d97b44474fbd43 Mon Sep 17 00:00:00 2001 From: Susan Douglas Date: Wed, 14 Oct 2026 11:30:33 -0400 Subject: [PATCH 10/10] docs: describe three modes in the introduction, format the coldfront extension name - index.md: list standalone partitioned mode as the third operating mode, say how each table gets its mode, move the SQL statements into the opening paragraph, and link the Lamport and Ricart-Agrawala algorithms. - Write the extension name as `coldfront` wherever prose names the coldfront extension. --- docs/architecture_decoupled.md | 10 +++++----- docs/architecture_guides/index.md | 24 +++++++++++------------ docs/changelog.md | 2 +- docs/configure_lakekeeper.md | 2 +- docs/distributed_setup.md | 4 ++-- docs/exploring_coldfront/index.md | 2 +- docs/index.md | 32 +++++++++++++++++++------------ docs/installation.md | 4 ++-- docs/usage_partitioner.md | 2 +- docs/walkthrough_decoupled.md | 2 +- 10 files changed, 46 insertions(+), 38 deletions(-) diff --git a/docs/architecture_decoupled.md b/docs/architecture_decoupled.md index 1658c6ba..ec84301a 100644 --- a/docs/architecture_decoupled.md +++ b/docs/architecture_decoupled.md @@ -25,10 +25,10 @@ this document, concern by concern: | `coldfront.tiered_views` row | The row is required for each managed table. | The row is required (with `is_iceberg_only = true`), and `create_iceberg_table()` registers it. | | Required at runtime | `pg_duckdb`, `coldfront`, Lakekeeper, an S3-compatible store or Azure ADLS Gen2 | `pg_duckdb`, `coldfront` (lazy catalog ATTACH, DML rewrite, and write serialization), Lakekeeper, an S3-compatible store or Azure ADLS Gen2 | -The coldfront extension provides the lazy catalog-attach glue: the C extension -hook intercepts the first query that touches a tiered view (read or write) and, -if the Iceberg catalog `ice` is not yet attached in this session, issues the -following against the GUCs `coldfront.warehouse` and +The `coldfront` extension provides the lazy catalog-attach glue: the C +extension hook intercepts the first query that touches a tiered view (read or +write) and, if the Iceberg catalog `ice` is not yet attached in this session, +issues the following against the GUCs `coldfront.warehouse` and `coldfront.lakekeeper_endpoint`: ```sql @@ -88,7 +88,7 @@ attaches the catalog and `ice.public.*` becomes available. ## Interface -With the coldfront extension loaded and the storage secret set, the wrapper +With the `coldfront` extension loaded and the storage secret set, the wrapper view supports the operations below. ### What Works diff --git a/docs/architecture_guides/index.md b/docs/architecture_guides/index.md index 3f3222f1..091f7d86 100644 --- a/docs/architecture_guides/index.md +++ b/docs/architecture_guides/index.md @@ -162,7 +162,7 @@ into a fresh instance) and the bakery's transient claim tables (`claims` / re-established a restored node serves hot reads but fails cold I/O cleanly. `ci/ops.sh` Check 4 exercises exactly this. -The Iceberg catalog ATTACH is **lazy**: the coldfront C extension hook issues +The Iceberg catalog ATTACH is **lazy**: the `coldfront` C extension hook issues `ATTACH IF NOT EXISTS` against Lakekeeper - using the cluster's `coldfront.warehouse` and `coldfront.lakekeeper_endpoint` GUCs - on the **first query that touches a tiered view** (read or write), per DuckDB cached @@ -245,7 +245,7 @@ WHERE r['ts'] < '2026-03-01'::timestamptz; ## Application Interface Applications use the transparent view exactly like a table. A -`post_parse_analyze_hook` in the coldfront extension intercepts +`post_parse_analyze_hook` in the `coldfront` extension intercepts `INSERT`/`UPDATE`/`DELETE`/`MERGE` whose target is a registered relation - resolved in `coldfront.tiered_views` by name (`schema_name`, `relname`) - and rewrites the parsed `Query` so it lands in the correct tier; cold-side writes @@ -338,7 +338,7 @@ ColdFront coordinates concurrent writes across the cluster as follows: lock acquisition (10 attempts, exponential backoff from 100 ms to 51.2 s). - Hot writes are replicated by Spock normally (standard PG DML). - Cold writes via `duckdb.raw_query()` from multiple nodes are serialized - PG-side by the **bakery protocol** in the coldfront extension - every + PG-side by the **bakery protocol** in the `coldfront` extension - every iceberg-only `INSERT`/`UPDATE`/`DELETE`/`MERGE` wraps in `coldfront._exec_iceberg_with_claim`, which holds a globally-ordered Snowflake ticket via the Spock-replicated `coldfront.claims` table and waits @@ -382,20 +382,20 @@ for the build. ### Transparent DDL via coldfront -A `ProcessUtility_hook` in the coldfront extension intercepts DDL that targets -a registered tiered table's hot heap or a decoupled table's wrapper view. A -hot heap is matched by resolving the DDL target relation to an OID and +A `ProcessUtility_hook` in the `coldfront` extension intercepts DDL that +targets a registered tiered table's hot heap or a decoupled table's wrapper +view. A hot heap is matched by resolving the DDL target relation to an OID and comparing it with the OID of the registry's `hot_table` - never by string, so it is schema-agnostic - and a view by its schema and name, the registry's key. A view the hook rebuilds keeps the owner and the table-level grants the view had. The hook rebuilds the view and updates the registry as the extension's owner, once the ownership check PostgreSQL makes for the statement has passed. -The role running the statement needs what PostgreSQL requires for the -statement itself, `USAGE` on the schema of the relation it names included, but -neither ownership of the view nor `CREATE` on its schema, and the hook's own -registry lookups and updates need no privilege on the registry. The Iceberg -change, which is cold I/O, runs as that role, so a column change also needs the -cold access `grant_app_access` gives, `SELECT` on the registry included. The +The role running the statement needs what PostgreSQL requires for the statement +itself, `USAGE` on the schema of the relation it names included, but neither +ownership of the view nor `CREATE` on its schema, and the hook's own registry +lookups and updates need no privilege on the registry. The Iceberg change, +which is cold I/O, runs as that role, so a column change also needs the cold +access `grant_app_access` gives, `SELECT` on the registry included. The following table summarizes the DDL it handles: | DDL | Behavior | diff --git a/docs/changelog.md b/docs/changelog.md index a5e4a138..8e6ab8c8 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -436,7 +436,7 @@ use. older data to Apache Iceberg on a watermark, presented to the application as a single unified view. - Decoupled mode stores a table entirely in Iceberg from the first row, with - PostgreSQL holding a thin wrapper view and the coldfront extension handling + PostgreSQL holding a thin wrapper view and the `coldfront` extension handling every data-modifying statement on that view. - Horizontal scale-out for decoupled mode across multiple PostgreSQL nodes sharing one Lakekeeper catalog and one object store, serialised by the bakery diff --git a/docs/configure_lakekeeper.md b/docs/configure_lakekeeper.md index abd9166d..84a52d19 100644 --- a/docs/configure_lakekeeper.md +++ b/docs/configure_lakekeeper.md @@ -1,7 +1,7 @@ # Configuring Lakekeeper ColdFront requires three services configured once per deployment: -PostgreSQL with the pg_duckdb and coldfront extensions, Lakekeeper, and +PostgreSQL with the pg_duckdb and `coldfront` extensions, Lakekeeper, and any object store (SeaweedFS, MinIO, AWS S3, GCS, or Azure ADLS Gen2). This guide brings up and bootstraps Lakekeeper, then installs and configures the PostgreSQL side; do this before creating any table. diff --git a/docs/distributed_setup.md b/docs/distributed_setup.md index 159359cb..6a524c80 100644 --- a/docs/distributed_setup.md +++ b/docs/distributed_setup.md @@ -120,8 +120,8 @@ SELECT coldfront.ensure_replicated(); The subscriptions set `synchronize_structure := false, synchronize_data := false` because the tables -already exist on every node from the coldfront extension, so no initial copy is -needed. +already exist on every node from the `coldfront` extension, so no initial copy +is needed. ### Timeouts diff --git a/docs/exploring_coldfront/index.md b/docs/exploring_coldfront/index.md index f7efd4bc..38a141e0 100644 --- a/docs/exploring_coldfront/index.md +++ b/docs/exploring_coldfront/index.md @@ -94,7 +94,7 @@ Setup runs before the demos begin. Setup starts the containers, waits for PostgreSQL and Lakekeeper to accept connections, and creates the Lakekeeper warehouse and namespace. The stack includes the following services: -- PostgreSQL 16, 17, or 18 with the pg_duckdb and coldfront extensions. +- PostgreSQL 16, 17, or 18 with the pg_duckdb and `coldfront` extensions. - SeaweedFS, a local S3-compatible object store standing in for a real cloud bucket. - Lakekeeper, the Iceberg REST catalog that tracks table metadata and file diff --git a/docs/index.md b/docs/index.md index 0f6d7416..27f9e35c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,8 +3,9 @@ ColdFront keeps tables in PostgreSQL and cold data in Apache Iceberg (Parquet on S3-compatible, Azure, or GCS storage). The cold tier is both readable and writable through the same SQL, with no application changes. The application -queries every table as an ordinary PostgreSQL relation, and both operating -modes present the same standard SQL surface. +queries every table as an ordinary PostgreSQL relation, with standard +`SELECT`, `INSERT`, `UPDATE`, and `DELETE`, and `MERGE` on PostgreSQL 17 and +later. !!! warning "Beta software" @@ -12,27 +13,34 @@ modes present the same standard SQL surface. production. Interfaces, on-disk formats, and behavior may change without notice, and data loss is possible. -ColdFront provides two operating modes: +ColdFront provides three operating modes: - Tiered mode keeps recent data in native PostgreSQL partitions and archives older data to Iceberg on a watermark; the application reads a single unified view, and the archiver moves rows from hot to cold on a schedule. - Decoupled mode stores the table entirely in Iceberg from the first row; - PostgreSQL holds a thin wrapper view and a registry row, and the coldfront + PostgreSQL holds a thin wrapper view and a registry row, and the `coldfront` extension handles every data-modifying statement on that view. +- Standalone partitioned mode manages time- or id-based PostgreSQL range + partitions with no cold tier; the `partitioner` binary creates upcoming + partitions and detaches and drops partitions past retention, on stock + PostgreSQL with no DuckDB or Iceberg. -Both modes coexist within one database, and you choose the mode per table at -creation time. The SQL surface is identical for both modes: standard `SELECT`, -`INSERT`, `UPDATE`, and `DELETE` against the relation, and `MERGE` on -PostgreSQL 17 and later. +Tables in different modes can share one database, and you choose the mode for +each table: tiered when you register an existing partitioned table with a hot +period, standalone partitioned when you register one without a hot period, +and decoupled when you create or adopt an Iceberg table. Decoupled mode scales out horizontally across many PostgreSQL nodes that share one Lakekeeper catalog and one object store. The bakery protocol in the -coldfront extension serializes Iceberg commits on the PostgreSQL side using +`coldfront` extension serializes Iceberg commits on the PostgreSQL side using Spock-replicated Snowflake tickets, so concurrent writers never collide at the -catalog. The protocol implements Lamport mutual exclusion with the -Ricart-Agrawala deferred-reply optimization, and the -[formal model](formal/README.md) verifies its safety with TLA+. +catalog. The protocol implements +[Lamport mutual exclusion](https://en.wikipedia.org/wiki/Lamport%27s_distributed_mutual_exclusion_algorithm) +with the +[Ricart-Agrawala](https://en.wikipedia.org/wiki/Ricart%E2%80%93Agrawala_algorithm) +deferred-reply optimization, and the [formal model](formal/README.md) verifies its +safety with TLA+. ## How It Works diff --git a/docs/installation.md b/docs/installation.md index aaae39ae..640775f0 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -153,7 +153,7 @@ expensive, stable compiles (pg_duckdb on DuckDB 1.5.4 + the patched duckdb-iceberg) live in the prebuilt **base**, published to `ghcr.io/pgedge/coldfront-duckdb-base:pg{16,17,18}`; the **app** build ([`docker/Dockerfile.duckdb15`](https://github.com/pgEdge/ColdFront/blob/main/docker/Dockerfile.duckdb15)) -`FROM`s it and compiles the coldfront extension in seconds. If you build the +`FROM`s it and compiles the `coldfront` extension in seconds. If you build the base yourself (step 1) the app layer `FROM`s your local image; otherwise it `FROM`s the published `ghcr.io/pgedge/coldfront-duckdb-base:pg`. Rebuild the published base via the @@ -264,7 +264,7 @@ for the full tier-and-verify journey. ## Building ColdFront on Bare Metal -The coldfront extension is a standard PGXS C extension: +The `coldfront` extension is a standard PGXS C extension: ```bash cd extension/coldfront diff --git a/docs/usage_partitioner.md b/docs/usage_partitioner.md index 54ecf6aa..cf351c7f 100644 --- a/docs/usage_partitioner.md +++ b/docs/usage_partitioner.md @@ -2,7 +2,7 @@ This mode needs only a working PostgreSQL instance, optionally in a Spock mesh; it uses no Lakekeeper, no object store, no pg_duckdb, and no -coldfront extension. A database set up this way can gain the cold tier +`coldfront` extension. A database set up this way can gain the cold tier later by following [Configuring ColdFront](configuration.md), [Configuring Lakekeeper](configure_lakekeeper.md), and [Configuring your Object Store](object_store.md). This page walks through building and registering the `partitioner` binary, operating it diff --git a/docs/walkthrough_decoupled.md b/docs/walkthrough_decoupled.md index e58f67e7..04f3d2c5 100644 --- a/docs/walkthrough_decoupled.md +++ b/docs/walkthrough_decoupled.md @@ -56,7 +56,7 @@ DELETE FROM events_lake WHERE id = 2; SELECT count(*) AS after_delete FROM events_lake; ``` -All four DML operations reach the Iceberg table transparently. The coldfront +All four DML operations reach the Iceberg table transparently. The `coldfront` extension intercepts each statement on the view and rewrites it to the Iceberg path via pg_duckdb.