Add Baseshift extension

[whummer]

Summary

Adds an extension for Baseshift, which creates masked, writable clones of production databases for development. The extension runs Baseshift clones (Docker snapshots of a Dub) next to LocalStack, so app code running in LocalStack can work against realistic, anonymized data.

• Default clone: set BASESHIFT_IMAGE and the clone starts once LocalStack is ready
• Clones API: POST/GET/DELETE http://baseshift.localhost.localstack.cloud:4566/clones to start and stop clones on demand (e.g. one per PR), including from images in the LocalStack ECR registry
• Connectivity: PostgreSQL clones are proxied through the gateway port 4566 (protocol handshake detection, like the ParadeDB extension) and published on host ports. MySQL clones only get a host port, since MySQL is server-speaks-first and can't be detected on the shared port
• Config passthrough: BASESHIFT_ENCRYPTION_PASSWORD maps to the clone's PASSWORD, and BASESHIFT_CLONE_<NAME> vars are passed to clones as <NAME>

Architecture

flowchart LR
    subgraph source["1 · Production data"]
        rds[("LocalStack RDS<br/>PostgreSQL with PII")]
    end

    cloud["Baseshift Cloud (SaaS)<br/>control plane"]

    subgraph baseshift["2 · Baseshift self-hosted components"]
        direction TB
        connector["Connector<br/>masking policy"]
        repserver["Replication server<br/>masked replica, snapshots"]
        connector -- "masked data" --> repserver
    end

    cloud -. "manages" .-> baseshift

    subgraph registry["3 · Snapshot registry"]
        ecr[("LocalStack ECR<br/>Docker snapshot images")]
    end

    subgraph clones["4 · Local clones (this extension)"]
        direction TB
        extension["Baseshift extension<br/>clones API"]
        clone1[("Clone 'default'<br/>:5432")]
        clone2[("Clone 'pr-123'<br/>:15432")]
        extension -- "start / stop" --> clone1
        extension -- "start / stop" --> clone2
    end

    gateway["LocalStack gateway :4566<br/>PostgreSQL routing"]

    subgraph consumers["5 · Consumers of masked data"]
        direction TB
        app["App code<br/>Lambda, ECS, ..."]
        dev["Developer / CI<br/>psql, IDE, tests"]
        subgraph analytics["Analytics pipeline"]
            direction LR
            elt["ELT job"] -- "CSV" --> s3[("LocalStack S3")] -- "COPY INTO" --> snowflake[("LocalStack<br/>Snowflake")]
        end
    end

    rds -- "replicate" --> connector
    repserver -- "push" --> ecr
    ecr -- "pull" --> extension
    clone1 --> gateway
    gateway -- "SQL" --> app
    gateway -- "SQL" --> dev
    gateway -- "SQL" --> elt
    clone2 -. "SQL via host port" .-> dev

Loading

Data flows left to right: Baseshift masks the production data (stage 2) before it ends up in snapshot images, so everything downstream of the registry (clones, apps, the analytics pipeline) only sees masked data. In the demo, stage 2 is currently a stand-in script, until we have access to the Baseshift replication server image.

Demo

baseshift/demo/ has an end-to-end flow, all in one LocalStack container (Snowflake emulator image, which also serves RDS/ECR/S3):

RDS Postgres with PII (logical replication enabled) → masked snapshot image in LocalStack ECR → clone via the extension → export to S3 → COPY INTO Snowflake → side-by-side view of the same record at each stage, with a check that no raw PII leaked.

The snapshot step is a stand-in for the Baseshift replication server for now (its image is private). demo/baseshift-selfhosted/ has Helm values for LocalStack (validated against the Baseshift chart schema) and a docker compose translation of the chart, for when we get access. Those are untested.

Testing

• CI matrix: emulator aws and snowflake × image tag latest and dev, using lstk, with postgres:17 as a stand-in clone image (real clone images are private per Baseshift customer)
• All jobs run the 8 integration tests: queries via gateway and host port, writes, concurrent connections, clones API (incl. a clone from an image pushed to LocalStack ECR, failed starts, validation errors)
• The snowflake jobs additionally run the full end-to-end demo (make demo), which fails if any raw PII value reaches the clone or Snowflake

Notes

• LocalStack RDS doesn't apply the rds.logical_replication parameter, so the demo sets wal_level=logical directly and reboots (the master user is a superuser in LocalStack)
• Running together with another extension that serves Postgres on port 4566 (e.g. ParadeDB) isn't supported, both claim the same handshakes
• Not yet added to CODEOWNERS

🤖 Generated with Claude Code

[whummer]

Commands for testing this end to end directly from the branch, no checkout needed. Requires Docker, lstk, psql, and LOCALSTACK_AUTH_TOKEN.

A plain postgres:17 image serves as a stand-in for a Baseshift clone image here (real clone images are private per Baseshift customer). lstk has no extension commands, so the extension is installed at startup via EXTENSION_AUTO_INSTALL, from the branch archive (git+https:// installs currently fail with the latest image due to the dulwich git shim).

# 1. Start LocalStack with the extension installed from the branch, and a default clone
LOCALSTACK_EXTENSION_AUTO_INSTALL="localstack-baseshift @ https://github.com/localstack/localstack-extensions/archive/refs/heads/add-baseshift-extension.tar.gz#subdirectory=baseshift" \
LOCALSTACK_BASESHIFT_IMAGE=postgres:17 \
LOCALSTACK_BASESHIFT_CLONE_POSTGRES_HOST_AUTH_METHOD=trust \
  lstk start

# 2. Check the clone status (starts in the background once LocalStack is ready)
curl -s http://baseshift.localhost.localstack.cloud:4566/clones

# 3. Query the default clone, through the LocalStack gateway and on the host port
psql -h localhost.localstack.cloud -p 4566 -U postgres -c "SELECT version()"
psql -h localhost -p 5432 -U postgres -c "SELECT version()"

# 4. Push a "snapshot image" to LocalStack ECR, and start a second clone from it via the clones API
REPO=000000000000.dkr.ecr.us-east-1.localhost.localstack.cloud:4566/baseshift/my-dub
lstk aws ecr create-repository --repository-name baseshift/my-dub
docker tag postgres:17 "${REPO}:latest" && docker push "${REPO}:latest"
docker rmi "${REPO}:latest"   # make sure the extension pulls it from the registry

curl -s -X POST http://baseshift.localhost.localstack.cloud:4566/clones \
  -d "{\"name\": \"pr-123\", \"image\": \"${REPO}:latest\", \"env\": {\"POSTGRES_HOST_AUTH_METHOD\": \"trust\"}}"
curl -s http://baseshift.localhost.localstack.cloud:4566/clones/pr-123   # wait for "status": "running", "hostPort": 15432

psql -h localhost -p 15432 -U postgres -c "SELECT 'hello from pr-123'"

# 5. Stop the second clone, and clean up
curl -s -X DELETE http://baseshift.localhost.localstack.cloud:4566/clones/pr-123
lstk stop

Note: use "${REPO}:latest" with braces as above - in zsh, $REPO:latest is interpreted as the :l modifier.

Optional: the end-to-end demo (RDS source with PII → masked snapshot in ECR → clone → S3 → Snowflake). Requires a license that includes the Snowflake emulator, and runs in the Snowflake emulator image, which also provides RDS, ECR and S3:

LOCALSTACK_EXTENSION_AUTO_INSTALL="localstack-baseshift @ https://github.com/localstack/localstack-extensions/archive/refs/heads/add-baseshift-extension.tar.gz#subdirectory=baseshift" \
  lstk start --type snowflake

mkdir baseshift-demo && cd baseshift-demo
curl -sO https://raw.githubusercontent.com/localstack/localstack-extensions/add-baseshift-extension/baseshift/demo/demo.py
curl -sO https://raw.githubusercontent.com/localstack/localstack-extensions/add-baseshift-extension/baseshift/demo/requirements.txt
python3 -m venv .venv && .venv/bin/pip install -q -r requirements.txt
.venv/bin/python demo.py all   # or step by step: source, snapshot, clone, pipeline, compare

lstk stop

The last step prints the same customer record at every stage (raw in RDS, masked in the clone and in Snowflake), and checks that no raw PII leaked downstream.

Note: lstk start --type snowflake records snowflake as the emulator type in your lstk config. Use lstk start --type aws next time to switch back.