PMM + SEP feature-build harness

sidecar/pmm-fb/.gitignore

@@ -0,0 +1,7 @@
+# .env is generated by bootstrap.sh. settings.yaml and pmm.conf are leftovers
+# from mechanisms this harness has retired — an existing checkout still holds
+# them, and both carry deployment secrets in cleartext, so they stay ignored to
+# keep an accidental commit off the table until the operator deletes them.
+.env
+settings.yaml
+pmm.conf

sidecar/pmm-fb/Containerfile.mysql

@@ -0,0 +1,52 @@
+# Image for the sep-mysql target node; see mysql-target.md.
+ARG PMM_FB_TAG
+FROM docker.io/perconalab/pmm-client-fb:${PMM_FB_TAG} AS pmmclient
+
+FROM docker.io/library/oraclelinux:9
+
+# The percona-release RPM is signed by a key no repo has taught rpm about yet,
+# so this one install skips the check; it then imports the keys that verify
+# everything below. Same ps84lts + pt choice as
+# build/ansible/ansible-mysql-el-test-instance.yaml, plus XtraBackup.
+RUN dnf install -y --nogpgcheck https://repo.percona.com/yum/percona-release-latest.noarch.rpm \
+    && percona-release setup -y ps84lts \
+    && percona-release enable pt \
+    && percona-release enable pxb-84-lts
+
+# mydumper is not in the Percona repos; it ships its own
+RUN rpm --import 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x79EA15C0E82E34BA' \
+    && printf '%s\n' '[mydumper]' 'name=MyDumper' \
+        'baseurl=https://mydumper.github.io/mydumper/repo/yum/$releasever' \
+        'enabled=1' 'gpgcheck=1' > /etc/yum.repos.d/mydumper.repo
+
+# python3-pip and gzip are for the dispatched run-python task, which builds a
+# venv and gunzips its result; percona-server-server, percona-toolkit and
+# python3-PyMySQL mirror the ansible playbook's set. zstd, lz4, gnupg2 and rsync
+# are named though they already arrive transitively, so a shift upstream cannot
+# quietly drop a compression, encryption or upload option the forms offer.
+RUN dnf install -y \
+        percona-server-server percona-toolkit python3-PyMySQL \
+        percona-xtrabackup-84 mydumper \
+        python3-pip gzip tar hostname procps-ng iproute \
+        zstd lz4 gnupg2 rsync \
+    && dnf clean all
+
+# Static Go binaries, so the feature build's own tree runs unchanged on OL9
+COPY --from=pmmclient /usr/local/percona/pmm /usr/local/percona/pmm
+ENV PATH="/usr/local/percona/pmm/bin:${PATH}"
+
+# Baked at build time so first boot needs no download
+ADD https://github.com/datacharmer/test_db/releases/download/v1.0.7/test_db-1.0.7.tar.gz /tmp/
+RUN tar -xzf /tmp/test_db-1.0.7.tar.gz -C /opt && rm -f /tmp/test_db-1.0.7.tar.gz
+
+COPY mysql-entrypoint.sh /usr/local/bin/mysql-entrypoint.sh
+
+EXPOSE 3306
+
+# Health is MySQL readiness only, deliberately: folding PMM registration in
+# would make a registration hiccup read as "the database is down". The start
+# period covers --initialize-insecure plus the employees import.
+HEALTHCHECK --interval=10s --timeout=5s --start-period=180s --retries=12 \
+    CMD mysqladmin --defaults-file=/root/.my.cnf --protocol=socket ping || exit 1
+
+ENTRYPOINT ["/usr/local/bin/mysql-entrypoint.sh"]

sidecar/pmm-fb/README.md

@@ -0,0 +1,222 @@
+# PMM + SEP feature-build harness
+
+Compose topology pairing the PMM feature build (SEP frontend, PostgreSQL
+exposure, secret provisioning and the native `/sep` proxy,
+[Percona-Lab/pmm-submodules#4500] = percona/pmm branch `PMM-15216`, PRs
+[percona/pmm#5653] + [percona/pmm#5700]) with the app-restricted SEP side-car:
+supervisord running the three APIs + Celery worker/beat + bundled Valkey,
+shipping only the `inventory`, `mysql_backups` and `atw` apps. The snippets
+management app is not shipped — the builtin snippet library is ingested and
+auto-approved at boot (SEP-1627) so atw can execute it, with no periodic or
+manual re-sync.
+
+## Which image to pin
+
+The restricted image is built on `main`, not on this branch, and the build emits
+a **single** artifact under a suffix-less tag — the `-sidecar` / `-embedded`
+variants no longer exist. `main`'s `image` target builds it with
+`SEP_RESTRICT_APPS=1`, deriving the shipped app set from `sidecar/settings.yaml`'s
+`SEP.APPS`, and publishes it as `percona/percona-sep:<commit-sha>`.
+
+`compose.yaml`'s `sep-sidecar` service therefore pins a main-line commit SHA.
+Repin to a newer one by picking a tag published from `main` — the tag list on
+Docker Hub is ordered by publish date.
+
+Two properties of the pinned image are load-bearing, and both are worth checking
+on the **artifact** rather than on the commit that built it:
+
+```bash
+TAG=<the tag you are pinning>
+
+# It must read secrets from a directory (SECRETS_DIR); expect a non-zero count.
+docker run --rm --entrypoint sh docker.io/percona/percona-sep:$TAG \
+  -c 'grep -c SECRETS_DIR /home/sep/app/settings-env.sh'
+
+# It must carry a HEALTHCHECK; expect a Test naming healthcheck.sh.
+skopeo inspect --config --raw docker://docker.io/percona/percona-sep:$TAG \
+  | python3 -c 'import json,sys; print(json.load(sys.stdin)["config"]["Healthcheck"])'
+```
+
+`skopeo inspect --config` **without** `--raw` normalizes to OCI, whose
+image-config spec has no `Healthcheck` field, so it reports `null` for every SEP
+tag including ones that carry the instruction. Only the raw config blob answers
+the question. Every side-car recipe builds in docker format for the same reason:
+OCI discards the instruction. The 150 s start period keeps the side-car out of
+`unhealthy` while PMM provisioning and SEP migrations finish.
+
+The pmm-server pin is subject to its own constraint — see
+[Caveats](#caveats).
+
+[Percona-Lab/pmm-submodules#4500]: https://github.com/Percona-Lab/pmm-submodules/pull/4500
+[percona/pmm#5653]: https://github.com/percona/pmm/pull/5653
+[percona/pmm#5700]: https://github.com/percona/pmm/pull/5700
+
+## Bring-up
+
+```bash
+git clone -b pmm git@github.com:percona/SEP.git  # already cloned: git checkout pmm && git pull
+cd SEP/sidecar/pmm-fb
+
+./bootstrap.sh                                # generate .env
+docker compose --profile mysql up -d --build  # or: podman compose ...
+```
+
+`--profile mysql` adds the `sep-mysql` task-execution target — a MySQL server,
+a PMM Client and the employees seed in one container, documented in
+[mysql-target.md](mysql-target.md). Omitting it is the supported fast path —
+you get the same two services this harness has always had, and auth or UI work
+does not pay for a MySQL build:
+
+```bash
+docker compose up -d        # pmm-server + sep-sidecar only
+```
+
+Note that `docker compose build` obeys the profile too, and reports `No
+services to build` rather than an error when you forget it.
+
+First boot takes a couple of minutes. The side-car does not start until
+pmm-server reports healthy, which now means "SEP's secrets are published" as
+well as "PMM is up" — so expect `sep-sidecar` to sit in `Created` for a while
+before it runs. `sep-mysql` initialises a datadir and imports the employees
+dataset on its first boot; the import runs for several minutes. Its healthcheck
+reports `healthy` as soon as MySQL answers — about a minute in, while the import
+is still running — because health here means "the database is up" and nothing
+more. Watch `docker compose logs -f sep-mysql` for `Imported the employees seed
+dataset` before expecting a backup to have data to copy. Then:
+
+- PMM UI: https://127.0.0.1:8443 (admin / admin) — SEP's pages are part of it
+- SEP's API through PMM's nginx: https://127.0.0.1:8443/sep/api/… . The side-car
+  serves no UI of its own, so `/sep/` itself answers `{"detail":"Not Found"}`;
+  that is the embedded topology working, not a routing fault.
+- SEP APIs directly: http://127.0.0.1:9000-9002 (sep / inventory / tasks)
+
+There is no token-minting step. PMM mints the Grafana service-account token
+itself and publishes it to the side-car, so Grafana-backed sign-in, the PMM
+syncer and task-lifecycle PMM annotations all work on a first boot.
+
+To probe the API by hand, exchange your PMM browser session for a short-lived
+SEP bearer rather than looking for a static token — an unauthenticated request
+is answered `401`, because nothing injects a bearer server-side any more:
+
+```bash
+curl -sk -c /tmp/pmm-cookies -X POST https://127.0.0.1:8443/graph/login \
+  -H 'Content-Type: application/json' -d '{"user":"admin","password":"admin"}'
+TOKEN=$(curl -sk -b /tmp/pmm-cookies -X POST \
+  https://127.0.0.1:8443/sep/api/oauth/session/exchange | python3 -c \
+  'import json,sys; print(json.load(sys.stdin)["access_token"])')
+curl -sk -H "Authorization: Bearer $TOKEN" https://127.0.0.1:8443/sep/api/apps/
+```
+
+## How the pieces connect
+
+- `bootstrap.sh` generates the gitignored `.env`, which now holds only the
+  passwords PMM cannot generate for itself: `SEP_PG_PASSWORD` (the password PMM
+  provisions SEP's PostgreSQL role with) and the three `sep-mysql` passwords.
+  Nothing secret is committed; re-running keeps an existing `.env` and appends
+  any slot it predates.
+- **PMM owns every SEP deployment secret.** With `PMM_ENABLE_SEP=1` it writes
+  six files into the `pmm-sep-secrets` volume, which pmm-server mounts at
+  `/srv/sep-secrets` and the side-car mounts read-only at `/run/secrets/sep`.
+  `SECRETS_DIR` points SEP at that directory and it reads each file as the
+  canonical setting the filename names:
+
+  | File | Written by | When |
+  |---|---|---|
+  | `SECRET_KEY` | the entrypoint | seconds after container start |
+  | `SEP__DATABASE__PASSWORD`, `INVENTORY__DATABASE__PASSWORD`, `TASKS__DATABASE__PASSWORD` | the entrypoint | seconds after container start |
+  | `AUTH__PROVIDER__GRAFANA__SERVICE_ACCOUNT_TOKEN`, `PMM__API_KEY` | `grafana-sep`, a supervisord one-shot | after Grafana's first-boot migration |
+
+  None of the six reaches the side-car as environment, so none appears in
+  `docker inspect` or in the process environment. `SEP_NOMAD_ENDPOINT` is the
+  one credential that does: PMM's stock `admin:admin`, a published default
+  rather than a provisioned secret.
+- **The two-stage write is why `sep-sidecar` waits on
+  `condition: service_healthy`.** SEP builds its settings once, at process
+  start, and never re-reads them: a side-car released between the two stages
+  would come up with Grafana authentication permanently inert. pmm-server's
+  healthcheck gates on the current provisioning run having published both token
+  files, so health means the side-car can safely read everything.
+- **Group 0, not a matching uid.** The side-car runs as uid/gid 1001 and PMM
+  writes the files mode 0640 owned by group `root` under a setgid `02770`
+  directory. `group_add: ["0"]` is what makes them readable, and it keeps
+  working if PMM's runtime uid changes.
+- `PMM_ENABLE_SEP=1` + `SEP_PG_PASSWORD` also make pmm-server's entrypoint
+  expose its embedded PostgreSQL on the compose network and provision the
+  low-privilege `sep` role owning the `sep` database (percona/pmm#5700).
+  Nothing is published on the host. `PMM_ENABLE_NOMAD=1` + `PMM_PUBLIC_ADDRESS`
+  start PMM's embedded Nomad, which SEP task execution dispatches through
+  (Nomad silently stays down if the public address is unset).
+- All three SEP services **and** the Celery beat store share that single
+  `sep` database — the exposure provisions exactly one db/role, and the three
+  Alembic tracks use distinct version tables with non-colliding table names.
+  The side-car's migration one-shots wait for `pmm-server:5432` and migrate
+  on first boot.
+- **PMM proxies SEP natively.** pmm-server ships its own `location /sep/`
+  block; `PMM_SEP_ADDRESS: sep-sidecar:9000` points it at the side-car, and the
+  baked settings profile sets `SEP.ROOT_PATH: /sep` to match. The value is a
+  bare `host:port` — a scheme or a trailing path is rejected, and the failure
+  surfaces as a routing fault rather than a config error. Nothing is bind-mounted
+  over PMM's nginx config.
+- **Neither container needs a fixed address.** PMM's SEP location proxies to a
+  *variable* under a location-scoped `resolver ... valid=10s`, which defers
+  resolution to request time, so the config loads with the side-car absent and
+  picks up a new address within the TTL when it is recreated. `sep-mysql`
+  (`172.28.9.40`) is still fixed, for an unrelated reason: SEP never updates an
+  existing inventory node's address, and the Mydumper path connects to exactly
+  that address, so an address that moved across recreates would leave SEP
+  pointing at a stale one.
+- `BASE_URL` is `http://sep-sidecar:9000/sep` — a compose service name, which
+  `../README.md` explicitly tells you not to use because it defines `BASE_URL`
+  as the side-car's address *as reachable from Nomad task executors*. That
+  prescription is for a real deployment, where PMM Client nodes run their own
+  Nomad clients and resolve no compose name. This harness has no PMM Client
+  nodes: its only executor is the Nomad client inside `pmm-server`, which shares
+  that container's network namespace and resolves `sep-sidecar` through Docker's
+  embedded DNS. The consequence is that **the harness does not exercise the
+  production artifact-download path**. The `/sep` suffix is required either way
+  — download URLs are joined onto `BASE_URL`'s path rather than replacing it.
+
+## Caveats
+
+- **The pmm-server pin must carry the SEP readiness gate** (PMM-15331, first in
+  `PR-4500-882b6ba`), which widens the start period to 720 s and holds the
+  container unhealthy until SEP provisioning completes. Against an older pin,
+  `condition: service_healthy` stops being a gate and becomes a bring-up
+  failure: the old 25 s start period flips pmm-server to `unhealthy` at ~37 s
+  while a cold start needs ~90 s to first pass `readyz`, and compose aborts the
+  dependent. If you ever repin backwards, revert `sep-sidecar`'s condition to
+  `service_started` in the same edit.
+- **`PMM_ENABLE_SEP` unset takes SEP down**, it does not degrade it. All six
+  files are removed on pmm-server's next start, and the side-car exits 1 with a
+  single actionable `SECRET_KEY is required` rather than coming up
+  half-configured. A side-car already running is unaffected until it restarts,
+  because its settings are in memory.
+- Both ports 8443 and 9000-9002 bind to loopback only. `sep-mysql` publishes
+  nothing; it is reachable only on the compose network.
+- Under **rootless podman**, `group_add: ["0"]` maps through the user namespace
+  differently and may need `--group-add keep-groups`. If the side-car exits
+  reporting it cannot read `/run/secrets/sep`, that is the first thing to check.
+- Rotating `SEP_PG_PASSWORD` takes two restarts, in order: pmm-server first, so
+  it moves the database password and rewrites the files, then the side-car,
+  which reads those files only at process start.
+- `sep-mysql` runs `privileged: true` with `cgroup: host` and a read-write
+  `/sys/fs/cgroup` mount. A containerised Nomad client needs it — the
+  fingerprinter reads cgroups and `raw_exec` places tasks into cgroups the
+  client creates — but it is a harness-only concession, not something the
+  side-car topology does.
+- The settings mount is gone, so editing a rendered YAML file is no longer a
+  way to try something out. Partial overrides are environment-variable-only; a
+  full override means bind-mounting over `/home/sep/app/settings.yaml`, which
+  replaces the baked profile wholesale. Add an ignore rule for any local
+  filename you mount that is not already in `.gitignore` — such a file holds the
+  deployment's secrets in cleartext.
+- **Upgrading a harness bootstrapped before PMM took over the secrets:** the
+  supported path is `docker compose down -v`, then `./bootstrap.sh` and a fresh
+  `up -d`. Dropping the volumes is what gets the secrets volume initialised with
+  PMM's `02770` permissions — Docker only initialises a *new* named volume from
+  the image, never an existing one. Then delete the leftover `pmm.conf` and
+  `settings.yaml`: both are inert now, both hold secrets in cleartext, and both
+  stay gitignored only to keep them out of a commit until you do. The retired
+  `SEP_SECRET_KEY`, `SEP_INTERNAL_TOKEN` and `SEP_GRAFANA_TOKEN` slots in an
+  existing `.env` are inert too — nothing reads them — but delete them for the
+  same reason.

sidecar/pmm-fb/bootstrap.sh

@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+# Generate this harness's per-deployment secrets (.env). Idempotent: an existing
+# .env is kept, and any slot it predates is appended. PMM owns every remaining
+# SEP secret — it writes them into the pmm-sep-secrets volume, which the side-car
+# reads as files; nothing secret reaches the side-car as environment.
+
+set -o nounset
+set -o pipefail
+
+if [[ ${DEBUG:-0} == "1" ]]; then
+    set -o xtrace
+fi
+
+error() { printf '✗ %s\n' "$*" >&2; }
+success() { printf '✓ %s\n' "$*" >&2; }
+info() { printf 'ℹ %s\n' "$*" >&2; }
+
+# Runs a command with xtrace suspended, restoring it on return (local -).
+# DEBUG=1 traces every expansion, and the secrets below reach the wrapped
+# commands both as arguments and as sourced assignments — tracing those would
+# print every per-deployment secret.
+without_xtrace() {
+    local -
+    set +o xtrace
+    "$@"
+}
+
+need_cmd() {
+    command -v "$1" > /dev/null 2>&1 || {
+        error "Missing required command: $1"
+        exit 2
+    }
+}
+
+need_cmd openssl
+need_cmd grep
+
+usage() {
+    cat << 'EOF'
+bootstrap.sh
+
+Generate .env (the PostgreSQL password PMM provisions SEP's role with, and the
+three sep-mysql passwords) next to this script if it does not exist. An existing
+.env is kept and any slot it predates is appended. Run before the first
+`docker compose up -d`, and re-run after rotating a secret.
+
+Options:
+  -h, --help    Show this help
+EOF
+    exit "${1:-0}"
+}
+
+[[ $# -eq 0 ]] || case "$1" in
+    -h | --help) usage 0 ;;
+    *)
+        error "Unknown argument: $1"
+        usage 2
+        ;;
+esac
+
+script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "${script_dir}" || exit 1
+
+ensure_slot() {
+    local name="$1" value="$2"
+    grep -q "^${name}=" .env && return 0
+    # A hand-edited .env may lack the trailing newline the append needs
+    [[ -z $(tail -c1 .env) ]] || printf '\n' >> .env
+    printf '%s=%s\n' "${name}" "${value}" >> .env
+    success "Added the ${name} slot to the existing .env"
+}
+
+# Wrapped by without_xtrace at the call site: the generated passwords reach
+# ensure_slot as arguments
+seed_missing_slots() {
+    local slot
+    for slot in SEP_MYSQL_ROOT_PASSWORD SEP_MYSQL_BACKUP_PASSWORD SEP_MYSQL_PMM_PASSWORD; do
+        ensure_slot "${slot}" "$(openssl rand -hex 16)"
+    done
+}
+
+if [[ -f .env ]]; then
+    info "Keeping existing .env"
+    chmod 600 .env
+    without_xtrace seed_missing_slots
+else
+    cat > .env << EOF
+# Generated by bootstrap.sh — per-deployment feature-build secrets (gitignored)
+SEP_PG_PASSWORD=$(openssl rand -hex 16)
+# Rotating these after first boot needs the sep-mysql-data volume dropped — the
+# datadir keeps the original passwords
+SEP_MYSQL_ROOT_PASSWORD=$(openssl rand -hex 16)
+SEP_MYSQL_BACKUP_PASSWORD=$(openssl rand -hex 16)
+SEP_MYSQL_PMM_PASSWORD=$(openssl rand -hex 16)
+EOF
+    chmod 600 .env
+    success "Generated .env"
+fi
+
+# Wrapped rather than run at top level: sourcing traces every assignment in
+# .env, and ${VAR:?} puts the value itself into the trace line
+load_env() {
+    # shellcheck disable=SC1091
+    source .env
+    : "${SEP_PG_PASSWORD:?missing in .env}"
+    : "${SEP_MYSQL_ROOT_PASSWORD:?missing in .env}" "${SEP_MYSQL_BACKUP_PASSWORD:?missing in .env}" \
+        "${SEP_MYSQL_PMM_PASSWORD:?missing in .env}"
+}
+
+without_xtrace load_env
+
+info "Next: docker compose --profile mysql up -d --build   (or: podman compose ...)"
+info "Omit --profile mysql to bring up just pmm-server and sep-sidecar"