Scoped feature flags: per-cluster and per-replica LaunchDarkly overrides#36959
Draft
antiguru wants to merge 14 commits into
Draft
Scoped feature flags: per-cluster and per-replica LaunchDarkly overrides#36959antiguru wants to merge 14 commits into
antiguru wants to merge 14 commits into
Conversation
Implements the foundational layer of the scoped feature flags design (per-cluster and per-replica LaunchDarkly overrides), corresponding to MVP step 0 plus the LaunchDarkly-facing context-kind change and the size-family prerequisite. - ParameterScope (Environment / Cluster / Replica) is declared on the dyncfg `Config` builder and on `VarDefinition`, surfaced through the `Var` trait and carried from dyncfg entries into the system var view. Defaults to `Environment`, so existing synced parameters are unchanged. - Annotates the two use cases' parameters: the optimizer feature flags reachable via `CREATE CLUSTER ... FEATURES` as `Cluster`, and `lgalloc` / the column-paged batcher pager / LZ4 as `Replica`. - Adds a `family` field to `ReplicaAllocation` (the per-size entry in the cluster replica size map) with a `family()` accessor that falls back to `cc` / `legacy` from `is_cc`. This is the `replica_size_family` source for replica-local evaluation. - Adds `cluster` and `replica` LaunchDarkly context kinds and a `scoped_ld_ctx` entry point that composes them with the existing environment/organization/build multi-context: cluster-coherent resolution is replica-free, replica-local resolution also carries the owning cluster. Both expose dual id/name attributes so an LD rule can pin an incarnation (by id) or target a durable role (by name/family). Covered by unit tests. The per-cluster/replica evaluation loop, in-memory override maps, controller/optimizer resolution, and introspection relations (MVP steps 1-3) build on this layer and are left for follow-ups.
Adds the controller-side capability for replica-local scoped feature flags (step 1 of the scoped feature flags design): the compute controller can carry a per-replica dyncfg override that is merged into the `UpdateConfiguration` command each replica receives. - `Instance` holds a sparse `BTreeMap<ReplicaId, ConfigUpdates>`. The command-history records the un-specialized base command; the override is merged per replica both on the broadcast send path and on the history replay used to hydrate new replicas, so a replica's override survives reconnects. - `ComputeController::update_replica_dyncfg_overrides` replaces the overrides per instance; instances absent from the map have theirs cleared (reverting those replicas to the environment-wide config). Inert by default: with an empty map every replica receives the unmodified environment-wide configuration, so behavior is unchanged until the environmentd-side wiring (next commit) populates it.
Completes step 1 of the scoped feature flags design: the system-parameter sync loop now evaluates replica-local parameters per live replica and reconciles the result into the compute controller's per-replica dyncfg override layer (added in the previous commit). - The sync loop takes a catalog snapshot each tick, builds a `cluster` + `replica` evaluation context for every live managed replica (carrying the replica's size and size family from its `ReplicaAllocation`), and asks the LaunchDarkly frontend to evaluate the `Replica`-scoped synced parameters. Only values that differ from the environment-wide value are kept, so the resulting maps stay sparse. With no `Replica`-scoped flags the per-replica evaluation is skipped entirely. - `SystemParameterFrontend::pull_replica_overrides` performs the per- replica LD evaluation; the file client returns nothing (replicas fall back to the environment-wide value, matching today's outage behavior). - The string overrides are parsed into typed `ConfigUpdates` using the catalog's dyncfg definitions (`ConfigEntry::parse_val`, new in mz-dyncfg) and delivered to the coordinator via a new `Command::UpdateReplicaScopedConfig`. The coordinator pushes them into the compute controller and re-pushes the environment-wide compute configuration so existing replicas observe the new values. This realizes use case 2: e.g. an LD rule on `replica_size_family = "legacy"` toggles `lgalloc` only on legacy replicas, with no SQL, catalog, or clusterd changes.
Re-architects the replica-local wiring to be persistence-ready, following the design update that makes scoped overrides durable (a catalog collection keyed by object id with an in-memory working copy; the sync loop is the sole writer). Previously the sync loop converted overrides to typed `ConfigUpdates` and the coordinator forwarded them straight to the controller, with no authoritative working copy anywhere. That cannot be persisted or loaded on startup. Now: - The coordinator holds the working copy as `ScopedParameters` (raw string values keyed by object id), the shape the durable collection will back and that the cluster-coherent path will reuse. - The sync loop sends the *complete* desired state each tick via `Command::UpdateScopedSystemParameters`, so the coordinator applies removals by replacing its working copy. - `Coordinator::reconcile_scoped_system_parameters` stores the working copy and resolves the `replica` layer into the compute controller's per-replica dyncfg overrides (parsing strings to typed values against the catalog's dyncfg definitions, routing each replica to its owning cluster), then re-pushes the environment-wide compute config. No behavior change yet for an empty working copy. The durable catalog collection, startup load, and lazy GC are the follow-up (step 2).
Step 2 of the scoped feature flags design is durable persistence of the scoped overrides (a catalog collection keyed by object id, so values survive an environmentd restart / LD outage and fall back to env-wide only on a cold cache). Completing the durable collection requires the catalog migration tooling and test suite (a CATALOG_VERSION bump, an objects_v86 snapshot, a no-op migration, and regenerated encoding golden files), which can't run in this environment. Rather than land a half-verifiable catalog migration, this commit adds: - `ReplicaSystemConfiguration` (the in-memory shape of the future `replica_system_configurations` collection) and inert stub `Transaction` accessors (`get/upsert/remove_replica_system_config`) so the intended durable API surface is concrete and step 2b can be written against it. The stubs are no-ops, so resolution falls back to the coordinator's in-memory working copy (unchanged behavior). - An implementation note, `doc/developer/design/20260610_scoped_feature_flags_persistence_notes.md`, with the schema decision (one collection per scope), the exact version-bump recipe, the ~27 collection sites mirroring `system_configurations`, and the bootstrap-load + sole-writer persist wiring — to be finished in a session with the proper toolchain.
Implements step 3 of the scoped feature flags design (use case 1):
optimizer features can differ per cluster, driven from LaunchDarkly,
e.g. an optimizer feature enabled on `mz_catalog_server` but not on user
clusters.
- The sync loop now also evaluates `Cluster`-scoped parameters per live
cluster using the replica-free `cluster` context
(`SystemParameterFrontend::pull_cluster_overrides`), populating
`ScopedParameters.cluster`. The per-context evaluation is shared with
the replica pass. An environment with no `Cluster`-scoped flags skips
the pass entirely.
- The coordinator exposes `cluster_scoped_optimizer_overrides`, turning a
cluster's scoped working-copy entries into `OptimizerFeatureOverrides`.
- That layer is fed into the optimizer at the plan-time sequencing sites
(`coord/sequencer/inner/{peek,create_index,create_materialized_view,
subscribe}.rs` and `coord/introspection.rs`), applied *after* the
cluster's manual `CREATE CLUSTER ... FEATURES` overrides — so precedence
is env-wide LD < manual FEATURES < cluster-scoped LD, matching the
design.
Resolution is decided per feature where LD serves a cluster-specific
value (detected by the value differing from the environment-wide value);
where LD is silent, the manual `FEATURES` value stands.
Known gaps (follow-ups): the frontend fast-path peek (`frontend_peek.rs`)
and the bootstrap expression-cache re-optimization closure operate on a
bare catalog snapshot without the coordinator's working copy, so they do
not yet apply cluster-scoped overrides; full coverage there wants the
working copy in `CatalogState` (which the durable collection from step 2
naturally provides via the snapshot). A more precise "LD has an opinion"
test via `variation_detail` (vs. value comparison) is also deferred.
Implements step 4 of the scoped feature flags design: exposes the
resolved scoped overrides via two `mz_internal` builtin tables for
debugging.
- `mz_internal.mz_cluster_system_parameters(cluster_id, name, value)`
- `mz_internal.mz_replica_system_parameters(replica_id, name, value)`
Each row is a scoped value that differs from the environment-wide value
(the working copy is sparse). The coordinator maintains the tables from
its in-memory `ScopedParameters` working copy: on each reconcile that
changes the working copy it retracts the previous rows and inserts the
new ones (via `pack_{cluster,replica}_system_parameter_update` +
`background` builtin-table writes). Reconciles that don't change the
working copy are now skipped entirely (`ScopedParameters: PartialEq`),
avoiding a redundant write and compute-config re-push every sync tick.
On bootstrap the system tables are reset to empty and repopulated by the
first sync, so nothing stale lingers across a restart.
No CATALOG_VERSION bump or durable migration is needed (builtins use the
runtime-alterable fingerprint); just new OIDs and `BUILTINS` entries.
Reviewer must regenerate, in a proper-toolchain environment, the
autogenerated golden files that enumerate system relations (the code is
otherwise complete and compiles):
- test/sqllogictest/autogenerated/mz_internal.slt
- test/sqllogictest/autogenerated/mz_catalog.slt
- test/sqllogictest/information_schema_tables.slt
- test/sqllogictest/oid.slt
via the sqllogictest `--rewrite` tooling. Reference docs for the two
tables are added to doc/.../system-catalog/mz_internal.md by hand.
Add `cluster_system_configurations` and `replica_system_configurations` durable collections, keyed by object id -> (parameter name -> value), backing the durable cache for per-cluster and per-replica scoped feature flags (step 2 of the scoped feature flags design). Each mirrors the existing `system_configurations` (`ALTER SYSTEM`) collection across the proto types, serialization, snapshot/transaction/persist/debug plumbing, and the catalog-debug tool. Bump CATALOG_VERSION 85 -> 86 with a no-op `v85_to_v86` migration (the change only adds JSON-compatible `StateUpdateKind` variants and message types) plus the regenerated `objects_v86` snapshot and golden encodings. Replace the inert `Transaction` accessor stubs with real `TableTransaction` bodies (`get`/`upsert`/`remove` per scope). The in-memory catalog apply path is a no-op for these variants: the working copy is owned by the coordinator and the durable layer is a restart cache. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the durable scoped-parameter cache into the coordinator's working copy. The sync-loop reconcile is now the sole writer: it persists the diff between the durable cache and LaunchDarkly's desired state through a new `Op::UpdateScopedSystemParameters`, which upserts changed entries, removes ones LD no longer serves, and lazily prunes entries whose owning object id is absent from the catalog. On bootstrap the coordinator restores the working copy from the durable cache (filtered to live objects) and applies it before the first LD sync, so the last-known values are in effect immediately and survive an LD outage; resolution falls back to the environment-wide value only on a cold cache. `reconcile_scoped_system_parameters` becomes async (it persists via `catalog_transact`) and the in-memory application is factored into `apply_scoped_system_parameters`, shared with the bootstrap restore path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the in-memory scoped (per-cluster / per-replica) system-parameter working copy from the coordinator into `CatalogState`, where it rides along in the `Arc<Catalog>` snapshot. `apply.rs` now maintains it from the durable collections and packs the `mz_cluster_system_parameters` / `mz_replica_system_parameters` introspection rows, so the working copy is restored automatically when the catalog is opened and stays consistent through the normal catalog-update path. This closes a Step 3 gap: the fast-path peek sequencer (`frontend_peek`) and the bootstrap re-optimization closure only have a catalog snapshot, not the coordinator, so they could not see cluster-scoped optimizer overrides. `cluster_scoped_optimizer_overrides` now reads from the catalog, and both sites plus the bootstrap closure layer it on top of the manual `CREATE CLUSTER ... FEATURES` pin (cluster-scoped LD wins). `reconcile_scoped_system_parameters` persists through the catalog transaction (which updates the working copy and introspection) and then re-pushes replica-local overrides to the compute controller; the explicit coordinator field and bootstrap load are gone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`evaluate_scoped_overrides` recorded an override only when the LD value differed from the environment-wide value, which let a manual `CREATE CLUSTER ... FEATURES` pin wrongly survive an LD rule that served the env-wide value. Switch to `variation_detail` and record an override exactly when LD has a scope-specific opinion (`RULE_MATCH` / `TARGET_MATCH` / `FALLTHROUGH`), independent of value equality; when LD is silent (`FLAG_NOT_FOUND` / error / off) the manual pin stands. This matches the per-feature precedence on the global path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerate the autogenerated catalog enumeration golden files for the new `mz_cluster_system_parameters` / `mz_replica_system_parameters` introspection relations: the `mz_internal` doc-lint enumeration, the OID allocations, and the `information_schema.tables` listing. The catalog doc lint (ci/test/lint-docs-catalog.py) passes against the reference-doc entries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the LaunchDarkly integration test with cluster-coherent and replica-local cases. Adds rule-based targeting (clauses on context-kind attributes) and boolean flags, then demonstrates: * a cluster-coherent optimizer feature served `true` to builtin clusters (targeted by the `cluster` context's `is_builtin`) while user clusters fall through to `false`, observed via `mz_cluster_system_parameters`; * `enable_lgalloc` served `false` only to `legacy` size-family replicas (targeted by the `replica` context's `replica_size_family`) while `cc` replicas keep the env-wide value, observed via `mz_replica_system_parameters`; * durability of both across a restart with the sync loop disabled. The LAUNCHDARKLY_KEY_MAP entries are `;`-separated (the CLI arg's value delimiter). The LaunchDarkly project/environment keys are overridable via LAUNCHDARKLY_PROJECT_KEY / LAUNCHDARKLY_ENVIRONMENT_KEY for running against a non-CI project. Verified end-to-end against a live LaunchDarkly test project. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
antiguru
force-pushed
the
claude/great-brahmagupta-ybi2yf
branch
from
b0ef838 to
57944e4
Compare
The two new `mz_internal` builtin tables shift catalog enumerations and ids. Update the golden artifacts CI flagged: * `catalog.td` — add the tables to `SHOW TABLES FROM mz_internal` and bump the `mz_tables` system-table count 52 -> 54. * `mz_catalog_server_index_accounting.slt` — rewritten (builtin GlobalId renumbering; no new indexes). * `open__initial_snapshot.snap` — durable Snapshot dump now lists the two new collections. * `mz_internal.md` — fix the `mz_cluster_replicas` / `mz_clusters` doc links to the `../mz_catalog/#…` form (htmltest). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
LaunchDarkly flags are currently evaluated once against a single
environment-wide context, so there is no way to say "this flag has value X
on cluster A" or "this flag is on for replicas of size family D". Two use
cases need finer granularity: per-cluster optimizer features (e.g. on
mz_catalog_serveronly) and per-replica flags keyed by size family (e.g.lgallocon the legacy sizes).Design: doc/developer/design/20260609_scoped_feature_flags.md (#36947).
Description
Introduces scoped system parameters. Each synced parameter declares a
scope class (
Environment/Cluster/Replica) at its definition. Thesync loop evaluates two additional LaunchDarkly context kinds —
cluster(replica-free) and
replica(carrying size, size family, owning cluster) —and records an override when LD has a scope-specific opinion
(
variation_detailreason), so a cluster-scoped rule beats a manualCREATE CLUSTER ... FEATURESpin.Scoped values are a durable, id-keyed catalog cache written solely by the
sync loop (new
cluster_system_configurations/replica_system_configurationscollections, CATALOG_VERSION 86). The in-memory working copy lives in
CatalogStateand is resolved at the two existing per-scope boundaries:plan-time
OptimizerFeatureOverridesforcluster, and the controller'sper-replica dyncfg push for
replica. Values survive a restart and an LDoutage; resolution falls back to environment-wide only on a cold cache.
Resolved values are surfaced in
mz_internal.mz_{cluster,replica}_system_parameters.Verification
Catalog migration round-trip + golden-encoding tests, regenerated
sqllogictest catalog enumerations, and the
test/launchdarklymzcomposeworkflow extended with cluster/replica/durability cases — validated
end-to-end against a live LaunchDarkly project.