docs: design the general member-rollout mechanism for mutable settings

[Timofei Larkin (lllamnyp)]

Summary

Design proposal, no code. Adds docs/member-rollout-design.md.

EtcdCluster has exactly one setting that changes a running cluster: spec.replicas. Everything else is either CEL-locked to delete-and-recreate, or accepted by the apiserver and then applied to nothing.

The second category is the part worth naming, because from the outside it looks like it worked. snapshotSpecIntoObserved copies version, storage, resources, affinity, topologySpreadConstraints, additionalMetadata, options and imagePullSecrets into status.observed; members mirror those fields only at creation; and reconciliationComplete asks only whether the member count matches, whether ClusterID is latched, and whether every member is Ready. It never asks whether a single member reflects the values just snapshotted.

So editing spec.resources on a live cluster snapshots into observed, sets Progressing=True/SpecChanged, and — because the count never changed and everyone is still Ready — immediately settles to Progressing=False/Reconciled. The CR reports it reconciled a change no member received. There is no signal to the user that anything was dropped.

What the design proposes

A rollout engine that compares each member structurally against the template derived from status.observed and rolls outdated members one at a time, branching on storage medium:

• PVC-backed — delete the Pod; the data dir survives and the member rejoins with its log intact.
• Memory-backed — delete the EtcdMember; the tmpfs data dir died with the Pod, so this is a replacement, and the gap-fill goes in via MemberAddAsLearner and is promoted only after catch-up.

Each step gates on cluster-level health — MemberList showing the expected voting members with none IsLearner, endpoint health on every endpoint, applied-index convergence, a linearizable read — rather than on Pod readiness, which is too weak for a member that rejoined empty.

Changes contribute ordered passes, each completed across all members before the next begins. Most changes are one pass. CA rotation is two (widen trust to concat(old, new), then swap leaves), and that structure is what makes a CA rotation expressible without a cluster-wide outage.

The doc also classifies every spec field as rollable or genuinely locked, and proposes a single chokepoint for member disruption so the one-at-a-time rule is structural rather than something every future code path has to remember.

Why one-at-a-time is not negotiable

With three members quorum is two, so exactly one may be down. On a PVC cluster exceeding that costs availability; on a memory-backed cluster it costs the data, because no survivor holds it. pdbMaxUnavailable already encodes the arithmetic, but the PodDisruptionBudget it feeds only gates the Eviction API — the operator deletes Pods directly, so the PDB does not constrain the operator at all. Self-governance is the only protection there is, which is the argument for enforcing it in one place.

Relationship to the TLS handover work

#349 is parked as a draft. Its simultaneous roll is both unnecessary (the two-pass CA sequence has no intermediate state where a pair of members cannot authenticate) and, on memory-backed clusters, unrecoverable. The parts of it that stand on their own — the CEL relaxation, the fail-closed conflict guard, the material-readiness gate, not rebuilding Pods the operator did not author — carry over onto this mechanism.

The companion research note docs/etcd-tls-reload.md is not included here: it cross-references a runbook section that currently exists only on the #349 branch, so committing it onto this branch would introduce a dangling link. The two should land together, or that one first.

Open questions in the doc

1. spec.updateStrategy — RollingUpdate vs OnDelete, so disruption lands in a chosen maintenance window?
2. Version-upgrade policy — gate to one minor step and refuse downgrades, or leave it to the user?
3. Is PVC expansion for storage.size in scope here, or separate work? It is a PVC patch, not a roll.
4. Concurrency on 5- and 7-member clusters — quorum would permit 2 or 3; proposal keeps it at 1 everywhere.
5. The PDB and the engine are now two encodings of the same arithmetic — collapse, or accept the split?

Status of the claims

Everything in the "confirmed" list is read off the code and cited with file:line. The rollout step ordering, victim-selection preferences and health-gate composition are proposed and unvalidated — the cheap check is a scratch three-member cluster of each medium, rolling resources end to end before anything harder is attempted.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c15379a3-33c4-42cf-a186-eba031cbc498

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[K.J. Valencik (kjvalencik)]

Why one-at-a-time is not negotiable

I just ran into this exact issue with a memory backed cluster. The problem with the PodDisruptionBudget is that it races while scaling down. My cluster lost quorum because the cluster shrunk to 3 from clean member removals during re-schedules, but the PDB still allowed for more members to be deleted.

I think the fix is to use minAvailable instead of maxUnavailable. Ideally, I want intended cluster size - 1, not current size. E.g., 5 should always have minAvailable: 4.

I can think of three ways to solve this, ordered by increasing complexity/flexibility:

• Allow disabling the generated PDB. This grants freedom to create one yourself without overlapping (which is equivalent to do-not-disrupt).
• Switch to minAvailable as the default as I described above
• Allow setting a podDisruptionBudget that has minAvailable, maxUnavailable or complete freedom