feat(tls): support handing TLS over from BYO Secrets to operator-managed certs

README.md

@@ -22,7 +22,7 @@ The full design rationale is in [docs/concepts.md](docs/concepts.md).
 - **Memory-backed storage (opt-in)**: `spec.storage.medium: Memory` switches each member's data dir to a tmpfs `emptyDir` whose lifetime is bound to the Pod. Members that lose their Pod (eviction, node failure) lose their data; the operator detects this, removes the member from etcd, and replaces it via the existing scale-up path. Suits scenarios where the etcd state is reconstructable and replication absorbs single-member losses. For production, set `spec.affinity` and `spec.resources.limits.memory` explicitly — neither is defaulted ([#16](https://github.com/lllamnyp/etcd-operator/issues/16)); see [docs/concepts.md](docs/concepts.md#storage).
 - **Apiserver-enforced validation**: CEL rules on the CRD (k8s 1.29+) reject `replicas: 0` with `storage.medium: Memory`, `storage.size: 0` with `storage.medium: Memory`, `storage.medium` changes after creation, and `storage.size` shrinks. No webhook / cert-manager dependency.
 - **PodDisruptionBudget**: per-cluster PDB selects voting members only (`role=voter`); `maxUnavailable = (voters-1)/2` so `kubectl drain` cannot voluntarily push the cluster below quorum.
-- **TLS (BYO Secrets or cert-manager)**: `spec.tls.client` / `spec.tls.peer` enable TLS on each surface independently. Material comes from either user-provided Secrets (`serverSecretRef` / `operatorClientSecretRef` / `secretRef`) or operator-emitted `cert-manager.io/v1` Certificates (`certManager.{serverIssuerRef,operatorClientIssuerRef,issuerRef}`) — mutually exclusive per subtree, enforced by CEL. mTLS is the implicit mode when an operator-client source is supplied; server-TLS-only when it isn't. The whole `tls` subtree is CEL-locked immutable post-create. cert-manager-emitted certs auto-renew via cert-manager; Pod-side rotation is a manual one-at-a-time `kubectl delete pod` either way. See [docs/concepts.md](docs/concepts.md#tls).
+- **TLS (BYO Secrets or cert-manager)**: `spec.tls.client` / `spec.tls.peer` enable TLS on each surface independently. Material comes from either user-provided Secrets (`serverSecretRef` / `operatorClientSecretRef` / `secretRef`) or operator-emitted `cert-manager.io/v1` Certificates (`certManager.{serverIssuerRef,operatorClientIssuerRef,issuerRef}`) — mutually exclusive per subtree, enforced by CEL. mTLS is the implicit mode when an operator-client source is supplied; server-TLS-only when it isn't. The `tls` subtree is CEL-locked immutable post-create except for one move: handing the material over from user-provided Secrets to operator-managed cert-manager issuance, which the operator performs itself — see [Handing TLS over to the operator](docs/operations.md#handing-tls-over-to-the-operator). cert-manager-emitted certs auto-renew via cert-manager; Pod-side rotation is a manual one-at-a-time `kubectl delete pod` either way. See [docs/concepts.md](docs/concepts.md#tls).
 - **Resource sizing**: `spec.resources` (a `corev1.ResourceRequirements`) sets the etcd container's CPU/memory requests and limits. Unset uses a conservative 100m/128Mi-request default. Updates take effect on newly-created members; pair with a `VerticalPodAutoscaler` targeting the cluster for live recommendation/rollout.
 - **Scheduling & extra metadata**: `spec.affinity` and `spec.topologySpreadConstraints` pass through to every member Pod (anti-affinity is not defaulted — set it for production); `spec.additionalMetadata` merges user labels/annotations onto every object the operator creates (member Pods, data PVCs, Services, PDB, `EtcdMember` CRs), with operator-owned keys winning on collision. All three apply on object creation and are latched like the rest of the spec. See [docs/concepts.md](docs/concepts.md#pod-scheduling-and-additional-metadata).
 - **Monitoring / autoscaling hooks**: every member Pod always exposes a plaintext `metrics` container port at `2381` (etcd's `/health` + Prometheus `/metrics`) for `VMPodScrape` / `PodMonitor`. The `EtcdCluster` CRD exposes the `/scale` subresource with a populated `status.selector`, making it a valid target for `kubectl scale` and `VerticalPodAutoscaler.targetRef`.

api/v1alpha2/cel_validation_test.go

@@ -292,11 +292,195 @@ func TestCEL_TLSSubfieldChangeRejected(t *testing.T) {
 	if err == nil {
 		t.Fatalf("apiserver accepted mTLS toggle (added operatorClientSecretRef); expected rejection")
 	}
-	if !strings.Contains(err.Error(), "spec.tls is immutable") {
+	if !strings.Contains(err.Error(), "spec.tls.client is immutable") {
 		t.Fatalf("error did not mention subtree immutability: %v", err)
 	}
 }
 
+// The one permitted spec.tls change: handing the material over from
+// user-provided Secrets to operator-managed cert-manager issuance, on both
+// planes at once.
+func TestCEL_TLSHandoverToCertManagerAccepted(t *testing.T) {
+	skipIfNoEnvtest(t)
+	ctx := context.Background()
+
+	c := validCluster("tls-handover-ok")
+	c.Spec.TLS = &lll.EtcdClusterTLS{
+		Client: &lll.ClientTLS{
+			ServerSecretRef: &corev1.LocalObjectReference{Name: "chart-server-tls"},
+		},
+		Peer: &lll.PeerTLS{
+			SecretRef: &corev1.LocalObjectReference{Name: "chart-peer-tls"},
+		},
+	}
+	if err := k8s.Create(ctx, c); err != nil {
+		t.Fatalf("Create BYO-TLS cluster: %v", err)
+	}
+	t.Cleanup(func() { _ = k8s.Delete(ctx, c) })
+
+	got := &lll.EtcdCluster{}
+	if err := k8s.Get(ctx, ctrlclient.ObjectKeyFromObject(c), got); err != nil {
+		t.Fatalf("Get: %v", err)
+	}
+	got.Spec.TLS.Client = &lll.ClientTLS{
+		CertManager: &lll.ClientCertManagerTLS{
+			ServerIssuerRef: lll.IssuerReference{Name: "etcd-issuer"},
+		},
+	}
+	got.Spec.TLS.Peer = &lll.PeerTLS{
+		CertManager: &lll.PeerCertManagerTLS{
+			IssuerRef: lll.IssuerReference{Name: "etcd-peer-issuer"},
+		},
+	}
+	if err := k8s.Update(ctx, got); err != nil {
+		t.Fatalf("apiserver rejected the BYO->cert-manager handover: %v", err)
+	}
+}
+
+// The handover is one-way. Going back to BYO Secrets would strand the
+// cluster on material the operator is about to GC.
+func TestCEL_TLSHandoverBackToSecretRefRejected(t *testing.T) {
+	skipIfNoEnvtest(t)
+	ctx := context.Background()
+
+	c := validCluster("tls-handover-reverse")
+	c.Spec.TLS = &lll.EtcdClusterTLS{
+		Peer: &lll.PeerTLS{
+			CertManager: &lll.PeerCertManagerTLS{
+				IssuerRef: lll.IssuerReference{Name: "etcd-peer-issuer"},
+			},
+		},
+	}
+	if err := k8s.Create(ctx, c); err != nil {
+		t.Fatalf("Create cert-manager cluster: %v", err)
+	}
+	t.Cleanup(func() { _ = k8s.Delete(ctx, c) })
+
+	got := &lll.EtcdCluster{}
+	if err := k8s.Get(ctx, ctrlclient.ObjectKeyFromObject(c), got); err != nil {
+		t.Fatalf("Get: %v", err)
+	}
+	got.Spec.TLS.Peer = &lll.PeerTLS{
+		SecretRef: &corev1.LocalObjectReference{Name: "chart-peer-tls"},
+	}
+
+	err := k8s.Update(ctx, got)
+	if err == nil {
+		t.Fatalf("apiserver accepted the reverse handover back to secretRef; expected rejection")
+	}
+	if !strings.Contains(err.Error(), "spec.tls.peer is immutable") {
+		t.Fatalf("error did not mention peer subtree immutability: %v", err)
+	}
+}
+
+// The handover must not be a vehicle for silently dropping client mTLS:
+// a cluster that presented an operator client cert has to keep presenting
+// one on the other side of the move.
+func TestCEL_TLSHandoverDroppingClientMTLSRejected(t *testing.T) {
+	skipIfNoEnvtest(t)
+	ctx := context.Background()
+
+	c := validCluster("tls-handover-mtls")
+	c.Spec.TLS = &lll.EtcdClusterTLS{
+		Client: &lll.ClientTLS{
+			ServerSecretRef:         &corev1.LocalObjectReference{Name: "chart-server-tls"},
+			OperatorClientSecretRef: &corev1.LocalObjectReference{Name: "chart-client-tls"},
+		},
+	}
+	if err := k8s.Create(ctx, c); err != nil {
+		t.Fatalf("Create mTLS cluster: %v", err)
+	}
+	t.Cleanup(func() { _ = k8s.Delete(ctx, c) })
+
+	got := &lll.EtcdCluster{}
+	if err := k8s.Get(ctx, ctrlclient.ObjectKeyFromObject(c), got); err != nil {
+		t.Fatalf("Get: %v", err)
+	}
+	// certManager without operatorClientIssuerRef == server-TLS only.
+	got.Spec.TLS.Client = &lll.ClientTLS{
+		CertManager: &lll.ClientCertManagerTLS{
+			ServerIssuerRef: lll.IssuerReference{Name: "etcd-issuer"},
+		},
+	}
+
+	err := k8s.Update(ctx, got)
+	if err == nil {
+		t.Fatalf("apiserver accepted a handover that silently disabled client mTLS; expected rejection")
+	}
+	if !strings.Contains(err.Error(), "mTLS posture") {
+		t.Fatalf("error did not mention the mTLS posture requirement: %v", err)
+	}
+}
+
+// The mTLS-preserving form of the same handover is accepted.
+func TestCEL_TLSHandoverPreservingClientMTLSAccepted(t *testing.T) {
+	skipIfNoEnvtest(t)
+	ctx := context.Background()
+
+	c := validCluster("tls-handover-mtls-ok")
+	c.Spec.TLS = &lll.EtcdClusterTLS{
+		Client: &lll.ClientTLS{
+			ServerSecretRef:         &corev1.LocalObjectReference{Name: "chart-server-tls"},
+			OperatorClientSecretRef: &corev1.LocalObjectReference{Name: "chart-client-tls"},
+		},
+	}
+	if err := k8s.Create(ctx, c); err != nil {
+		t.Fatalf("Create mTLS cluster: %v", err)
+	}
+	t.Cleanup(func() { _ = k8s.Delete(ctx, c) })
+
+	got := &lll.EtcdCluster{}
+	if err := k8s.Get(ctx, ctrlclient.ObjectKeyFromObject(c), got); err != nil {
+		t.Fatalf("Get: %v", err)
+	}
+	got.Spec.TLS.Client = &lll.ClientTLS{
+		CertManager: &lll.ClientCertManagerTLS{
+			ServerIssuerRef:         lll.IssuerReference{Name: "etcd-issuer"},
+			OperatorClientIssuerRef: &lll.IssuerReference{Name: "etcd-issuer"},
+		},
+	}
+	if err := k8s.Update(ctx, got); err != nil {
+		t.Fatalf("apiserver rejected an mTLS-preserving handover: %v", err)
+	}
+}
+
+// The TLS subtrees themselves still cannot appear or vanish — the handover
+// exception is about the source of the material, not about turning a plane
+// on or off.
+func TestCEL_TLSPeerSubtreeAddRejected(t *testing.T) {
+	skipIfNoEnvtest(t)
+	ctx := context.Background()
+
+	c := validCluster("tls-peer-add")
+	c.Spec.TLS = &lll.EtcdClusterTLS{
+		Client: &lll.ClientTLS{
+			ServerSecretRef: &corev1.LocalObjectReference{Name: "chart-server-tls"},
+		},
+	}
+	if err := k8s.Create(ctx, c); err != nil {
+		t.Fatalf("Create client-only TLS cluster: %v", err)
+	}
+	t.Cleanup(func() { _ = k8s.Delete(ctx, c) })
+
+	got := &lll.EtcdCluster{}
+	if err := k8s.Get(ctx, ctrlclient.ObjectKeyFromObject(c), got); err != nil {
+		t.Fatalf("Get: %v", err)
+	}
+	got.Spec.TLS.Peer = &lll.PeerTLS{
+		CertManager: &lll.PeerCertManagerTLS{
+			IssuerRef: lll.IssuerReference{Name: "etcd-peer-issuer"},
+		},
+	}
+
+	err := k8s.Update(ctx, got)
+	if err == nil {
+		t.Fatalf("apiserver accepted adding the peer TLS subtree post-create; expected rejection")
+	}
+	if !strings.Contains(err.Error(), "spec.tls.peer cannot be added") {
+		t.Fatalf("error did not mention peer subtree add/remove: %v", err)
+	}
+}
+
 // TestCEL_StorageClassNameAddRejected verifies that adding a
 // storageClassName after Create is rejected. A PVC's storageClassName
 // is itself immutable, so the operator can only honour a value chosen

api/v1alpha2/etcdcluster_types.go

@@ -24,11 +24,24 @@ import (
 
 // EtcdClusterTLS configures transport-layer security for the cluster's two
 // etcd surfaces: the client API (port 2379) and the peer API (port 2380).
-// Each subtree is independently optional. Subtree fields are immutable
-// post-create — flipping TLS on or off on an existing cluster is a
-// non-trivial rolling change (the operator's own etcd client must switch
-// protocols in lockstep with the members), so v1 punts that to delete-and-
-// recreate.
+// Each subtree is independently optional and immutable post-create, with
+// one exception: the source of the material may be handed over from
+// user-provided Secrets to operator-managed cert-manager issuance
+// (Client.ServerSecretRef → Client.CertManager, Peer.SecretRef →
+// Peer.CertManager). The operator drives that handover itself — it issues
+// the new material, waits for cert-manager to populate it, then rolls every
+// member onto it at once.
+//
+// Nothing else about the subtree may change. Flipping TLS on or off, or
+// handing back from cert-manager to BYO Secrets, still requires delete-and-
+// recreate: the operator's own etcd client would have to switch protocols
+// or trust roots in lockstep with the members, and there is no safe
+// operator-driven sequence for that.
+//
+// The handover is deliberately one-way. Going back to BYO would mean
+// trusting material the operator cannot verify is in place, and the
+// cert-manager Certificates it owns would be GC'd out from under a running
+// cluster the moment the spec stopped referencing them.
 type EtcdClusterTLS struct {
 	// Client configures TLS for the etcd client API (port 2379). Absent
 	// means plaintext. See ClientTLS for the mTLS-toggle semantics.
@@ -307,6 +320,31 @@ const (
 	ClusterProgressing = "Progressing"
 	// ClusterDegraded indicates some members are unhealthy but quorum holds.
 	ClusterDegraded = "Degraded"
+	// ClusterTLSHandover tracks the one-way move from user-provided TLS
+	// Secrets to operator-managed cert-manager material. True while a
+	// handover is in flight; False once it has settled or when it is
+	// blocked. It is deliberately a type of its own rather than a reason
+	// on Available/Degraded/Progressing: updateStatus rewrites those three
+	// from quorum health on every pass, so a handover signal parked on any
+	// of them would be clobbered within seconds.
+	ClusterTLSHandover = "TLSHandover"
+)
+
+// Condition reasons for ClusterTLSHandover.
+const (
+	// TLSHandoverAwaitingMaterial: the operator has requested the new
+	// Certificates and is waiting for cert-manager to populate the Secrets.
+	// No member has been touched yet.
+	TLSHandoverAwaitingMaterial = "AwaitingMaterial"
+	// TLSHandoverRollingMembers: every member has been repointed at the new
+	// material and their Pods are being rebuilt.
+	TLSHandoverRollingMembers = "RollingMembers"
+	// TLSHandoverBlocked: a piece of TLS material the operator must own is
+	// held by another controller. Requires human intervention; the cluster
+	// keeps running on the material it already has.
+	TLSHandoverBlocked = "Blocked"
+	// TLSHandoverComplete: every member runs on operator-managed material.
+	TLSHandoverComplete = "Complete"
 )
 
 // BootstrapSpec configures one-time cluster initialization. Consulted only
@@ -425,10 +463,25 @@ type StorageSpec struct {
 //     the tmpfs is unbounded against node memory, which defeats the whole
 //     point of opting into memory backing.
 //
+//   - spec.tls is immutable post-create with exactly one exception: the
+//     one-way handover from user-provided Secrets to operator-managed
+//     cert-manager issuance (client.serverSecretRef → client.certManager,
+//     peer.secretRef → peer.certManager). Everything else about the TLS
+//     subtree stays frozen — the plaintext↔TLS flip, the reverse handover
+//     back to BYO Secrets, swapping one Secret ref for another, and
+//     toggling the client-mTLS posture. The handover is permitted because
+//     the operator can drive it safely end to end (issue the new material,
+//     wait for it, then roll every member onto it in one step); the
+//     others have no such path and still require delete-and-recreate.
+//     Progress is reported on the TLSHandover status condition.
+//
 // +kubebuilder:validation:XValidation:rule="!(has(self.replicas) && self.replicas == 0 && has(self.storage) && has(self.storage.medium) && self.storage.medium == 'Memory')",message="spec.replicas=0 with spec.storage.medium=Memory is unsupported: pausing a memory-backed cluster wedges on resume. Delete and recreate the cluster instead."
 // +kubebuilder:validation:XValidation:rule="!(has(self.storage) && has(self.storage.medium) && self.storage.medium == 'Memory') || quantity(string(self.storage.size)).isGreaterThan(quantity('0'))",message="spec.storage.size must be > 0 when spec.storage.medium=Memory (the tmpfs sizeLimit cannot be zero)."
 // +kubebuilder:validation:XValidation:rule="has(self.tls) == has(oldSelf.tls)",message="spec.tls cannot be added to or removed from an existing cluster; delete and recreate"
-// +kubebuilder:validation:XValidation:rule="!has(self.tls) || !has(oldSelf.tls) || self.tls == oldSelf.tls",message="spec.tls is immutable post-create; delete and recreate the cluster to change TLS configuration"
+// +kubebuilder:validation:XValidation:rule="!has(self.tls) || !has(oldSelf.tls) || has(self.tls.client) == has(oldSelf.tls.client)",message="spec.tls.client cannot be added to or removed from an existing cluster; delete and recreate"
+// +kubebuilder:validation:XValidation:rule="!has(self.tls) || !has(oldSelf.tls) || has(self.tls.peer) == has(oldSelf.tls.peer)",message="spec.tls.peer cannot be added to or removed from an existing cluster; delete and recreate"
+// +kubebuilder:validation:XValidation:rule="!has(self.tls) || !has(oldSelf.tls) || !has(self.tls.client) || self.tls.client == oldSelf.tls.client || (has(oldSelf.tls.client.serverSecretRef) && has(self.tls.client.certManager) && has(oldSelf.tls.client.operatorClientSecretRef) == has(self.tls.client.certManager.operatorClientIssuerRef))",message="spec.tls.client is immutable post-create except for the one-way handover from serverSecretRef to certManager, which must preserve the mTLS posture (operatorClientSecretRef set iff certManager.operatorClientIssuerRef is set)"
+// +kubebuilder:validation:XValidation:rule="!has(self.tls) || !has(oldSelf.tls) || !has(self.tls.peer) || self.tls.peer == oldSelf.tls.peer || (has(oldSelf.tls.peer.secretRef) && has(self.tls.peer.certManager))",message="spec.tls.peer is immutable post-create except for the one-way handover from secretRef to certManager"
 // +kubebuilder:validation:XValidation:rule="has(self.storage.storageClassName) == has(oldSelf.storage.storageClassName)",message="spec.storage.storageClassName cannot be added to or removed from an existing cluster; delete and recreate"
 // +kubebuilder:validation:XValidation:rule="!has(self.storage.storageClassName) || !has(oldSelf.storage.storageClassName) || self.storage.storageClassName == oldSelf.storage.storageClassName",message="spec.storage.storageClassName is immutable post-create (a PVC's storageClassName itself is immutable, and the operator does not roll PVCs); delete and recreate the cluster to change the StorageClass"
 // +kubebuilder:validation:XValidation:rule="has(self.auth) == has(oldSelf.auth)",message="spec.auth cannot be added to or removed from an existing cluster; delete and recreate"