Problem
A CocoonHibernation is not pinned to one target, in two independent ways.
1. spec.podRef.name is mutable
The CRD constrains it with minLength: 1 only — there is no x-kubernetes-validations block anywhere in cocoonset.cocoonstack.io_cocoonhibernations.yaml, so a CR can be retargeted from pod A to pod B at any time.
Status.VMName is written only by setPhase(..., vmName). markPending does not rewrite it, so after a retarget to a pod that does not exist yet (or has no VMName annotation yet), status keeps naming A's VM indefinitely. Spec and status then disagree about which VM the CR owns:
reconcileDelete drops the tag of Status.VMName (A's VM) — correct, that is the snapshot this CR created.
- Everything else resolves from
spec.podRef.name (B).
A CR can therefore be simultaneously "about" B and still own A's snapshot.
2. Nothing rejects a second CR on one target
cocoon-webhook has no validator for the kind and the CRD has no uniqueness constraint, while the controller explicitly anticipates multiple CRs per pod (indexPodRefName's comment says "resolve a pod event back to the CRs that target it"; hibernationsTargetingPod returns a list).
Two live CRs with opposing desires on one pod do not merely race — they never converge. The Pod watch uses AnnotationChangedPredicate and fans every annotation change out to every CR targeting that pod, so:
- A(Hibernate) patches
hibernate=true → annotation-changed event → re-enqueues A and B
- B(Wake) reads
true, clears it to false → annotation-changed event → re-enqueues A and B
- A patches
true again → …
This flip-flop is independent of MaxConcurrentReconciles — it happens identically at one worker. Along the way the snapshot can be deleted by B after A has already certified Hibernated, and cocoonset.podsRestorableByCR treats Hibernated as "restore from :hibernate".
3. Corollary: the pre-lock Pod read (only reachable via 2)
Reconcile reads the Pod before taking the VM lock, so with two opposing CRs a waiter can act on an object it read before the wait: wake.go's if meta.ReadHibernateState(pod) guard, and PatchHibernateState's own if ReadHibernateState(pod) == state { return nil } short-circuit, both consult that stale copy and skip the clear.
Worth recording, but it is a symptom rather than a cause, and it is not fixable at this layer: r.Client is the cached client and Pods are watched, so re-reading inside the lock still consults the informer cache and can return the same stale annotation. Only mgr.GetAPIReader() is authoritative — and even that would not stop the flip-flop above. Closing precondition 2 removes the whole class.
Note 1 and 2 interact: uniqueness alone does not close it. Once a CR retargets from A to B, a new CR for A is perfectly legitimate under any per-spec.podRef uniqueness rule — yet the retargeted CR's stale status still makes it delete A's tag.
Already done (not this issue)
#16 keys a striped lock on the VM being touched (resolved vmName on the live paths, Status.VMName on the delete path, precisely so a retargeted CR still serializes against its stale VM). That makes MaxConcurrentReconciles > 1 safe for every well-formed configuration — one CR per pod means distinct CRs touch distinct VMs and the lock never even contends — and it stops the malformed case from interleaving registry calls. It is deliberately only serialization: it cannot make two opposing CRs converge, because nothing at that layer can.
Proposed scope
Pin a CR to one target so spec and status can never disagree, and so two CRs can never fight:
- Make
spec.podRef immutable via a CEL rule on the CRD (self == oldSelf). A hibernate/wake is expressed by flipping spec.desire, never by retargeting; retargeting has no legitimate use and is what lets status go stale.
- Add a validating webhook rejecting a CREATE whose
spec.podRef.name already has a live (non-deleting) CR in the namespace.
With podRef immutable, Status.VMName can only ever name the CR's one pod; with uniqueness, one target has exactly one owner, the flip-flop is unconstructable, and the pre-lock read in (3) becomes unreachable.
Acceptance criteria
- Editing
spec.podRef on an existing CR is rejected; flipping spec.desire remains allowed.
- Creating a second CocoonHibernation for a pod that already has a live one is rejected with an actionable message naming the existing CR.
- A CR whose predecessor is deleting (has
DeletionTimestamp) is admitted.
- The per-VM lock in the controller stays as defence in depth.
- Tests cover retarget-rejected, desire-flip-allowed, create-collision, and deleting-predecessor.
Problem
A
CocoonHibernationis not pinned to one target, in two independent ways.1.
spec.podRef.nameis mutableThe CRD constrains it with
minLength: 1only — there is nox-kubernetes-validationsblock anywhere incocoonset.cocoonstack.io_cocoonhibernations.yaml, so a CR can be retargeted from pod A to pod B at any time.Status.VMNameis written only bysetPhase(..., vmName).markPendingdoes not rewrite it, so after a retarget to a pod that does not exist yet (or has noVMNameannotation yet), status keeps naming A's VM indefinitely. Spec and status then disagree about which VM the CR owns:reconcileDeletedrops the tag ofStatus.VMName(A's VM) — correct, that is the snapshot this CR created.spec.podRef.name(B).A CR can therefore be simultaneously "about" B and still own A's snapshot.
2. Nothing rejects a second CR on one target
cocoon-webhook has no validator for the kind and the CRD has no uniqueness constraint, while the controller explicitly anticipates multiple CRs per pod (
indexPodRefName's comment says "resolve a pod event back to the CRs that target it";hibernationsTargetingPodreturns a list).Two live CRs with opposing desires on one pod do not merely race — they never converge. The Pod watch uses
AnnotationChangedPredicateand fans every annotation change out to every CR targeting that pod, so:hibernate=true→ annotation-changed event → re-enqueues A and Btrue, clears it tofalse→ annotation-changed event → re-enqueues A and Btrueagain → …This flip-flop is independent of
MaxConcurrentReconciles— it happens identically at one worker. Along the way the snapshot can be deleted by B after A has already certifiedHibernated, andcocoonset.podsRestorableByCRtreatsHibernatedas "restore from:hibernate".3. Corollary: the pre-lock Pod read (only reachable via 2)
Reconcilereads the Pod before taking the VM lock, so with two opposing CRs a waiter can act on an object it read before the wait:wake.go'sif meta.ReadHibernateState(pod)guard, andPatchHibernateState's ownif ReadHibernateState(pod) == state { return nil }short-circuit, both consult that stale copy and skip the clear.Worth recording, but it is a symptom rather than a cause, and it is not fixable at this layer:
r.Clientis the cached client and Pods are watched, so re-reading inside the lock still consults the informer cache and can return the same stale annotation. Onlymgr.GetAPIReader()is authoritative — and even that would not stop the flip-flop above. Closing precondition 2 removes the whole class.Note 1 and 2 interact: uniqueness alone does not close it. Once a CR retargets from A to B, a new CR for A is perfectly legitimate under any per-
spec.podRefuniqueness rule — yet the retargeted CR's stale status still makes it delete A's tag.Already done (not this issue)
#16 keys a striped lock on the VM being touched (resolved
vmNameon the live paths,Status.VMNameon the delete path, precisely so a retargeted CR still serializes against its stale VM). That makesMaxConcurrentReconciles > 1safe for every well-formed configuration — one CR per pod means distinct CRs touch distinct VMs and the lock never even contends — and it stops the malformed case from interleaving registry calls. It is deliberately only serialization: it cannot make two opposing CRs converge, because nothing at that layer can.Proposed scope
Pin a CR to one target so spec and status can never disagree, and so two CRs can never fight:
spec.podRefimmutable via a CEL rule on the CRD (self == oldSelf). A hibernate/wake is expressed by flippingspec.desire, never by retargeting; retargeting has no legitimate use and is what lets status go stale.spec.podRef.namealready has a live (non-deleting) CR in the namespace.With podRef immutable,
Status.VMNamecan only ever name the CR's one pod; with uniqueness, one target has exactly one owner, the flip-flop is unconstructable, and the pre-lock read in (3) becomes unreachable.Acceptance criteria
spec.podRefon an existing CR is rejected; flippingspec.desireremains allowed.DeletionTimestamp) is admitted.