Skip to content

fix(hibernation): pin a CocoonHibernation to one target — immutable podRef + uniqueness #17

Description

@CMGS

Problem

A CocoonHibernation is not pinned to one target, in two independent ways.

1. spec.podRef.name is mutable

The CRD constrains it with minLength: 1 only — there is no x-kubernetes-validations block anywhere in cocoonset.cocoonstack.io_cocoonhibernations.yaml, so a CR can be retargeted from pod A to pod B at any time.

Status.VMName is written only by setPhase(..., vmName). markPending does not rewrite it, so after a retarget to a pod that does not exist yet (or has no VMName annotation yet), status keeps naming A's VM indefinitely. Spec and status then disagree about which VM the CR owns:

  • reconcileDelete drops the tag of Status.VMName (A's VM) — correct, that is the snapshot this CR created.
  • Everything else resolves from spec.podRef.name (B).

A CR can therefore be simultaneously "about" B and still own A's snapshot.

2. Nothing rejects a second CR on one target

cocoon-webhook has no validator for the kind and the CRD has no uniqueness constraint, while the controller explicitly anticipates multiple CRs per pod (indexPodRefName's comment says "resolve a pod event back to the CRs that target it"; hibernationsTargetingPod returns a list).

Two live CRs with opposing desires on one pod do not merely race — they never converge. The Pod watch uses AnnotationChangedPredicate and fans every annotation change out to every CR targeting that pod, so:

  1. A(Hibernate) patches hibernate=true → annotation-changed event → re-enqueues A and B
  2. B(Wake) reads true, clears it to false → annotation-changed event → re-enqueues A and B
  3. A patches true again → …

This flip-flop is independent of MaxConcurrentReconciles — it happens identically at one worker. Along the way the snapshot can be deleted by B after A has already certified Hibernated, and cocoonset.podsRestorableByCR treats Hibernated as "restore from :hibernate".

3. Corollary: the pre-lock Pod read (only reachable via 2)

Reconcile reads the Pod before taking the VM lock, so with two opposing CRs a waiter can act on an object it read before the wait: wake.go's if meta.ReadHibernateState(pod) guard, and PatchHibernateState's own if ReadHibernateState(pod) == state { return nil } short-circuit, both consult that stale copy and skip the clear.

Worth recording, but it is a symptom rather than a cause, and it is not fixable at this layer: r.Client is the cached client and Pods are watched, so re-reading inside the lock still consults the informer cache and can return the same stale annotation. Only mgr.GetAPIReader() is authoritative — and even that would not stop the flip-flop above. Closing precondition 2 removes the whole class.

Note 1 and 2 interact: uniqueness alone does not close it. Once a CR retargets from A to B, a new CR for A is perfectly legitimate under any per-spec.podRef uniqueness rule — yet the retargeted CR's stale status still makes it delete A's tag.

Already done (not this issue)

#16 keys a striped lock on the VM being touched (resolved vmName on the live paths, Status.VMName on the delete path, precisely so a retargeted CR still serializes against its stale VM). That makes MaxConcurrentReconciles > 1 safe for every well-formed configuration — one CR per pod means distinct CRs touch distinct VMs and the lock never even contends — and it stops the malformed case from interleaving registry calls. It is deliberately only serialization: it cannot make two opposing CRs converge, because nothing at that layer can.

Proposed scope

Pin a CR to one target so spec and status can never disagree, and so two CRs can never fight:

  1. Make spec.podRef immutable via a CEL rule on the CRD (self == oldSelf). A hibernate/wake is expressed by flipping spec.desire, never by retargeting; retargeting has no legitimate use and is what lets status go stale.
  2. Add a validating webhook rejecting a CREATE whose spec.podRef.name already has a live (non-deleting) CR in the namespace.

With podRef immutable, Status.VMName can only ever name the CR's one pod; with uniqueness, one target has exactly one owner, the flip-flop is unconstructable, and the pre-lock read in (3) becomes unreachable.

Acceptance criteria

  • Editing spec.podRef on an existing CR is rejected; flipping spec.desire remains allowed.
  • Creating a second CocoonHibernation for a pod that already has a live one is rejected with an actionable message naming the existing CR.
  • A CR whose predecessor is deleting (has DeletionTimestamp) is admitted.
  • The per-VM lock in the controller stays as defence in depth.
  • Tests cover retarget-rejected, desire-flip-allowed, create-collision, and deleting-predecessor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions