Windows clone stranded at lifecycle-state=creating when markReadyAfterIP ctx is canceled mid-wait (never ready, never failed)

[tonicmuroq]

Summary

A Windows mode: clone pod can strand at lifecycle-state=creating forever — never ready, never failed — while the VM underneath is fully healthy. Root cause: markReadyAfterIP returns silently on ctx.Err() (neither marks ready nor failed), so if the readiness goroutine's context is canceled mid-wait (observed: a concurrent UpdatePod burst), the pod is left permanently in creating.

Observed on prod-cocoon (c3-metal), vk-cocoon rev 7d254cf.

Symptom

CocoonSet win11 (mode: clone, os: windows, Windows 11 base snapshot), one of three identical Windows clone lanes (win10 / win11 / winserver):

• Pod win11-0: Running 1/1, status.podIP=172.24.1.161, but vm.cocoonstack.io/lifecycle-state=creating for 14+ minutes.
• The VM is completely healthy: kubectl exec works (Windows 11 Pro), egress works (curl ifconfig.me → node ext IP), the node can ping the VM IP, and cocoon-net holds the lease (aa:19:42:20:d4:ba → 172.24.1.161, matching the VM's NIC MAC).
• win10 and winserver — same batch, same clone path — reached ready normally.

Gotcha for operators: lifecycle-state here is a false negative. kubectl get pod -o wide shows the pod IP and exec works; the annotation lies.

vk log (node-2)

01:51:29  INF update pod cocoon-e2e/win11-0  func=Provider.UpdatePod   (x several — a burst)
01:51:36  INF post-clone setup succeeded for cocoon-e2e/win11-0 vm=LPL2G67... total_dur=7.351s func=Provider.runPostCloneSetup
01:51:36  INF update pod cocoon-e2e/win11-0  func=Provider.UpdatePod
<then silence — NO markLifecycleState ready/failed for win11-0 ever>

Root cause

The clone ready path is markReadyAfterIP (provider/cocoon/postclone.go:125):

• postclone.go#L125-L147 — waits for the fresh DHCP lease (waitForFreshIP), then marks ready; on timeout marks failed.
• Line 127: if ctx.Err() != nil { return } — returns without marking ready or failed if the context is canceled during the wait. There is no reconcile that re-drives the readiness decision afterward, so the pod is stranded at creating.

The concurrent Provider.UpdatePod burst at 01:51:29–36 is the likely canceler of the post-clone readiness goroutine's context.

Why win11 specifically, and why intermittently

• This was the first clone of the win11 base snapshot on the node → vk pulled ~8.9 GB from AR (~14 min); the pod stayed creating the whole time, widening the race window.
• win11 also has the documented DHCP-on-clone quirk (update.go#L35, #L329 — "wedges on APIPA after a NAK … never re-DISCOVERs"), so waitForFreshIP legitimately waits longer for a cold win11 clone.
• Non-deterministic: deleting + recreating the CocoonSet (snapshot now cached → fast clone, short/no wait) reached ready cleanly. So this is a timing race, exposed by a long clone (cold pull + slow win11 DHCP), not a deterministic win11 failure.

Impact

• A healthy, exec-able, network-up VM is reported as never-ready. vm-service gates on ready to read the pod IP, so the VM is unusable to it despite being up. A persistent creating may also trip operator rebuild logic.

Suggested direction

Make the readiness decision reconcile-driven / idempotent rather than a one-shot goroutine that can be silently canceled: on ctx.Err(), don't strand — either re-drive markReadyAfterIP from the next UpdatePod for a pod still in creating with a running VM + observable lease, or don't cancel the in-flight post-clone readiness goroutine on a benign UpdatePod.

Repro (approx)

1. mode: clone, os: windows CocoonSet from a Windows 11 base snapshot not yet cached on the target node (forces a long AR pull).
2. Watch lifecycle-state vs status.podIP/kubectl exec. Intermittently the VM comes up healthy but lifecycle-state stays creating indefinitely.
3. Delete + recreate (now cached) → reaches ready.