Nodes unable to connect to volumes after cluster restart due to bid being too low.

[bhechinger]

So I wasn't paying attention and my cluster got de-provisioned due to the bid being too low (goodbye $8/month cluster!).

I increased the bid to get the cluster re-provisioned, which is was.

But now I'm having issues with my pods and volumes. In this case I even deleted the deployment/pvc and completely re-created it (no stateful data, so not a big deal), but the issue has persisted.

I'm currently stuck with my pods not starting due to:

Warning  FailedAttachVolume      100s               attachdetach-controller  AttachVolume.Attach failed for volume "pvc-490c2c6d-3a3b-4c1e-ad00-1e8cb674af8a" : rpc error: code = Internal desc = [ControllerPublishVolume] failed to attach volume: [2025-07-27T20:22:51Z] [attempt-1] Volume 8f58fcfb-ced9-4c90-a801-7fac50a414f5 is not yet attached to instance ed616107-08ce-4d73-aadd-d036724e1898
  Warning  FailedAttachVolume      82s (x5 over 99s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-490c2c6d-3a3b-4c1e-ad00-1e8cb674af8a" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach 8f58fcfb-ced9-4c90-a801-7fac50a414f5 volume to ed616107-08ce-4d73-aadd-d036724e1898 compute: Bad request with: [POST https://csi-proxy-lon-prod.spot.rackspace.com/lon.servers/v2/10079929/servers/ed616107-08ce-4d73-aadd-d036724e1898/os-volume_attachments], error message: {"badRequest": {"message": "Invalid volume: volume '8f58fcfb-ced9-4c90-a801-7fac50a414f5' status must be 'available'. Currently in 'attaching'", "code": 400}}
  Warning  FailedAttachVolume      65s                attachdetach-controller  AttachVolume.Attach failed for volume "pvc-490c2c6d-3a3b-4c1e-ad00-1e8cb674af8a" : rpc error: code = Internal desc = [ControllerPublishVolume] Attach Volume failed with error failed to attach 8f58fcfb-ced9-4c90-a801-7fac50a414f5 volume to ed616107-08ce-4d73-aadd-d036724e1898 compute: Bad request with: [POST https://csi-proxy-lon-prod.spot.rackspace.com/lon.servers/v2/10079929/servers/ed616107-08ce-4d73-aadd-d036724e1898/os-volume_attachments], error message: {"badRequest": {"message": "Invalid volume: volume '8f58fcfb-ced9-4c90-a801-7fac50a414f5' status must be 'available'. Currently in 'in-use'", "code": 400}}
  Normal   SuccessfulAttachVolume  33s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-490c2c6d-3a3b-4c1e-ad00-1e8cb674af8a"

The PVC is bound, however, so I'm not sure what's going on:

➜  deploy git:(main) ✗ k -n clinic-evolve get pvc                                    
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
clinic-evolve-webroot   Bound    pvc-490c2c6d-3a3b-4c1e-ad00-1e8cb674af8a   5Gi        RWO            ssd            <unset>                 21m

[sahil-lakhwani]

Hey @bhechinger , I think the events that you see convey a temporary problem. The PV is correctly bound the pod.
The pod seems to have an initContainer which is crashing for some reason.
I think once that is fixed, the other containers should just come up fine.

You can take a look; if you see anything different, let use know.

[bhechinger]

Ah, you are absolutely correct. I was so focused on the PV errors I got stuck and didn't look at the init container logs.

Sorry for the noise!