pvcviewer-controller-manager creates pvcviewer pod on different node from notebook server pod

[muratyarali]

I'd like to ask you something about the pvcviewer-controller-manager.

1- I want to see the volume content attached to a notebook by clicking the "Open PVCViewer" icon on Volumes page.

2- But after that, it is waiting forever to open the volume content on Kubeflow UI and says "The PVCViewer is currently starting"

3- When i checked the details from the CLI, The pods that belong to the notebook and pvcviewer are created and they are on different nodes.

$ k get po -A -o wide | grep e2e-mlops
kubeflow-user-example-com              e2e-mlops-0                                      2/2     Running   0  43h     10.2.11.25   node-3
kubeflow-user-example-com              pvcviewer-e2e-mlops-workspace-7fbcb66bc7-sd2xd   0/2     Init:0/1  0  2m33s   <none>       node-2

4- When i describe pvcviewer-e2e-mlops-workspace-7fbcb66bc7-sd2xd pod, i can see that there is "Multi-Attach error"

$ k describe po -n kubeflow-user-example-com pvcviewer-e2e-mlops-workspace-7fbcb66bc7-sd2xd

Events:
  Type     Reason              Age                 From                     Message
  ----     ------              ----                ----                     -------
  Warning  FailedAttachVolume  8m4s                attachdetach-controller  Multi-Attach error for volume "pvc-d3e10e54-33ee-4832-97b4-ee7895ddb471" Volume is already used by pod(s) e2e-mlops-0
  Warning  FailedMount         89s (x3 over 6m1s)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[viewer-volume], unattached volumes=[viewer-volume], failed to process volumes=[]: timed out waiting for the condition

5- When i check the related persistent volume

$ k get pv pvc-d3e10e54-33ee-4832-97b4-ee7895ddb471 -o wide
NAME                                      CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                                          STORAGECLASS
pvc-d3e10e54-33ee-4832-97b4-ee7895ddb471  5Gi       RWO           Delete          Bound   kubeflow-user-example-com/e2e-mlops-workspace  csi-rbd-sc

5- The problem here is;
csi-rbd-sc storage class can only create a volume with RWO access mode so second pod (pvcviewer-e2e-mlops-workspace-7fbcb66bc7-sd2xd) can not connect the volume and gets Multi-Attach error.

The only possible solution here is I must configure pvcviewer-controller-manager to create pvcviewer pod in the exact same node with notebook server pod (e2e-mlops-0). I tested it with manuel configuration and it works.

@thesuperzapper can you track a fix?

[andreyvelich]

/transfer dashboard