csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s) #765

black-dragon74 · 2025-02-13T10:42:12Z

This patch adds a watcher to csi-addons sidecar which is responsible for re-creating CSIAddonNode(s) in case they
are deleted manually.

Fixes: #762

black-dragon74 · 2025-02-13T10:50:34Z

Logs from testing


// Delete CephFS CSIAddonsNode
❯ oc delete csiaddonsnode/minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner
csiaddonsnode.csiaddons.openshift.io "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner" deleted

///
// Logs
//

// CephFS sidecar
...
2025-02-14T10:35:38.438Z        INFO    Starting watcher for CSIAddonsNode: minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner
2025-02-14T10:37:31.852Z        INFO    WARNING: An active CSIAddonsNode: minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner was deleted, it will be recreated
2025-02-14T10:37:31.895Z        INFO    CSIAddonsNode: minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner recreated
...

// CSI Addons controller
...
2025-02-14T10:37:31.835Z        INFO    Deleting connection     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "897863db-e488-4c14-a71f-0526cfcab4a2", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070", "Key": "rook-ceph/csi-cephfsplugin-provisioner-6fff6848bf-hg4nm"}
2025-02-14T10:37:31.836Z        INFO    Removing finalizer      {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "897863db-e488-4c14-a71f-0526cfcab4a2", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070"}
2025-02-14T10:37:31.852Z        INFO    CSIAddonsNode resource not found        {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "fe82a4bc-4adc-46c0-bab2-f4674210c6db"}
2025-02-14T10:37:31.893Z        INFO    Adding finalizer        {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "c1232f97-9b1b-414b-ab0b-867e14918f6c", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070"}
2025-02-14T10:37:31.904Z        INFO    Connecting to sidecar   {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "c1232f97-9b1b-414b-ab0b-867e14918f6c", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070"}
2025-02-14T10:37:31.918Z        INFO    No networkfenceclasses found in annotation      {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "c1232f97-9b1b-414b-ab0b-867e14918f6c", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070"}
2025-02-14T10:37:31.918Z        INFO    Successfully connected to sidecar       {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "c1232f97-9b1b-414b-ab0b-867e14918f6c", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070"}
2025-02-14T10:37:31.918Z        INFO    Added connection to connection pool     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner", "reconcileID": "c1232f97-9b1b-414b-ab0b-867e14918f6c", "NodeID": "minikube", "DriverName": "rook-ceph.cephfs.csi.ceph.com", "EndPoint": "10.244.0.20:9070", "Key": "rook-ceph/csi-cephfsplugin-provisioner-6fff6848bf-hg4nm"}
...

#########################################################################

// Delete RBD CSIAddonsNode
❯ oc delete csiaddonsnode/minikube-rook-ceph-deployment-csi-rbdplugin-provisioner
csiaddonsnode.csiaddons.openshift.io "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner" deleted

//
// Logs
//

// RBD Sidecar
...
2025-02-14T10:35:40.473Z        INFO    Starting watcher for CSIAddonsNode: minikube-rook-ceph-deployment-csi-rbdplugin-provisioner
2025-02-14T10:37:38.026Z        INFO    WARNING: An active CSIAddonsNode: minikube-rook-ceph-deployment-csi-rbdplugin-provisioner was deleted, it will be recreated
2025-02-14T10:37:38.051Z        INFO    CSIAddonsNode: minikube-rook-ceph-deployment-csi-rbdplugin-provisioner recreated
...

// CSI Addons controller
...
2025-02-14T10:37:38.013Z        INFO    Deleting connection     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "66037391-e262-4a69-89ac-3b18c2ec404e", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070", "Key": "rook-ceph/csi-rbdplugin-provisioner-57d7d9fc45-2s7fv"}
2025-02-14T10:37:38.013Z        INFO    Removing finalizer      {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "66037391-e262-4a69-89ac-3b18c2ec404e", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070"}
2025-02-14T10:37:38.026Z        INFO    CSIAddonsNode resource not found        {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "aa6ee9a0-3b7f-43e9-b682-2af18640355c"}
2025-02-14T10:37:38.052Z        INFO    Adding finalizer        {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "94336eb0-51ec-47d9-9c69-6fe4afd5fb63", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070"}
2025-02-14T10:37:38.061Z        INFO    Connecting to sidecar   {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "94336eb0-51ec-47d9-9c69-6fe4afd5fb63", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070"}
2025-02-14T10:37:38.073Z        INFO    No networkfenceclasses found in annotation      {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "94336eb0-51ec-47d9-9c69-6fe4afd5fb63", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070"}
2025-02-14T10:37:38.073Z        INFO    Successfully connected to sidecar       {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "94336eb0-51ec-47d9-9c69-6fe4afd5fb63", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070"}
2025-02-14T10:37:38.073Z        INFO    Added connection to connection pool     {"controller": "csiaddonsnode", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "CSIAddonsNode", "CSIAddonsNode": {"name":"minikube-rook-ceph-deployment-csi-rbdplugin-provisioner","namespace":"rook-ceph"}, "namespace": "rook-ceph", "name": "minikube-rook-ceph-deployment-csi-rbdplugin-provisioner", "reconcileID": "94336eb0-51ec-47d9-9c69-6fe4afd5fb63", "NodeID": "minikube", "DriverName": "rook-ceph.rbd.csi.ceph.com", "EndPoint": "10.244.0.19:9070", "Key": "rook-ceph/csi-rbdplugin-provisioner-57d7d9fc45-2s7fv"}
...

#########################################################################

// Check if resources are recreated (name and age)
❯ oc get csiaddonsnode
NAME                                                         NAMESPACE   AGE   DRIVERNAME                      ENDPOINT                                                             NODEID
minikube-rook-ceph-deployment-csi-cephfsplugin-provisioner   rook-ceph   12s   rook-ceph.cephfs.csi.ceph.com   pod://csi-cephfsplugin-provisioner-6fff6848bf-hg4nm.rook-ceph:9070   minikube
minikube-rook-ceph-deployment-csi-rbdplugin-provisioner      rook-ceph   5s    rook-ceph.rbd.csi.ceph.com      pod://csi-rbdplugin-provisioner-57d7d9fc45-2s7fv.rook-ceph:9070      minikube

Madhu-1 · 2025-02-13T11:32:18Z

The reconciler should not be creating the CR, it should be done by the sidecar which is creating it earlier, watch for delete event and recreate it from the sidecar.

nixpanic

The reconciler should not be creating the CR, it should be done by the sidecar which is creating it earlier, watch for delete event and recreate it from the sidecar.

Indeed, the sidecar creates the initial CSIAddonsNode, and so should it re-create one if it was deleted (and report a big warning about the unexpected deletion of it).

sidecar/internal/csiaddonsnode/csiaddonsnode.go

nixpanic

The 2nd commit is not related to the actual issue reported, it is cleaner to split it out.

sidecar/main.go

sidecar/internal/csiaddonsnode/csiaddonsnode.go

black-dragon74 · 2025-02-17T08:17:56Z

The 2nd commit is not related to the actual issue reported, it is cleaner to split it out.

Done at #770

sidecar/internal/csiaddonsnode/csiaddonsnode.go

Madhu-1

small nit, LGTM

sidecar/internal/csiaddonsnode/csiaddonsnode.go

Madhu-1

I hope it been tested by restarting the pod many times in a loop and also deleting the CR manually in a loop

This patch adds `watch` and `list` verbs for CSIAddonsNode objects. These permissions are required by the csi-addons sidecar. Ref: csi-addons/kubernetes-csi-addons#765 Signed-off-by: Niraj Yadav <[email protected]>

mergify · 2025-02-17T15:26:02Z

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

Madhu-1 · 2025-02-17T15:35:18Z

@Mergifyio requeue

mergify · 2025-02-17T15:35:24Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

This patch adds a watcher to csi-addons sidecar which is responsible for re-creating CSIAddonNode(s) in case they are deleted manually. Signed-off-by: Niraj Yadav <[email protected]>

This patch adds `watch` and `list` verbs for CSIAddonsNode objects. These permissions are required by the csi-addons sidecar. Ref: csi-addons/kubernetes-csi-addons#765 Signed-off-by: Niraj Yadav <[email protected]> (cherry picked from commit a0d2dc6)

mergify bot requested review from iPraveenParihar, Madhu-1, nixpanic, Rakshith-R and yati1998 February 13, 2025 10:42

nixpanic requested changes Feb 13, 2025

View reviewed changes

black-dragon74 force-pushed the node-recreate branch from 73b2317 to b7d707b Compare February 13, 2025 16:42

black-dragon74 requested a review from nixpanic February 13, 2025 16:44

black-dragon74 force-pushed the node-recreate branch from b7d707b to 6685400 Compare February 13, 2025 16:46

Madhu-1 reviewed Feb 13, 2025

View reviewed changes

sidecar/internal/csiaddonsnode/csiaddonsnode.go Outdated Show resolved Hide resolved

Madhu-1 reviewed Feb 13, 2025

View reviewed changes

sidecar/internal/csiaddonsnode/csiaddonsnode.go Outdated Show resolved Hide resolved

black-dragon74 force-pushed the node-recreate branch 2 times, most recently from 194fa16 to e46f08a Compare February 14, 2025 10:44

black-dragon74 changed the title ~~csiaddonsNode: Recreate if an active sidecar connection is found~~ csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s) Feb 14, 2025

black-dragon74 force-pushed the node-recreate branch 5 times, most recently from bf41ace to 35e0993 Compare February 14, 2025 13:32

nixpanic reviewed Feb 14, 2025

View reviewed changes

sidecar/main.go Outdated Show resolved Hide resolved

black-dragon74 force-pushed the node-recreate branch from 35e0993 to c50213d Compare February 17, 2025 08:06

Madhu-1 requested changes Feb 17, 2025

View reviewed changes

sidecar/main.go Outdated Show resolved Hide resolved

sidecar/internal/csiaddonsnode/csiaddonsnode.go Outdated Show resolved Hide resolved

Madhu-1 reviewed Feb 17, 2025

View reviewed changes

sidecar/internal/csiaddonsnode/csiaddonsnode.go Outdated Show resolved Hide resolved

Madhu-1 reviewed Feb 17, 2025

View reviewed changes

sidecar/internal/csiaddonsnode/csiaddonsnode.go Outdated Show resolved Hide resolved

black-dragon74 force-pushed the node-recreate branch 2 times, most recently from ab56f47 to 957b289 Compare February 17, 2025 08:52

Madhu-1 reviewed Feb 17, 2025

View reviewed changes

sidecar/internal/csiaddonsnode/csiaddonsnode.go Show resolved Hide resolved

sidecar/internal/csiaddonsnode/csiaddonsnode.go Show resolved Hide resolved

black-dragon74 force-pushed the node-recreate branch from 957b289 to aab8694 Compare February 17, 2025 08:57

Madhu-1 approved these changes Feb 17, 2025

View reviewed changes

black-dragon74 mentioned this pull request Feb 17, 2025

csi: update RBACs for CSIAddonsNode ceph/ceph-csi-operator#204

Merged

Madhu-1 requested a review from nixpanic February 17, 2025 09:44

nixpanic approved these changes Feb 17, 2025

View reviewed changes

black-dragon74 force-pushed the node-recreate branch from aab8694 to 398089c Compare February 17, 2025 15:18

csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s)

b46e722

This patch adds a watcher to csi-addons sidecar which is responsible for re-creating CSIAddonNode(s) in case they are deleted manually. Signed-off-by: Niraj Yadav <[email protected]>

black-dragon74 force-pushed the node-recreate branch from 398089c to b46e722 Compare February 17, 2025 15:35

mergify bot merged commit 4cdbe23 into csi-addons:main Feb 17, 2025
15 checks passed

Madhu-1 mentioned this pull request Feb 18, 2025

manifest: Add list and watch capabilities to provisioners rook/rook#15399

Merged

Madhu-1 mentioned this pull request Feb 18, 2025

DFBUGS-848: [release-4.18] csi: update RBACs for CSIAddonsNode red-hat-storage/ceph-csi-operator#82

Merged

black-dragon74 mentioned this pull request Feb 18, 2025

DFBUGS-848: [release-4.18] manifest: add list and watch capabilities to provisioners red-hat-storage/rook#829

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s) #765

csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s) #765

black-dragon74 commented Feb 13, 2025 •

edited

Loading

black-dragon74 commented Feb 13, 2025 •

edited

Loading

Madhu-1 commented Feb 13, 2025

nixpanic left a comment

nixpanic left a comment

black-dragon74 commented Feb 17, 2025

Madhu-1 left a comment

Madhu-1 left a comment

mergify bot commented Feb 17, 2025

Madhu-1 commented Feb 17, 2025

mergify bot commented Feb 17, 2025

csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s) #765

csiaddonsNode: Recreate CSIAddonsNode with active sidecar(s) #765

Conversation

black-dragon74 commented Feb 13, 2025 • edited Loading

black-dragon74 commented Feb 13, 2025 • edited Loading

Logs from testing

Madhu-1 commented Feb 13, 2025

nixpanic left a comment

Choose a reason for hiding this comment

nixpanic left a comment

Choose a reason for hiding this comment

black-dragon74 commented Feb 17, 2025

Madhu-1 left a comment

Choose a reason for hiding this comment

Madhu-1 left a comment

Choose a reason for hiding this comment

mergify bot commented Feb 17, 2025

Madhu-1 commented Feb 17, 2025

mergify bot commented Feb 17, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

black-dragon74 commented Feb 13, 2025 •

edited

Loading

black-dragon74 commented Feb 13, 2025 •

edited

Loading