Volume deletion is not idempotent with CSI migration

[jsafrane]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug

What happened:
When the CSI driver receives multiple DeleteVolume calls for a single in-tree PV migrated to CSI

1. The first call succeeds and deletes the volume
2. The second call fails with an error:

2024-07-24T18:38:59.521133288Z vanilla/controller.go:2092 failed to get VolumeID from volumeMigrationService for volumePath: "[vsanDatastore-2] be583566-a662-6970-2903-3cfdfe8f21e0/_006c/_005a/c01a7c7317a24e0b934b5c233dfad560.vmdk"
sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).DeleteVolume.func1
        /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2092
sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).DeleteVolume
        /go/src/github.com/kubernetes-sigs/vsphere-csi-driver/pkg/csi/service/vanilla/controller.go:2166
github.com/container-storage-interface/spec/lib/go/csi._Controller_DeleteVolume_Handler

What you expected to happen:
The second call should be idempotent and succeed.

How to reproduce it (as minimally and precisely as possible):
When using external-provisioner v5.0.1 which has a bug that it calls DeleteVolume multiple times. That's a bug on the provisioner side, but even when it gets fixed, the external-provisioner can send DeleteVolume multiple times, it's allowed in CSI and it will happen e.g. during container updates or node drains.

[xing-yang]

cc @divyenpatel

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jsafrane]

/remove-lifecycle stale