[BUG] delete volume job is failed

[huangguoqiang]

Uploading juicefs-csi-controller-0_juicefs-plugin.log…
What happened:
delete volume job is failed

What you expected to happen:
quickly delete volume

Anything else we need to know?
this log :

I0531 17:16:30.687477       7 pod_mount.go:449] waitUtilJobCompleted: Job juicefs-4370bc01-delvol is completed
I0531 17:16:34.692467       7 pod_mount.go:449] waitUtilJobCompleted: Job juicefs-17089f91-delvol is completed
E0531 17:16:42.688278       7 provisioner.go:218] provisioner: delete vol error waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:16:42.688306       7 controller.go:1482] delete "pvc-5c8882a3-847b-40e4-880d-ef2913579a54": volume deletion failed: unable to provision delete volume: waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:16:42.688375       7 controller.go:1031] error syncing volume "pvc-5c8882a3-847b-40e4-880d-ef2913579a54": unable to provision delete volume: waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] <INFO>: Volume is formatted as {
I0531 17:16:42.688387       7 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-5c8882a3-847b-40e4-880d-ef2913579a54", UID:"5a28a7a6-af1a-438d-9738-cad80daafd62", APIVersion:"v1", ResourceVersion:"370197729", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' unable to provision delete volume: waitUtilJobCompleted: job juicefs-26aa4266-delvol isn't completed: 2024/05/31 17:16:03.714372 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:16:57.087501       7 provisioner.go:218] provisioner: delete vol error waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:16:57.087523       7 controller.go:1482] delete "pvc-e36fcd01-b570-4617-b4ba-0bac10360324": volume deletion failed: unable to provision delete volume: waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:16:57.087568       7 controller.go:1031] error syncing volume "pvc-e36fcd01-b570-4617-b4ba-0bac10360324": unable to provision delete volume: waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] <INFO>: Volume is formatted as {
I0531 17:16:57.087614       7 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-e36fcd01-b570-4617-b4ba-0bac10360324", UID:"2f14a035-634a-4796-891a-37b81417d27e", APIVersion:"v1", ResourceVersion:"370190284", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' unable to provision delete volume: waitUtilJobCompleted: job juicefs-fc0876fe-delvol isn't completed: 2024/05/31 17:16:18.103205 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:17:12.288128       7 provisioner.go:218] provisioner: delete vol error waitUtilJobCompleted: job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] <INFO>: Volume is formatted as {
job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] <INFO>: Volume is formatted as {
E0531 17:17:12.288184       7 controller.go:1031] error syncing volume "pvc-8ce7e37d-9540-432e-9763-448c0dd88016": unable to provision delete volume: waitUtilJobCompleted: job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] <INFO>: Volume is formatted as {
I0531 17:17:12.288416       7 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-8ce7e37d-9540-432e-9763-448c0dd88016", UID:"49b36f09-97ca-4b24-b06d-4cb3cd2df1b3", APIVersion:"v1", ResourceVersion:"358856640", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' unable to provision delete volume: waitUtilJobCompleted: job juicefs-fec29e38-delvol isn't completed: 2024/05/31 17:16:33.074298 juicefs[7] <INFO>: Volume is formatted as {

and event show log:

26m         Normal    Killing             pod/juicefs-rke-10-5-40-134-pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc   Stopping container jfs-mount
26m         Warning   FailedPreStopHook   pod/juicefs-rke-10-5-40-134-pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc   Exec lifecycle hook ([sh -c umount /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc -l && rmdir /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc]) for Container "jfs-mount" in Pod "juicefs-rke-10-5-40-134-pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc_juicefs(b7736bc5-d5c4-46e9-9cf7-51239a8206e2)" failed - error: command 'sh -c umount /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc -l && rmdir /jfs/pvc-18179844-afee-407d-ad02-d26e6653e197-hxiesc' exited with 137: , message: ""
60m         Normal    Killing             pod/juicefs-rke-10-5-40-134-pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu   Stopping container jfs-mount
60m         Warning   FailedPreStopHook   pod/juicefs-rke-10-5-40-134-pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu   Exec lifecycle hook ([sh -c umount /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu -l && rmdir /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu]) for Container "jfs-mount" in Pod "juicefs-rke-10-5-40-134-pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu_juicefs(29b2375f-9736-45df-bcfb-77871141365a)" failed - error: command 'sh -c umount /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu -l && rmdir /jfs/pvc-1e61ed83-87af-45a0-a8ae-6267cb4f7cc3-sloseu' exited with 137: , message: ""
60m         Normal    Pulled              pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg   Container image "juicedata/mount:ce-v1.1.1" already present on machine
60m         Normal    Created             pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg   Created container jfs-mount
60m         Normal    Started             pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg   Started container jfs-mount
59m         Normal    Killing             pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg   Stopping container jfs-mount
59m         Warning   FailedPreStopHook   pod/juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg   Exec lifecycle hook ([sh -c umount /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg -l && rmdir /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg]) for Container "jfs-mount" in Pod "juicefs-rke-10-5-40-134-pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg_juicefs(e5d67dad-a222-4f1f-b894-f7f60368f41f)" failed - error: command 'sh -c umount /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg -l && rmdir /jfs/pvc-27036fcd-4abc-4219-a938-b84752adab0d-djvmmg' exited with 137: , message: ""

Environment:

• JuiceFS CSI Driver version (which image tag did your CSI Driver use): 0.19.5
• Kubernetes version (e.g. kubectl version): 1.21.11
• Object storage (cloud provider and region): ceph s3
• Metadata engine info (version, cloud provider managed or self maintained): redis senstinel
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others: