Notebook pods get stuck in terminating

[Jose-Matsuda]

Hi kubeflow team

We're currently using kubeflow 1.7 on a k8s 1.29 (in the process of going to kf 1.9) and we've noticed that some notebooks don't terminate after being sent the delete from the frontend.

This isn't something that we've experienced in our past few years of using kf (albeit in the past we have not had the amount of users that we have now) as every time we've submitted the delete the pod terminated and all the resources cleaned themselves up.

Something odd that we've noticed is that the a finalizer gets added to the pod metadata and as a result it cannot clean itself up. Normally when a notebook is slated for deletion we will see that finalizer added to both the notebook and statefulset but it somehow appears in the pod.
At this stage we cannot edit the pod metadata to remove the finalizer and must force delete it to get it out.

I've exported a gist containing the notebook, statefulset, and pod spec here

Any help would be greatly appreciated!