Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default force deletion to grace-period=0 exposes risk of data corruption #2819

Open
xiesheng211 opened this issue Jul 31, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@xiesheng211
Copy link

xiesheng211 commented Jul 31, 2024

Describe the bug
#1883 changed the grace-period default from 1 to 0. This is a dangerous & unrecommended operation, and we've seen enterprise operations using k9s to trigger force delete and caused data corruption/lose for stateful workloads.

Force delete has very different implication between grace-period=0 vs. >0, with 0 it will basically skip validating that the pod is successfully SIGKILLed before removing the Pod object from the API server. This very dangerous for stateful workload, where in case of split-brain (e.g. kubelet down) a new Pod can get created (e.g. by statefulset controller) with the same "identity".

kubectl also defaults to non-zero and give very clear warning if user want to explicitly set to 0: https://github.com/kubernetes/kubectl/blob/master/pkg/cmd/delete/delete.go#L62C3-L67C71

Some related articles why force delete could be problematic: https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/#statefulset-considerations

Expected behavior
Default grace-period to 1.
Bonus: provide a more hidden way to trigger grace-period=0, with enough warning to let user aware of the danger & underlying implication.

@derailed derailed added the enhancement New feature or request label Aug 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants