Enable graceful node shutdown for 1.30+ on AL2

[cartermckinnon]

Description of changes:

Configures a node shutdown period of 45 seconds, with the last 15 seconds reserved for critical pods, for Kubernetes versions 1.30 and above.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[cartermckinnon]

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

Kubernetes version	Build	Launch	Test
1.23	failure ❌	skipped ⏭️	skipped ⏭️
1.24	failure ❌	skipped ⏭️	skipped ⏭️
1.25	failure ❌	skipped ⏭️	skipped ⏭️
1.26	failure ❌	skipped ⏭️	skipped ⏭️
1.27	failure ❌	skipped ⏭️	skipped ⏭️
1.28	failure ❌	skipped ⏭️	skipped ⏭️

[cartermckinnon]

CI is broken after Packer finally pulled the plug on builtin plugins. Should be fixed by #1545 .

[cartermckinnon]

/ci

let's see if that did it...

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

Kubernetes version	Build	Launch	Test
1.23	success ✅	success ✅	success ✅
1.24	success ✅	success ✅	success ✅
1.25	success ✅	success ✅	success ✅
1.26	success ✅	success ✅	success ✅
1.27	success ✅	success ✅	success ✅
1.28	success ✅	success ✅	success ✅

[cartermckinnon]

In my tests, this config didn't seem sufficient to actually block shutdown. I think we need to increase logind's InhibitDelayMaxSec for this to have any effect. Kubelet will attempt to increase it itself, but maybe this doesn't work on AL2?

[youwalther65]

@cartermckinnon EBS CSI FAQ troubleshooting section here proposes InhibitDelayMaxSec as well and following it I got:

#  journalctl  -u kubelet | grep -i shutdown
Apr 09 08:02:36 ip-10-0-136-54.eu-west-1.compute.internal kubelet[1011931]: I0409 08:02:36.347456 1011931 nodeshutdown_manager_linux.go:137] "Creating node shutdown manager" shutdownGracePeriodRequested="45s" shutdownGracePeriodCriticalPods="15s" shutdownGracePeriodByPodPriority=[{"Priority":0,"ShutdownGracePeriodSeconds":30},{"Priority":2000000000,"ShutdownGracePeriodSeconds":15}]

# systemd-inhibit --list
     Who: kubelet (UID 0/root, PID 1011931/kubelet)
    What: shutdown
     Why: Kubelet needs time to handle node shutdown
    Mode: delay

1 inhibitors listed.

But I still wonder how Karpenter or MNG node drain plays together with kubelet graceful shutdown.

[cartermckinnon]

But I still wonder how Karpenter or MNG node drain plays together with kubelet graceful shutdown.

MNG and Karpenter will evict all the pods from a node before shutting it down, so you don't really need kubelet's graceful shutdown feature by the time the shutdown signal arrives (though some daemonsets could still benefit from it). The graceful shutdown feature is intended to honor your pods' own termination grace periods and cleanly shut them down when e.g. there is no orchestrator handling that for you (such as when the shutdown is unexpected).

[cartermckinnon]

/ci
+workflow:os_distros al2

[cartermckinnon]

/ci
+workflow:os_distros al2

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.21 / al2	success ✅	success ✅
1.22 / al2	success ✅	success ✅
1.23 / al2	success ✅	success ✅
1.24 / al2	success ✅	success ✅
1.25 / al2	success ✅	success ✅
1.26 / al2	success ✅	success ✅
1.27 / al2	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.29 / al2	success ✅	success ✅

[cartermckinnon]

/ci
+workflow:os_distros al2
+workflow:k8s_versions 1.30

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.30 / al2	failure ❌	skipped ⏭️

[youwalther65]

But I still wonder how Karpenter or MNG node drain plays together with kubelet graceful shutdown.

MNG and Karpenter will evict all the pods from a node before shutting it down, so you don't really need kubelet's graceful shutdown feature by the time the shutdown signal arrives (though some daemonsets could still benefit from it). The graceful shutdown feature is intended to honor your pods' own termination grace periods and cleanly shut them down when e.g. there is no orchestrator handling that for you (such as when the shutdown is unexpected).

Agree, especially DaemonSets like EBS CSI would benefit from kubelet graceful node shutdown and the team even recommends it in its FAQ

[cartermckinnon]

/ci
+workflow:k8s_versions 1.30

[stevehipwell]

@cartermckinnon is this blocked for a reason? Also what about AL2023?

I was just looking through the code for Bottlerocket and I can't see any handling for InhibitDelayMaxSec despite this being supported there, but I suspect that this should be configurable given the default termination grace period for a K8s pod is 300s?