upgrade k8s to 1.24 (or later) on dev and prod

[mbjones]

Our infrastructure is currently running k8s v 1.23.3 on both dev and prod clusters. Due to changes in how serviceaccount tokens are handled, we need to upgrade to 1.24 or 1.25 on both environments. In addition, 1.24 finalized the move away from support for dockershim, we will need to move to a new container runtime at the same time that we do the upgrade. From my reading, it seems like containerd directly is a good choice, but we should consider others. As we already have containerd installed, it might just be a reconfiguration away from using dockershim.

Because we are currently not running in a high availability configuration, this upgrade will probably require downtime as we upgrade the control plane node. Once it is upgraded, the worker nodes should be able to be updated without further downtime.

• Decide whether to upgrade to 1.24 or to continue on to 1.25
• Decide on a container runtime
• Upgrade dev-k8s
  • reconfigure to use container runtime
  • control plane
  • worker nodes
  • test deployments for compatibility
• Upgrade dev-k8s
  • reconfigure to use new container runtime
  • control plane
  • worker nodes

[mbjones]

Related to upgrading the OS on these hosts, adding additional nodes to the clusters (#31), and reconfiguring for a HA control plane (#1).

[nickatnceas]

https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/find-out-runtime-you-use/#which-endpoint

Confirmed that we are currently running dockershim socket with Docker Engine:

outin@k8s-dev-node-1:~$ tr \\0 ' ' < /proc/"$(pgrep kubelet)"/cmdline
/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.6

[nickatnceas]

Kubernetes provides migration docs for containerd and cri-dockerd (and notably no others in this section). Both containerd and docker have supported packages provided by Ubuntu's package repos, though containerd is in main with longer support from core Ubuntu devs, while docker.io is in universe.

I'm leaning towards containerd due to it being included in main.

https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/
https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/migrate-dockershim-dockerd/

[mbjones]

I'm generally a fan of containerd and used it for quite a long time as my main docker runtime on my mac. I think its solid.