Multi-node kubernetes clusters

[afbjorklund]

Description

Currently the lima clusters, in k3s.yaml and k8s.yaml, set up a single-node cluster.

The control plane node is "tainted", to allow running user application pods on it:

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

If networking between two instances is set up, it is possible to add additional nodes.

This would implicitly require deploying a multi-host CNI, e.g. flannel - not bridge

Discussion:

• Adding worker node to K8s cluster created using Lima #1181

Initial PR:

• Add feature to install kubernetes worker nodes #1183

[afbjorklund]

The template will have a "boolean", if it is a control plane node or a worker node.
(In order to add more worker nodes, you will first need to have set up a control plane)

In k3s these are called "server" and "agent", since they run more than just kubernetes.
But the same concept applies. In k8s, they were previously called "master" and "minion".

[afbjorklund]

The "k8s.sh" script will follow the same pattern as the "k3s.sh" script is doing...

If there is a K8S_TOKEN, it is a worker node. Otherwise, it is a control plane node.

examples/k3s.yaml-    # To install a worker node, fill in and uncomment this line:
examples/k3s.yaml:    #export K3S_TOKEN=xxx K3S_URL=https://server-url:6443
--
examples/k8s.yaml-    # To install a worker node, fill in and uncomment this line:
examples/k8s.yaml:    #export K8S_URL=server-url:6443 K8S_TOKEN=xxx K8S_HASH=sha256:<hash>

Currently there is no good way of automatically set up the token, for the workers.

You will have to read the comments, run the command, and edit the template...

examples/k3s.yaml:    # To get the token to use for the variable, run this command:
examples/k3s.yaml-    # $ limactl shell k3s sudo cat /var/lib/rancher/k3s/server/node-token
--
examples/k8s.yaml:    # To get the token to use for the variable, run this command:
examples/k8s.yaml-    # $ limactl shell k8s sudo kubeadm token create --print-join-command

[afbjorklund]

I updated the k3d and kind examples, as an alternative to running multiple virtual machines.

https://github.com/afbjorklund/lima/blob/k3d/examples/k3d.yaml

NAME                     STATUS   ROLES                  AGE   VERSION
k3d-mycluster-agent-0    Ready    <none>                 23s   v1.24.4+k3s1
k3d-mycluster-agent-2    Ready    <none>                 22s   v1.24.4+k3s1
k3d-mycluster-agent-1    Ready    <none>                 22s   v1.24.4+k3s1
k3d-mycluster-server-0   Ready    control-plane,master   28s   v1.24.4+k3s1

https://github.com/afbjorklund/lima/blob/kind/examples/kind.yaml

NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   32s   v1.26.0
kind-worker          Ready    <none>          12s   v1.26.0
kind-worker2         Ready    <none>          11s   v1.26.0
kind-worker3         Ready    <none>          12s   v1.26.0

Each node will run as a privileged system docker container, in the same shared virtual machine.

https://k3d.io/stable/#quick-start

CONTAINER ID   IMAGE                            COMMAND                  CREATED          STATUS          PORTS                             NAMES
da9eb00191f3   ghcr.io/k3d-io/k3d-proxy:5.4.6   "/bin/sh -c nginx-pr…"   53 seconds ago   Up 41 seconds   80/tcp, 0.0.0.0:42067->6443/tcp   k3d-mycluster-serverlb
f0bc463ecbbd   rancher/k3s:v1.24.4-k3s1         "/bin/k3d-entrypoint…"   58 seconds ago   Up 46 seconds                                     k3d-mycluster-agent-2
beabbddb25a1   rancher/k3s:v1.24.4-k3s1         "/bin/k3d-entrypoint…"   58 seconds ago   Up 46 seconds                                     k3d-mycluster-agent-1
5198cfbfa486   rancher/k3s:v1.24.4-k3s1         "/bin/k3d-entrypoint…"   58 seconds ago   Up 47 seconds                                     k3d-mycluster-agent-0
766aee74bd69   rancher/k3s:v1.24.4-k3s1         "/bin/k3d-entrypoint…"   58 seconds ago   Up 52 seconds                                     k3d-mycluster-server-0

https://kind.sigs.k8s.io/docs/user/quick-start/#multinode-clusters

CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS          PORTS                       NAMES
a22ba4d36cce   kindest/node:v1.26.0   "/usr/local/bin/entr…"   About a minute ago   Up 58 seconds                               kind-worker2
c23ba3f0a94f   kindest/node:v1.26.0   "/usr/local/bin/entr…"   About a minute ago   Up 58 seconds                               kind-worker
b6f7272e03c0   kindest/node:v1.26.0   "/usr/local/bin/entr…"   About a minute ago   Up 59 seconds   127.0.0.1:46697->6443/tcp   kind-control-plane
fed5f8b56507   kindest/node:v1.26.0   "/usr/local/bin/entr…"   About a minute ago   Up 58 seconds                               kind-worker3

The nodes will use dockerd to run, but use containerd for the Kubernetes container runtime (CR).

The host will not need any docker installation (only the client, if desired - but not Docker Desktop).

[afbjorklund]

The main issue with the virtual machines is the VM networking and setting the node IP for each.

It also requires more resources, compared with overprovisioning containers on the same host...