Newbie installation question: cni plugin not initialized

[kwoot]

Hi,

I run K3S in 3 vm's on a proxmox server.
I followed the K3S quickstart instructions. One server, 2 agents.
The node list is correct but everything stays stuck in NotReady.
When I descibre the nodes, and also when I systemctl status k3s-agent I see the following error:

KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

I guess this must be some kind of RTFM. My problem is: which one?

Any and all hints are highly appreciated.

Regards,
Jeroen

[brandond]

That generally just means that the CNI pod isn't running on that node yet. Check kubectl get pod -A output and look for a CNI pod on that node. If it's not running, describe the pod and check the pod logs to see why.

[brandond]

Sorry, forgot which repo I was on - you won't see a CNI pod on k3s, but disabling flannel would definitely do what you're reporting! Glad you figured it out.

[kwoot]

Thanks. Yes, I'm a complete newbie at this. Now trying to figure out how to get AWX running properly on this. The challenge continues! :-)

[kwoot]

Hi!

Well, no cni pod in sight. Since this is a fresh install, could this be a bug?

# k3s kubectl get pod -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE
kube-system   local-path-provisioner-957fdf8bc-rpv5k   0/1     Pending   0          18h
kube-system   metrics-server-648b5df564-snvz4          0/1     Pending   0          18h
kube-system   coredns-77ccd57875-2fpwv                 0/1     Pending   0          18h
kube-system   helm-install-traefik-9pgkv               0/1     Pending   0          18h
kube-system   helm-install-traefik-crd-9mfd2           0/1     Pending   0          18h
# kubectl get nodes -o wide
NAME    STATUS     ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k3sm1   NotReady   control-plane,master   18h   v1.27.6+k3s1   10.1.1.48     <none>        Ubuntu 22.04.3 LTS   5.15.0-84-generic   containerd://1.7.6-k3s1.27
k3sa2   NotReady   <none>                 18h   v1.27.6+k3s1   10.1.1.50     <none>        Ubuntu 22.04.3 LTS   5.15.0-84-generic   containerd://1.7.6-k3s1.27
k3sa1   NotReady   <none>                 18h   v1.27.6+k3s1   10.1.1.49     <none>        Ubuntu 22.04.3 LTS   5.15.0-84-generic   containerd://1.7.6-k3s1.27

Hm looking at containerd log/var/lib/rancher/k3s/agent/containerd/containerd.log.
I guess the answer must be somewhere in this garbled mess :-)

time="2023-09-28T08:34:04.474307792+02:00" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:true] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreBlockIONotEnabledErrors:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/opt/cni/bin NetworkPluginConfDir:/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginSetupSerially:false NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath: Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:10010 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:rancher/mirrored-pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:true EnableUnprivilegedICMP:true EnableCDI:false CDISpecDirs:[/etc/cdi /var/run/cdi] ImagePullProgressTimeout:1m0s DrainExecSyncIOTimeout:0s} ContainerdRootDir:/var/lib/rancher/k3s/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}"
time="2023-09-28T08:34:04.474418026+02:00" level=info msg="Connect containerd service"
time="2023-09-28T08:34:04.474488452+02:00" level=info msg="using legacy CRI server"
time="2023-09-28T08:34:04.474507883+02:00" level=info msg="using experimental NRI integration - disable nri plugin to prevent this"
time="2023-09-28T08:34:04.474581628+02:00" level=info msg="Get image filesystem path \"/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs\""
time="2023-09-28T08:34:04.531518185+02:00" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
time="2023-09-28T08:34:04.532094928+02:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock.ttrpc
time="2023-09-28T08:34:04.532167324+02:00" level=info msg=serving... address=/run/k3s/containerd/containerd.sock
time="2023-09-28T08:34:04.532377778+02:00" level=info msg="Start subscribing containerd event"
time="2023-09-28T08:34:04.532450061+02:00" level=info msg="Start recovering state"
time="2023-09-28T08:34:04.532606460+02:00" level=info msg="Start event monitor"
time="2023-09-28T08:34:04.532709981+02:00" level=info msg="Start snapshots syncer"
time="2023-09-28T08:34:04.532744933+02:00" level=info msg="Start cni network conf syncer for default"
time="2023-09-28T08:34:04.532762064+02:00" level=info msg="Start streaming server"
time="2023-09-28T08:34:04.532810973+02:00" level=info msg="containerd successfully booted in 0.167174s"
time="2023-09-28T08:34:10.667264217+02:00" level=info msg="No cni config template is specified, wait for other system components to drop the config."

[kwoot]

Of course it was a PEBKAC. Turned out I had not followed the quickstart instructions at https://docs.k3s.io/quick-start.
Instead I read the page at https://docs.k3s.io/installation/configuration and did the --flannel-backend none option.
Sorry to have waisted somebody's time.

[tbernacchi]

Hi @kwoot @brandond - I believe I'm facing the same issue here. I've setup my k3s cluster with this command:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --flannel-backend none" sh -s -

The installation process it's done successfully, but when I check the nodes it's in NotReady state:

root@raspberrypi4-4:~# k get nodes -o wide
NAME             STATUS     ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
raspberrypi4-4   NotReady   control-plane,master   16m   v1.29.6+k3s1   192.168.0.105   <none>        Ubuntu 24.04 LTS   6.8.0-1006-raspi   containerd://1.7.17-k3s1

Looking at the k3s.service I see a lot of Container runtime network not ready messages:

Jul 04 20:10:46 raspberrypi4-4 k3s[2172]: E0704 20:10:46.136880    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:10:51 raspberrypi4-4 k3s[2172]: E0704 20:10:51.139347    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:10:56 raspberrypi4-4 k3s[2172]: E0704 20:10:56.141538    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:01 raspberrypi4-4 k3s[2172]: E0704 20:11:01.143896    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:05 raspberrypi4-4 k3s[2172]: E0704 20:11:05.935709    2172 resource_quota_controller.go:440] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1
Jul 04 20:11:06 raspberrypi4-4 k3s[2172]: E0704 20:11:06.146518    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:06 raspberrypi4-4 k3s[2172]: I0704 20:11:06.613514    2172 garbagecollector.go:835] "failed to discover some groups" groups="<internal error: json: unsupported type: map[schema.GroupVersion]error>"
Jul 04 20:11:11 raspberrypi4-4 k3s[2172]: E0704 20:11:11.148889    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:16 raspberrypi4-4 k3s[2172]: E0704 20:11:16.151590    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:21 raspberrypi4-4 k3s[2172]: E0704 20:11:21.153658    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:26 raspberrypi4-4 k3s[2172]: E0704 20:11:26.156641    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:31 raspberrypi4-4 k3s[2172]: E0704 20:11:31.159741    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:35 raspberrypi4-4 k3s[2172]: E0704 20:11:35.950935    2172 resource_quota_controller.go:440] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1
Jul 04 20:11:36 raspberrypi4-4 k3s[2172]: E0704 20:11:36.162330    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:36 raspberrypi4-4 k3s[2172]: I0704 20:11:36.638848    2172 garbagecollector.go:835] "failed to discover some groups" groups="<internal error: json: unsupported type: map[schema.GroupVersion]error>"
Jul 04 20:11:41 raspberrypi4-4 k3s[2172]: E0704 20:11:41.165899    2172 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plug>
Jul 04 20:11:41 raspberrypi4-4 k3s[2172]: E0704 20:11:41.359740    2172 handler_proxy.go:137] error resolving kube-system/metrics-server: no endpoints available for service "metrics-server"

Any ideas?

Any help will be appreciate.

Thank you

[brandond]

It's not ready because you don't have a CNI. Did you want to install one?

[tbernacchi]

Yes - I'm trying to install Cilium. I'm following this https://docs.cilium.io/en/stable/installation/k3s/ - with no luck

[brandond]

I would defer to their docs. The node won't be ready until your replacement CNI is successful installed.

[tbernacchi]

This is an odd situation. In my first attempt I chose the flag --flannel-backend none to not install flannel as CNI and the cluster didn't boot up, and the Container runtime network not ready message makes sense. My second attempt without the flag:

root@raspberrypi4-4:~# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s -
[INFO]  Finding release for channel stable
[INFO]  Using v1.29.6+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.29.6+k3s1/sha256sum-arm64.txt
[INFO]  Skipping binary downloaded, installed k3s matches hash
[INFO]  Skipping installation of SELinux RPM
[INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kubectl
[INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists
[INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s

root@raspberrypi4-4:~# k get nodes -o wide
NAME             STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
raspberrypi4-4   Ready    control-plane,master   3s    v1.29.6+k3s1   192.168.0.105   <none>        Ubuntu 24.04 LTS   6.8.0-1006-raspi   containerd://1.7.17-k3s1

root@raspberrypi4-4:~# k get pods -n kube-system -owide -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS   AGE     IP          NODE             NOMINATED NODE   READINESS GATES
kube-system   coredns-6799fbcd5-bgl4n                  1/1     Running   0          9m34s   10.42.0.5   raspberrypi4-4   <none>           <none>
kube-system   local-path-provisioner-6f5d79df6-9c87l   1/1     Running   0          9m33s   10.42.0.4   raspberrypi4-4   <none>           <none>
kube-system   metrics-server-54fd9b65b-hbd6j           1/1     Running   0          9m34s   10.42.0.3   raspberrypi4-4   <none>           <none>
kube-system   svclb-traefik-9eb16f95-nlczs             2/2     Running   0          20s     10.42.0.2   raspberrypi4-4   <none>           <none>
kube-system   traefik-7d5f6474df-55mhj                 1/1     Running   0          9m34s   10.42.0.6   raspberrypi4-4   <none>           <none>

Cilium

root@raspberrypi4-4:~# helm install cilium cilium/cilium --version=1.15.6 --set global.tag="v1.15.6" --set global.containerRuntime.integration="containerd" --set global.containerRuntime.socketPath="/var/run/k3s/containerd/containerd.sock" --set global.kubeProxyReplacement="strict" --namespace kube-system
Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "https://192.168.0.105:6443/version": tls: failed to verify certificate: x509: certificate signed by unknown authority

root@raspberrypi4-4:~# cilium install --version 1.15.6
ℹ️  Using Cilium version 1.15.6
🔮 Auto-detected cluster name: kubernetes
⏭️ Skipping auto kube-proxy detection

Error: Unable to install Cilium: Kubernetes cluster unreachable: Get "https://192.168.0.105:6443/version": tls: failed to verify certificate: x509: certificate signed by unknown authority

Any ideas how to tackle this?

[tbernacchi]

[UPDATE] - For some reason the /etc/rancher/k3s/k3s.yaml was pointing to 127.0.0.1 on server::

I've changed that to my internal IP and everything went well:

server: https://192.168.0.105:6443

root@raspberrypi4-4:~# cilium install --version 1.15.6 --set=ipam.operator.clusterPoolIPv4PodCIDRList="10.42.0.0/16"
🔮 Auto-detected Kubernetes kind: K3s
ℹ️  Using Cilium version 1.15.6
🔮 Auto-detected cluster name: default

root@raspberrypi4-4:~# k get pods -A
NAMESPACE     NAME                                     READY   STATUS    RESTARTS     AGE
kube-system   cilium-operator-f45f4975f-4h56n          1/1     Running   0            2m35s
kube-system   cilium-xmlch                             1/1     Running   0            2m35s
kube-system   coredns-6799fbcd5-q8m5x                  1/1     Running   0            2m16s
kube-system   local-path-provisioner-6f5d79df6-9c87l   1/1     Running   1 (7m ago)   25m
kube-system   metrics-server-54fd9b65b-hbd6j           1/1     Running   1 (7m ago)   25m
kube-system   svclb-traefik-9eb16f95-nlczs             2/2     Running   2 (7m ago)   16m
kube-system   traefik-7d5f6474df-55mhj                 1/1     Running   1 (7m ago)   25m

root@raspberrypi4-4:~# k get nodes -w
NAME             STATUS   ROLES                  AGE   VERSION
raspberrypi4-4   Ready    control-plane,master   16m   v1.29.6+k3s1

Life it's good again =)