k3s server kube-system CrashLoopBackOff after installing nftables v1.1.0 #10883
Replies: 5 comments 1 reply
-
|
When you say you're installing it after building it from source, what exactly do you mean? Are you just replacing the binary on the host? Are you perhaps building and installing a package that triggers post install scripts that interfere with Kubernetes networking? It really looks like something you're doing is breaking the flannel or kube-proxy iptables rules. |
Beta Was this translation helpful? Give feedback.
-
Debian 11 , nft version is
Even if assume the cause is just nftables 1.1.0, why after uninstalling it sill pods fails ? I can think if a library dependency that has been overridden by nft 1.1.0 causing the issue for kube-proxy or flannel , for testing this I uninstalled iptable, nft-libs* all of them dpkg -l | grep nft
ii libnftables1:amd64 0.9.8-3.1+deb11u2 amd64 Netfilter nftables high level userspace API library
ii libnftnl-dev:amd64 1.1.9-1 amd64 Development files for libnftnl
ii libnftnl11:amd64 1.1.9-1 amd64 Netfilter nftables userspace API library
ii nftables 0.9.8-3.1+deb11u2 amd64 Program to control packet filtering rules by Netfilter projectbut did not help either Also reinstalling the default Debian 11 nft dd not help :| Any idea to trace more and find the root cause ? |
Beta Was this translation helpful? Give feedback.
-
|
I deleted all pods in kube-system so be created again kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-576bfc4dc7-hfvbx 0/1 Running 0 42s
local-path-provisioner-6795b5f9d8-dmrz2 1/1 Running 1 (15s ago) 46s
metrics-server-557ff575fb-rrjck 1/1 Running 0 23mnow metrics-server is fine , local-path keeps restarting, and coredns does not get ready CoreDNS error Warning Unhealthy 4m16s (x22 over 4m55s) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503Local Path error Warning BackOff 5s (x9 over 3m20s) kubelet Back-off restarting failed container local-path-provisioner in pod local-path-provisioner-6795b5f9d8-dmrz2_kube-system(48ac961f-da91-4b20-9aa3-e6f5323725d4) |
Beta Was this translation helpful? Give feedback.
-
|
K3s is statically linked and does not directly call into nft itself. It does however use the host iptables commands, and if you're breaking the iptables-nft backend by replacing the nft libraries with a completely different major version that causes subtle changes in behavior, that would be a problem. You can tell k3s to use our statically linked iptables binaries, instead of the host iptables wrappers, by starting k3s with I'm going to convert this to a discussion, because I don't see that there is anything wrong with k3s itself. You're replacing distro packages with incompatible versions and stuff is breaking. |
Beta Was this translation helpful? Give feedback.
-
|
rebuild the machine with Debian 12 , which has nft 1.0.6 by default , kube-system pods are failing on a fresh Debian 12 get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-576bfc4dc7-9p24q 1/1 Running 0 2m3s
local-path-provisioner-6795b5f9d8-llb2d 0/1 Error 2 (55s ago) 117s
metrics-server-557ff575fb-cqhkc 0/1 Running 2 (46s ago) 110sos cat /etc/os-*
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookwormis k3s supported on Debian 12 ? |
Beta Was this translation helpful? Give feedback.
-
|
I did not find the exact root cause of the issue , but further testing showed the same error on Debian 12 as well . Now I have k3s server with nft 1.1.0 with no issue I someone knows the exact root cause please let me know |
Beta Was this translation helpful? Give feedback.
-
Environmental Info:
K3s Version:
Node(s) CPU architecture, OS, and Version:
Cluster Configuration:
Describe the bug:
k3s is installed on Debian 11, but sometimes (randomly) when nftables 1.1.0 is also installed on that machine , then k3s server kube-system pods are failed . This tested in two ways
If we install nft 1.1.0 and then k3s server immediately all kube-system pods fail
if we install k3s first , there is no error, but immediately after installing nft 1.1.0 all kube-system pods fail
Steps To Reproduce:
Expected behavior:
No kube-system pod failure
Actual behavior:
all of them seem to fail due to liveness/readiness probe failure
Additional context / logs:
checking the metrics-server
root(0)100m1(0) k3s# root(0)100m1(0) k3s# kubectl describe pod -n kube-system metrics-server-557ff575fb-cqh58 Name: metrics-server-557ff575fb-cqh58 Namespace: kube-system Priority: 2000001000 Priority Class Name: system-node-critical Service Account: metrics-server Node: 100m1/172.31.100.254 Start Time: Sun, 01 Sep 2024 12:43:24 +0330 Labels: k8s-app=metrics-server pod-template-hash=557ff575fb Annotations: <none> Status: Running IP: 10.12.0.33 IPs: IP: 10.12.0.33 IP: 2001:cafe:12::21 Controlled By: ReplicaSet/metrics-server-557ff575fb Containers: metrics-server: Container ID: containerd://cfffeb75fb79cf7327867cb885f834f10fe0bff3526f60f7ebb2a524c1273b32 Image: rancher/mirrored-metrics-server:v0.7.0 Image ID: docker.io/rancher/mirrored-metrics-server@sha256:20b8b36f8cac9e25aa2a0ff35147b13643bfec603e7e7480886632330a3bbc59 Port: 10250/TCP Host Port: 0/TCP Args: --cert-dir=/tmp --secure-port=10250 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=15s --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 12 Sep 2024 23:46:58 +0330 Finished: Thu, 12 Sep 2024 23:47:29 +0330 Ready: False Restart Count: 68 Requests: cpu: 100m memory: 70Mi Liveness: http-get https://:https/livez delay=60s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get https://:https/readyz delay=0s timeout=1s period=2s #success=1 #failure=3 Environment: <none> Mounts: /tmp from tmp-dir (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-mnk58 (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: tmp-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-mnk58: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: Burstable Node-Selectors: <none> Tolerations: CriticalAddonsOnly op=Exists node-role.kubernetes.io/control-plane:NoSchedule op=Exists node-role.kubernetes.io/master:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 25m (x11 over 25m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 25m (x5 over 25m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 25m kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": dial tcp 10.12.0.33:10250: connect: connection refused Warning BackOff 24m (x2 over 25m) kubelet Back-off restarting failed container metrics-server in pod metrics-server-557ff575fb-cqh58_kube-system(b8eb3707-b5a2-461a-8f36-8a256c57b18e) Normal Pulled 24m (x2 over 25m) kubelet Container image "rancher/mirrored-metrics-server:v0.7.0" already present on machine Normal Created 24m (x2 over 25m) kubelet Created container metrics-server Normal Started 24m (x2 over 25m) kubelet Started container metrics-server Warning Unhealthy 20m (x2 over 20m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 19m (x14 over 20m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 19m kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": dial tcp 10.12.0.33:10250: connect: connection refused Warning BackOff 19m (x2 over 19m) kubelet Back-off restarting failed container metrics-server in pod metrics-server-557ff575fb-cqh58_kube-system(b8eb3707-b5a2-461a-8f36-8a256c57b18e) Normal Created 19m (x2 over 20m) kubelet Created container metrics-server Normal Pulled 19m (x2 over 20m) kubelet Container image "rancher/mirrored-metrics-server:v0.7.0" already present on machine Normal Started 19m (x2 over 20m) kubelet Started container metrics-server Normal Created 19m kubelet Created container metrics-server Warning Unhealthy 19m kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": dial tcp 10.12.0.33:10250: connect: connection refused Normal Started 19m kubelet Started container metrics-server Warning Unhealthy 18m (x2 over 19m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 18m (x2 over 19m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": context deadline exceeded Warning Unhealthy 18m (x17 over 19m) kubelet Readiness probe failed: Get "https://10.12.0.33:10250/readyz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Normal Pulled 14m (x5 over 19m) kubelet Container image "rancher/mirrored-metrics-server:v0.7.0" already present on machine Warning BackOff 4m7s (x59 over 18m) kubelet Back-off restarting failed container metrics-server in pod metrics-server-557ff575fb-cqh58_kube-system(b8eb3707-b5a2-461a-8f36-8a256c57b18e)here is the error for this pod
https://10.12.0.33:10250/readyz": dial tcp 10.12.0.33:10250: connect: connection refusedalso
ncthe ip and port failbut can
pingitand the logs of this pod
within the host , the endpoint is available
root(0)100m1(0) k3s# curl -k https://10.13.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 }root(0)100m1(0) k3s#and
ncnc -t -zv 10.13.0.1 443 Connection to 10.13.0.1 443 port [tcp/https] succeeded!Blaming the nftables for the cause of issue, I uninstalled nft 1.1.0 and its related files but did did help
Also
has not changed and
iptables-savehas no errorlastly I should mention that ntables 1.1.0 just installed without applying any rule so there is no rule to have conflict with legacy iptables
iptables-save.txt
Any idea why kube-system pods fail ?
Beta Was this translation helpful? Give feedback.
All reactions