kube-proxy with ipvs and lc does not work as expected (perhaps a conflict with flannel rules)

[stenreijers]

Environmental Info:
K3s Version: v1.29.6+k3s2 (b4b156d)

Node(s) CPU architecture, OS, and Version:
Linux us-central-43abs 6.1.0-22-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64 GNU/Linux

Cluster Configuration:

3 servers in HA mode

Describe the bug:

I want to setup a HA cluster over tailscale using IPVS instead of IPTABLES for kube-proxy. When I setup the initial cluster node, the startup works fine and the cluster starts. After setting up the second node, it connects to the cluster but after flannel sets the IPTABLES rules (eventhough kube-proxy is using IPVS) the connection stops and k3s output shows that it is not able to reach the master node anymore. A simple ping to the master node does not even work anymore, probably due to conflicting iptables rules.

Steps To Reproduce:

Run the following on all nodes to enable ipvs and ipvs least conn load balancing:

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_lc
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

Setup the master node

curl -sfL https://get.k3s.io | K3S_TOKEN=mypassword sh -s - server \
    --cluster-init \
    --node-ip=100.101.63.89 \
    --kube-proxy-arg=ipvs-scheduler=lc \
    --kube-proxy-arg=proxy-mode=ipvs

Setup the second node

curl -sfL https://get.k3s.io | K3S_TOKEN=mypassword sh -s - server \
    --server https://100.101.63.89:6443 \
    --node-ip=100.81.174.114 \
    --kube-proxy-arg=ipvs-scheduler=lc \
    --kube-proxy-arg=proxy-mode=ipvs

See that the node connects in the logs and directly after flannel sets its iptable rules, see that all further communciation just blocks. Do a simple ping test and see that it also not works anymore.

Expected behavior:

I expect that the nodes connect and stay available with correct iptable rules.

Actual behavior:

Some iptable rules from flannel prevent or conflict with the ipvs rules.

Additional context / logs:

[brandond]

Just out of curiosity, does this still happen if you disable traefik or servicelb? I think I remember hearing something about the ipvs entries that kube-proxy adds for the traefik LoadBalancer service causing problems in some environments.

[stenreijers]

I have been playing for the last 3 hours with this issue and I got it working finally using:

curl -sfL https://get.k3s.io | K3S_TOKEN=mysecret sh -s - server \
    --cluster-init \
    --node-ip=100.101.63.89 \
    --disable traefik \
    --disable metrics-server \
    --disable servicelb \
    --cluster-cidr 172.16.0.0/16 \
    --service-cidr 172.17.0.0/16 \
    --cluster-dns=172.17.0.10 \
    --kube-proxy-arg ipvs-scheduler=lc \
    --kube-proxy-arg proxy-mode=ipvs \
    --flannel-iface tailscale0

curl -sfL https://get.k3s.io | K3S_TOKEN=mysecret sh -s - server \
   --server https://100.101.63.89:6443 \
   --node-ip=100.81.174.114 \
   --disable traefik \
   --disable metrics-server \
   --disable servicelb \
   --cluster-cidr 172.16.0.0/16 \
   --service-cidr 172.17.0.0/16 \
   --cluster-dns=172.17.0.10 \
   --kube-proxy-arg ipvs-scheduler=lc \
   --kube-proxy-arg proxy-mode=ipvs \
   --flannel-iface tailscale0

Disabling traefik combined with setting non-default cidr did the trick.

[brandond]

I suspect that you'll run into the same problem any time you use ServiceLB, since kube-proxy will add IPVS VIPs for the remote nodes IPs to each node (since those are the loadbalancer IPs advertised by ServiceLB), and that probably breaks communication with those remote nodes.