Unable to access service/ingress to a offloaded pod: 504 Gateway Time-out #2909
Labels
fix
Fixes a bug in the codebase.
Comments
|
it seems the lable |
|
Hi @remmen-io, can you give us more insight about how you configured your ingress? |
|
Hi @cheina97 Herer is the full deployment |
Merged
|
Hi @remmen-io, I think we fixed your issue in this PR #2924. We found a bug in IPs remapping algorithm. Thanks for helping us to spot it. |
|
Here are the IPs The config is provider Currently I have the following netpols inplace consumer provider I did not see a log of errors, one thing might be that the provider gw I see This looks good on the consumer gw |
|
Can you share the configuration resource of the 2 clusters |
|
|
Hi @remmen-io, i just checked your use case in environment which is similar to yours. I'm using 3 clusters (1 consumer and 2 providers) made with Kind. The podcidr is "10.127.64.0/18" in every cluster with cilium. ❯ k get no
NAME STATUS ROLES AGE VERSION
cheina-cluster1-control-plane Ready control-plane 13m v1.32.1
cheina-cluster1-worker Ready <none> 13m v1.32.1
cheina-cluster2 Ready agent 106s v1.32.1
cheina-cluster3 Ready agent 105s v1.32.1
❯ k get foreignclusters.core.liqo.io -o wide
NAME ROLE CLUSTERID AGE
cheina-cluster2 Provider cheina-cluster2 2m37s
cheina-cluster3 Provider cheina-cluster3 2m16s
❯ k get configurations.networking.liqo.io -A
NAMESPACE NAME DESIRED POD CIDR REMAPPED POD CIDR AGE
liqo-tenant-cheina-cluster2 cheina-cluster2 ["10.127.64.0/18"] ["10.71.0.0/18"] 2m45s
liqo-tenant-cheina-cluster3 cheina-cluster3 ["10.127.64.0/18"] ["10.69.0.0/18"] 2m24s
❯ k get po -n liqo-demo -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-5869d7778c-tc5jw 1/1 Running 0 106s 10.69.1.198 cheina-cluster3 <none> <none>
❯ k get svc -n liqo-demo nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx ClusterIP 10.111.166.93 <none> 80/TCP 16s
❯ k get ingress -n liqo-demo
NAME CLASS HOSTS ADDRESS PORTS AGE
nginx nginx kind.local 172.19.0.16 80 5m34s
❯ curl -I kind.local
HTTP/1.1 200 OK
Date: Tue, 25 Feb 2025 09:29:38 GMT
Content-Type: text/html
Content-Length: 615
Connection: keep-alive
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
ETag: "67a34638-267"
Accept-Ranges: bytesDo you think there is some huge difference in my setup? |
|
Are you using cilium as CNI? We did a lot of debugging and finally we saw the following on the provider cluster Where 10.127.66.173 is the ip of the pod on the consumer cluster and 10.127.66.106 is the pod on the provider cluster I then stumbled across this article about IP Spoofing protection: ...we can see that since the source IP wasn’t provided by Cilium’s IPAM subsystem, we know it’s a spoofed IP address and Cilium automatically blocks the traffic. So I'm wondering if this could be the root cause? |
|
Thought a bit more about it.
Fixed with disabling SourceIPVerification Could it be that the issue is due to not performing a source ip nat? When I observe the packet from the pod on the consumer cluster to the provider cluster I noticed the source IP did not change on the liqo gateway on the provider cluster, only the destination IP. Is this the correct behavior? This would explain the SourceIPVerification and unroutable IP from cilium. This seems to be a know problem with KubeOVN as well |
|
And just verified with my initial curl request. The request gets dropped as the source IP is of the ingress-nginx on the |
|
Hi @remmen-io, when you sniff the traffic on the receiving pod, the source IP should be remapped on this network 10.71.0.0/18. But it seems that the source IP of the coming packets has not been remapped, can you confirm? |
|
Yes, I can confirm. On the receiving pod I see the original IP of the pod on the consumer cluster and not a remapped one |
|
Can you check all the firewallconfiguration resources status? |
|
They seem to been applied on both clusters. Should I check a specific one? ⎈provider cluster ⎈ consumer |
|
consumer provider From the events I see consumer provider I the firewall configuration on the provider not applied? |
|
debug |
|
Can you check sniffing traffic if the src nat is working and what outgoing interface is being used for traffic coming from the consumer cluster? |
|
tcpdump on gateway pod on provider cluster Pods on consumer Pods on provider |
|
Ok, the problem is that the traffic is going out from the gateway using eth0, but it should use one of the geneve interfaces present on the gateway. Can you give me more information about your setup? Like kernel, linux distribution and kubernetes distribution? It seems that for some reason the policy routing is not working on your gateway, I had a similar issue with RHEL in the past. |
|
In this cluster we have:
On the provider cluster I've seen the following events (the last 100) on the consumer I see |
|
In the provider gateway can you give me the output of |
|
Can you also send the output of these 2 commands? kubectl get routeconfiguration -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)"' | while read -r resource; do namespace=$(echo "$resource" | cut -d'/' -f1); name=$(echo "$resource" | cut -d'/' -f2); echo "$namespace/$name: $(kubectl get routeconfiguration -n "$namespace" "$name" -o yaml | wc -l) lines"; donekubectl get firewallconfiguration -A -o json | jq -r '.items[] | "\(.metadata.namespace)/\(.metadata.name)"' | while read -r resource; do namespace=$(echo "$resource" | cut -d'/' -f1); name=$(echo "$resource" | cut -d'/' -f2); echo "$namespace/$name: $(kubectl get firewallconfiguration -n "$namespace" "$name" -o yaml | wc -l) lines"; done |
|
provider cluster gw-e1-k8s-mfmm-lab-b-7849c4d7d4-rfsmx:/# ip route show table all gw-e1-k8s-mfmm-lab-b-7849c4d7d4-rfsmx:/# ip rule gw-e1-k8s-mfmm-lab-b-7849c4d7d4-rfsmx:/# ip a |
|
provider cluster Consumer cluster |
|
You don't have a route in your gateway for If you notice you don't have routes for pods in Can you also check on the liqo-controller-manager logs if you are receiving errors from "pod_internalnet" controller? Just grep the logs using "pod_internalnet" |
|
Can you also check if a resource called internalnode with the same name of the node where the pod is scheduled exists? |
|
Do you refer to the podCIDR? Yes, it seems that we have different podCIDR per node on the cluster, but using the same range on all clusters consumer provider I'm not able to ping the ip I do fin it on the consumer On the consumer I find the following errors I do not find a routerconfiguration for gpupool-gw-node internalnode consumer provider |
|
Just for my understanding If I understood you correctly, there is no IP SRC remapping once the icmp packet arrvices on the GW pod on the provider cluster, only the IP DST is remapped to the effective pod IP. For the return path a direct host route must/should be applied pointing the consumer source IP to the provider gw pod. Correct? Is there a specific reason why there is no bi-directional NAT? If the packet would also adjust the SRC IP to the GW pods IP this should work? Then IP spoofing won't occur and the return path would be known. |
The Pod you cannot reach is
You need to do it from another pod scheduled on the consumer. |
|
Can you share the content of e1-k8s-alsu108.mm.ch-gw-node routeconfiguration on provider cluster? |
Not correct, the src IP remapping must happen in your case, because you have the same podcidr in each cluster, so the gateway on the provider cluster should perform a src NAT using the remapped podcidr contained in the "configuration" resource on the provider. The problem here is that the src NAT happens only if the gateway use a geneve interface to output the traffic, for some reasons the correct routing rules have not been applied and the traffic for pods scheduled on e1-k8s-alsu108.mm.ch uses the eth0 interface instead of our geneve tunnels. |
Let's make an example, I'm a pod in the consumer cluster and I want to reach a pod in the provider. I need to use an IP that will be routed from the GW to the provider. This means that I will always need to receive returning traffic with a SRC IP which is not part of your CIlium IPAM |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
I've deployed a pod,svc and ingress on a cluster with network fabric and offloading enabled.
svc and ingress are excluded from the resource-reflection.
The pod is successfully started and I can see the logs, I can access the service or the pod direclty with
kubectl port-forwardAccessing the ingress I get a 504
What you expected to happen:
Access the service of the ingress
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
We are using cilium as CNI with native routing. There is no networkpolicy preventing any traffic
We have noticed, that the pod gets an IP in the range of 10.71.72.225 which is not in the 10.71.0.0/18 range.
Therefore we have seen that traffic from a pod to this ip gets routed over the default gateway, which we think is wrong
On the node where the debug pod was running (with curl on the the service/pod ip)
But even if adding manually a route, I still got no response. So we might be wrong
Additional Informations:
Provider: e1-k8s-mfmm-lab-t
Consumer: e1-k8s-mfmm-lab-b
Liqo Status
Deployment
Environment:
kubectl version): v1.30.4The text was updated successfully, but these errors were encountered: