Having pods from 2 clusters communicate

[cscetbon]

I'd like to do something similar to kubernetes-sigs/kind#926 (comment) where pods from 2 clusters A and B can communicate. Is there a way to accomplish the same with 2 k3d clusters ?

[cscetbon]

@iwilltry42 It doesn't seem to work #101 (comment) ? If I'm wrong can you show me an example on how to connect a pod of demoA cluster to another pod in demoB cluster ?

$ k get service -A --context k3d-demoB -o wide
NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE     SELECTOR
default       kubernetes           ClusterIP      10.43.0.1       <none>        443/TCP                      4m38s   <none>
kube-system   kube-dns             ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       4m36s   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      10.43.107.151   <none>        443/TCP                      4m35s   k8s-app=metrics-server
kube-system   traefik-prometheus   ClusterIP      10.43.153.102   <none>        9100/TCP                     4m8s    app=traefik,release=traefik
kube-system   traefik              LoadBalancer   10.43.17.107    172.28.0.2    80:30101/TCP,443:32180/TCP   4m8s    app=traefik,release=traefik

$ k get service -A --context k3d-demoA -o wide
NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE     SELECTOR
default       kubernetes           ClusterIP      10.43.0.1       <none>        443/TCP                      5m18s   <none>
kube-system   kube-dns             ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP       5m16s   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      10.43.67.118    <none>        443/TCP                      5m16s   k8s-app=metrics-server
kube-system   traefik-prometheus   ClusterIP      10.43.13.114    <none>        9100/TCP                     4m44s   app=traefik,release=traefik
kube-system   traefik              LoadBalancer   10.43.159.161   172.27.0.2    80:30833/TCP,443:31680/TCP   4m44s   app=traefik,release=traefik

[iwilltry42]

Hi @cscetbon , this is an interesting thing to do.
There are many things to consider though.

• The kind issue you referenced already talks about choosing the correct PodCIDR (and ServiceCIDR), so that both clusters don't use the same subnets (how should the system know, where it should route?). In your terminal output of the both clusters, we see, that both clusters use at least the same ServiceCIDR, so that would have to be changed
• The k3d issue you referenced is about communication to localhost (i.e. the host running docker in which the k3d/k3s containers are running). This allows communication between pods via either exposed ports or services (using the k3d port-mapping to localhost when creating the cluster) or Ingress (via port-mapping on the respective loadbalancer when creating the cluster).
• If you'd want to achieve a communication between pods of two clusters without port-mapping, you'd have to at least follow the guides from the kind issue (CIDRs) and also either create both clusters in the same docker network (--network flag) or, after creating the cluster, manually add all nodes (containers) of clusterB to the network of clusterA (docker network connect)

I hope this helps already 👍
What's your use case for this?

[cscetbon]

@iwilltry42 the goal is to be able to test a multi-cluster architecture.

I was showing on purpose that my CIDRs overlap and was asking for a way to change it. the question is how to achieve the same thing with k3d cause it doesn't seem you can tell it what CIDR to use. Would it be automatically achieve by just using the same network when creating both clusters ? It seems you're saying both are needed, using 2 CIDRs + same network. Then how do you do force them to use specific CIDRs ?

[iwilltry42]

CIDRs are not in the realm of k3d but k3s inside k3d.
Here you can find the k3s networking options: https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/#networking (including --cluster-cidr) which you can use in k3d via e.g. k3d cluster create --k3s-server-arg='--cluster-cidr “10.42.0.0/16”'.