Ingress is going on all the nodes

[simonfoilen]

Environmental Info:
K3s Version:

k3s version v1.29.6+k3s1 (83ae095a)
go version go1.21.11

Node(s) CPU architecture, OS, and Version:

• Linux 1.tor1.do1.k.foilen.com 6.8.0-36-generic Typo in the project description #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Linux 1.fra1.do1.k.foilen.com 6.8.0-36-generic Typo in the project description #36-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 10 10:49:14 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

• 1 server
• 1 agent

In 2 different datacenters.

Describe the bug:
I am setting up k3s on 2 nodes (just for testing. Will add more later). I could deploy a test website on a specific node (using labels), but then, when creating an Ingress, it is installing it on all the nodes (the ingress has all the external addresses instead of only the one I want).

Steps To Reproduce:

Common variables:

export VOLUME_PATH=/kube/volumes
export K3S_MAIN=do1.k.foilen.com

My K3S server config:

cat > /etc/rancher/k3s/config.yaml << _EOF
data-dir: $VOLUME_PATH/k3s

tls-san:
  - "$K3S_MAIN"

node-external-ip: $(curl https://checkip.foilen.com)
flannel-backend: wireguard-native
flannel-external-ip: true

embedded-registry: true
cluster-init: true
secrets-encryption: true

node-label:
  - "cloud=digitalocean"
  - "datacenter=tor1"
  - "svccontroller.k3s.cattle.io/lbpool=tor1_1"
  - "svccontroller.k3s.cattle.io/enablelb=true"
_EOF

My K3S agent config:

cat > /etc/rancher/k3s/config.yaml << _EOF
data-dir: $VOLUME_PATH/k3s

node-external-ip: $(curl https://checkip.foilen.com)

node-label:
  - "cloud=digitalocean"
  - "datacenter=fra1"
  - "svccontroller.k3s.cattle.io/lbpool=fra1_1"
  - "svccontroller.k3s.cattle.io/enablelb=true"
_EOF

I tried this https://docs.k3s.io/networking/networking-services#controlling-servicelb-node-selection .

My nodes have different labels:

• 1.tor1.do1.k.foilen.com
  • svccontroller.k3s.cattle.io/enablelb: true
  • svccontroller.k3s.cattle.io/lbpool: tor1_1
• 1.fra1.do1.k.foilen.com
  • svccontroller.k3s.cattle.io/enablelb: true
  • svccontroller.k3s.cattle.io/lbpool: fra1_1

Then, I wasn't sure which label and on which Kind of resource to place it, so I applied both labels:

• svccontroller.k3s.cattle.io/lbpool: tor1_1
• lbpool: tor1_1

on resources:

• Deployment, template (so the pods)
• Service
• Ingress

Expected behavior:

That checking the ingresses would show only 1 address service it like:

kubectl get ingress -o wide --all-namespaces

NAMESPACE   NAME                                  CLASS     HOSTS                 ADDRESS                          PORTS     AGE
kubetest    kubetest-foilen-org-service-ingress   traefik   kubetest.foilen.org   143.110.208.100   80, 443   62m

Actual behavior:

Checking the ingresses, the 2 nodes are still serving it:

kubectl get ingress -o wide --all-namespaces

NAMESPACE   NAME                                  CLASS     HOSTS                 ADDRESS                          PORTS     AGE
kubetest    kubetest-foilen-org-service-ingress   traefik   kubetest.foilen.org   143.110.208.100,46.101.106.102   80, 443   62m

Additional context / logs:
Here is my full test resources:

apiVersion: v1
kind: Namespace
metadata:
  name: kubetest

---

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kubetest
  name: kubetest-foilen-org
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: kubetest-foilen-org-service
      app.kubernetes.io/part-of: kubetest
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kubetest-foilen-org-service
        app.kubernetes.io/part-of: kubetest
        svccontroller.k3s.cattle.io/lbpool: tor1_1
        lbpool: tor1_1
    spec:
      nodeSelector:
        datacenter: fra1
      containers:
        - name: service
          image: nginx
          ports:
            - containerPort: 80
              name: web

---

apiVersion: v1
kind: Service
metadata:
  namespace: kubetest
  name: kubetest-foilen-org-service
  labels:
    svccontroller.k3s.cattle.io/lbpool: tor1_1
    lbpool: tor1_1
spec:
  selector:
    app.kubernetes.io/name: kubetest-foilen-org-service
    app.kubernetes.io/part-of: kubetest
  ports:
    - protocol: TCP
      port: 80
      targetPort: web

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubetest-foilen-org-service-ingress
  namespace: kubetest
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
    cert-manager.io/cluster-issuer: letsencrypt-digitalocean-dns-issuer
  labels:
    svccontroller.k3s.cattle.io/lbpool: tor1_1
    lbpool: tor1_1
spec:
  rules:
    - host: kubetest.foilen.org
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kubetest-foilen-org-service
                port:
                  number: 80
  tls:
    - secretName: kubetest-foilen-org-service-cert
      hosts:
        - kubetest.foilen.org

[brandond]

The ingress is exposed via a loadbalancer. the default loadbalancer for k3s is servicelb, which is documented here: https://docs.k3s.io/networking/networking-services#service-load-balancer

The docs cover limiting which nodes the loadbalancer uses.

[simonfoilen]

yes that is the documentation I said I was following.

I tried putting the labels everywhere without success (as I explained)

I have no clue what is missing right now. So, there is one problem:

• documentation is imcomplete/unclear
• documentation has a bug
• or the implementation has a bug

Could you give me some pointers if that is just a documentation issue?

thanks

[simonfoilen]

I downloaded K3S's code and it looks like the problem is that the k3s code is only selecting nodes for LoadBalancer
and if we check the "traefik" service that k3s installs, its LoadBalancer part doesn't have a svccontroller.k3s.cattle.io/lbpool (so it picks all nodes).

So, in other words, that cannot work.

It seems I would need to:

• disable traefik
• install traefik myself with
  • 2 services:
    • traefik-tor1_1 : with svccontroller.k3s.cattle.io/lbpool=tor1_1
    • traefik-fra1_1 : with svccontroller.k3s.cattle.io/lbpool=fra1_1
  • 2 Ingress classes
    • traefik-tor1_1
    • traefik-fra1_1
• Then, I will be able to set on my Ingress the ingressClassName for the right load-balancer pool to use.

You might consider automatically doing that. For instance, when svccontroller.k3s.cattle.io/enablelb=true, you could split traefik like that.
What do you think?

[brandond]

if we check the "traefik" service that k3s installs, its LoadBalancer part doesn't have a svccontroller.k3s.cattle.io/lbpool

It is easy enough to do that for yourself, if that's something you want. Most people don't.

If you just wanted the ingress to only use a single node, you could just only label that node with svccontroller.k3s.cattle.io/enablelb=true and do nothing else; nodes that don't have that label will not be used for servicelb.

If you want more fine grained control, you could:

1. Label the node you want the ingress to use with svccontroller.k3s.cattle.io/lbpool=traefik and svccontroller.k3s.cattle.io/enablelb=true
2. Add a HelmChartConfig to add this same label to the traefik service, as described at https://docs.k3s.io/helm#customizing-packaged-components-with-helmchartconfig

   apiVersion: helm.cattle.io/v1
   kind: HelmChartConfig
   metadata:
     name: traefik
     namespace: kube-system
   spec:
     valuesContent: |-
       service:
         labels:
           svccontroller.k3s.cattle.io/lbpool: traefik

If you want to be able to use different ingress classes for different nodes then yes, that will take much more customization and likely multiple installations of the traefik ingress controller, each bound to its own service and lbpool.

[simonfoilen]

brandond commented 6 hours ago

it seems noone will follow up since it is closed and marked done

Please don't open a new issue, keep discussion on that one. No one is responding because it's the weekend and we (generally speaking) have other things to do with our lives outside working hours.

Wow, this is so hard to communicate with you guys (that other issue is locked).
I wanted to say: "sure, but since you closed (in a hurry) the issue in less than 1 hour and left it closed, that looked as if the discussion was closed. I suggest you don't close an issue until it is fixed or the discussion is completed"

That being said, going back to the initial issue, that issue is also locked and converted to a discussion when you actually confirmed that there was a bug in the documentation. That must be reopened so that this can be fixed.

• as a newcomer, by looking at that doc, it seemed to indicate traefik would use different "lbpool", I wasted hours trying to make it work per the doc, opened a ticket, was told it was possible per that doc, downloaded the code and finaly it was clear it wasn't supported.
• you, as an expert, also assumed it would work and pointed me to that documentation

so, newcomers and experts alike cannot quickly tell per that doc that the loadbalancer with "lbpool" documentation doesn't apply to traefik. The doc must be fixed.

Please reopen the issue.
thanks

[simonfoilen]

Most people don't

How can you be sure? I saw a lot of tutorials where they just disable your traefik config to use their own config since that didn't fit their need.

Personally, for now, I can live with all ingress on all nodes. I don't mind for now. I was just trying to get it working as the doc suggested since I found it useful to segregate it. If the doc would have said right away: "that doesn't apply to traefik, traefik is configured on all nodes and with the configuration of all ingresses of the full cluster on all the nodes", I would have just said "ok, fine. Moving to my next thing to test" and would have saved hours

[brandond]

How can you be sure?

I mean, this is the internet, I can't be sure about anything. Most of the users and customers I've communicated with use our packaged chart. If someone doesn't use it, hopefully they're not coming here asking questions about things we weren't involved in.

I saw a lot of tutorials where they just disable your traefik config to use their own config since that didn't fit their need.

There certainly are a lot of tutorials out there. Some of them are even created by people who know what they're doing.

I'm not really looking to argue... we distribute a very lightly modified version of the Traefik project's helm chart. Anything you can do with it, you should be able to do with values passed via HelmChartConfig. If you want to install multiple copies of traefik, you'll need to deploy additional charts. If you want to get really wild with it, go ahead and disable it and deploy something else. No skin off my nose.

[simonfoilen]

The important point here is: "there is an issue with the doc".
Are you saying you don't want anyone to fix it? That is weird

[simonfoilen]

I guess your problem with that issue is that it is on the codes project instead of the docs project. Thus I created k3s-io/docs#292 for the docs project.
I hope that is the correct way since you don`t lead much.