Question about external access to Redpanda cluster on Kubernetes

[thrashwerk]

Hello,

I've got a Kubernetes cluster on bare-metal servers with a HAProxy LB in front of it for ingresses and NodePorts.

I've deployed a 3 node Redpanda cluster with Helm and configured the external listener.

It created a NodePort service redpanda-external that forwards 31092 on the nodes to 9094 on the pods.

As far as I know, Kafka clients need access to all of the Kafka/Redpanda nodes but the K8s service load balances between the pods and you can't access specific pods.

So how should external clients properly connect to the Redpanda cluster on K8s?

[JakeSCahill]

Hi @thrashwerk, the recommendation is to use the NodePort Service for external access. You can configure each Redpanda broker to advertise the address of its worker node and the exposed node port.

We published a new doc for best practices in Kubernetes. The section about external networking explains why we recommend the NodePort Service.

We also have an open pull request with a doc that goes into the details of networking in Kubernetes.

Note: The how-to guides in that PR for NodePort and LoadBalancers are works in progress.

[miguel-cardoso-mindera]

Hey @JakeSCahill I'm facing the same issue as the original post. Your reply points to the documentation to setup a nodeport for the bootstrap endpoint, but what we need is also a nodeport for each individual broker so the clients can maintain connectivity to a single broker.

Is this natively supported by redpanda helm deployment (in my case)?

[JakeSCahill]

Hey @miguel-cardoso-mindera, the recommendation is to run each broker on a dedicated worker node. That way, you can use a NodePort Service to expose each broker and the address of the worker node to access each one individually.

You may also use LoadBalancer Services.

See https://docs.redpanda.com/current/manage/kubernetes/networking/external/

[miguel-cardoso-mindera]

Yes I understand that, it's how it is currently configured. What I'm saying is, in addition to a bootstrap nodeport service that loadbalances between the different pods, we would also require a nodeport per broker.

Here is an example from our current Apache Kafka cluster:

Each of the cluster-kafka-outside-# is a nodeport service that points to only one broker. This allows our java clients (streams) to maintain connectivity with the same broker

[JakeSCahill]

The default isn’t a bootstrap load balancer. You get a NodePort Service that exposes each broker on the same port. You address each broker externally using the address of the worker node and the node port.

[miguel-cardoso-mindera]

I didn't mean a bootstrap, I meant the endpoint that acts as a loadbalancer for the brokers is usually called bootstrap.

Yes, initially you start the connection using the the single nodeport which connects you to a random broker, but after that, the broker advertises its host and port so the client can talk directly to it.

And in the example I provided above, for example, broker-1 would advertise this:
host: mydomain.kafka.com
port: 3000

and cluster-kafka-outside-1 is a nodeport service for port 3000 that points to only broker-1.

Hopefully now it's more clear.