Kafka on Kubernetes

Transparent Kafka setup that you can grow with. Good for both experiments and production.

How to use:

• Run a Kubernetes cluster, minikube or real.
• Quickstart: use the kubectl applys below.
• Kafka for real: fork and have a look at addons.
• Join the discussion in issues and PRs.

Why? See for yourself, but we think this project gives you better adaptability than helm charts. No single readable readme or template can properly introduce both Kafka and Kubernets. Back when we read Newman we were beginners with both. Now we've read Kleppmann, Confluent and SRE and enjoy this "Streaming Platform" lock-in 😄.

What you get

Keep an eye on kubectl --namespace kafka get pods -w.

The goal is to provide Bootstrap servers: kafka-0.broker.kafka.svc.cluster.local:9092,kafka-1.broker.kafka.svc.cluster.local:9092,kafka-2.broker.kafka.svc.cluster.local:9092 `

Zookeeper at zookeeper.kafka.svc.cluster.local:2181.

Start Zookeeper

The Kafka book recommends that Kafka has its own Zookeeper cluster with at least 5 instances.

kubectl apply -f ./zookeeper/

To support automatic migration in the face of availability zone unavailability we mix persistent and ephemeral storage.

Start Kafka

kubectl apply -f ./

You might want to verify in logs that Kafka found its own DNS name(s) correctly. Look for records like:

kubectl -n kafka logs kafka-0 | grep "Registered broker"
# INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(kafka-0.broker.kafka.svc.cluster.local,9092,PLAINTEXT)

That's it. Just add business value 😉. For clients we tend to use librdkafka-based drivers like node-rdkafka. To use Kafka Connect and Kafka Streams you may want to take a look at our sample Dockerfiles. Don't forget the addons.

Tests

kubectl apply -f test/
# Anything that isn't READY here is a failed test
kubectl get pods -l test-target=kafka,test-type=readiness -w --all-namespaces