This project simulates the exchanging of messages between producers and consumers. Using a multi-broker Kafka setup, deployed using Strimzi.io. Which provides a way to deploy Kafka components on Kubernetes through CRDs (Custom Resource Definitions).
- Dockerfile
- Generic docker file:
- Builds golang code
- Copies the binaries into a slim image and will execute them based on the command provided by k8s.
- Generic docker file:
- Kafka config map
- Contains configurations used by all the Deployments.
- Kafka
- Deployed on kubernetes using Strimzi.io (with Zookeeper).
- Creates 2 topics in the deployment file.
- Message topic
- DLQ topic
- Producer
- Deployed on Kubernetes with a custom number of replicas.
- Contains Health Checks that verify the Kafka connection.
- Produces message bursts, using a defined Protobuf message structure.
- There is a 5% chance of producing an invalid message, for each message.
- Consumer
- Deployed on Kubernetes with a custom number of replicas.
- Contains Health Checks that verify the Kafka connection.
- Creates a producer, that will be used to send messages to the DLQ topic.
- Reads in messages from a topic.
- If message is invalid, produce it to DLQ.
# Create a cluster
kind create cluster
# Note: if it's the first time running, you must apply the config map first, in order for the setup.sh to not fail
pushd deployments
kubectl apply -f kafka-config.yaml
popd
# Setup kubernetes cluster
chmod +x setup.sh
./setup.sh <producers_replicas> <consumers_replicas>
# To delete cluster
kind delete clusterTo check the Health checks for consumers and producers, you can execute:
chmod +x healthcheck.sh
./healthcheck.sh consumer # for consumers pods
./healthcheck.sh producer # for producers podsTo check the metrics in Prometheus:
kubectl port-forward service/prometheus-service 9090:9090 -n monitoring # prometheus
kubectl port-forward svc/grafana-service 3000:3000 -n monitoring # grafanaThis link will show you all our custom rules
Go over to grafana and look for the kafka metrics dashboard.