Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DO NOT MERGE: Supertubes rebranding #37

Open
wants to merge 109 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
109 commits
Select commit Hold shift + click to select a range
60db307
Update links for supertubes rebranding
fekete-robert May 19, 2021
7934762
Delete some obsolete Pipeline references
fekete-robert May 19, 2021
d33bbc4
Supertubes is called Cisco Streaming Data Manager
fekete-robert May 20, 2021
6913bce
Change contact url to Cisco email address
fekete-robert May 20, 2021
0a47882
Featured image fix
fekete-robert Jun 10, 2021
d1f8547
Update cert-manager version to 1.3.1 from 0.15.1
baluchicken Jun 4, 2021
eca51ee
Starts updating the compatibility matrix for 1.7.0
fekete-robert Aug 3, 2021
74f6cab
Deletes references to old unsupported 0.14.0 kafka operator versions
fekete-robert Aug 4, 2021
6c9cbf2
Adds new component images available on ghcr.io
fekete-robert Aug 4, 2021
136704a
Doc updates for https://github.com/banzaicloud/kafka-operator/issues/398
fekete-robert Aug 9, 2021
a174d82
Updates javaagent version number
fekete-robert Aug 9, 2021
9f899f6
Formatting corretions on the benchmark page
fekete-robert Mar 5, 2021
07f2292
Sample topic corrections
fekete-robert Mar 5, 2021
7b84473
Formatting corrections in topics.md
fekete-robert Mar 5, 2021
dac1bd0
Ports changes related to kafka operator rebranding
fekete-robert Aug 25, 2021
00e2bd0
Typo fix
fekete-robert Sep 29, 2021
a57e58b
Updates cert-manager version numbers
fekete-robert Sep 30, 2021
23fb33e
Updates suppoerted Kafka version numbers
fekete-robert Sep 30, 2021
6919615
Updates the compatibility matrix
fekete-robert Sep 30, 2021
e841020
Deletes buggy koperator releases from the compatibility matrix
fekete-robert Sep 30, 2021
24c7400
Update docs/compatibility-matrix.md
fekete-robert Sep 30, 2021
b6957c6
Review corrections
fekete-robert Sep 30, 2021
a780b72
Update cert-manager version to 1.3.1 from 0.15.1
baluchicken Jun 4, 2021
c79ec3c
Koperator needs k8s 1.19
fekete-robert Oct 7, 2021
86f576c
Remove vault related doc since it was removed from the operator
fekete-robert Oct 12, 2021
7906284
Remove unused values from ssl certs
baluchicken Oct 13, 2021
2a82dff
Update zk example config to align with the results
baluchicken Nov 17, 2021
32ad76f
Install the cert-manager CRDs first, before the helm chart
adamantal Nov 25, 2021
80305a6
Updates Kafka version
fekete-robert Nov 26, 2021
05e4318
Updates default kafka version
fekete-robert Nov 26, 2021
70e07f8
Compatibility matrix update for 0.20.0
fekete-robert Nov 26, 2021
3b8a860
Update docs/compatibility-matrix.md
fekete-robert Nov 26, 2021
0e5f179
Update docs/compatibility-matrix.md
fekete-robert Nov 26, 2021
2d71187
Adds new koperator images
fekete-robert Nov 26, 2021
080686e
Update docs/compatibility-matrix.md
fekete-robert Nov 26, 2021
c1f57cd
Update docs/compatibility-matrix.md
fekete-robert Nov 26, 2021
4401497
Update cert-manager to 1.5.3
gecube Dec 11, 2021
89c026e
Tips and tricks - new sections with minor doc snippets (#22)
fekete-robert Dec 13, 2021
40bd0ff
Fix Kafka operator strings that sneaked back from old PR (#53)
fekete-robert Dec 13, 2021
c12d07d
Add documentation about ssl cert per listener
fekete-robert Dec 15, 2021
cfcbbd8
Refactor 1.
bartam1 Dec 9, 2021
d770f4b
Refactor 2.
bartam1 Dec 9, 2021
a6da8a1
Refactor 3.
bartam1 Dec 9, 2021
e25ecb8
Fix typo
bartam1 Dec 9, 2021
f0777fd
Documentation for per listener based certificate usage
fekete-robert Dec 11, 2021
385ca71
Remove unnecessary files
bartam1 Dec 11, 2021
17b2add
Fix path errors from ported patches
fekete-robert Dec 15, 2021
6378c7e
Replace kafkacat with kcat
fekete-robert Jan 11, 2022
2b57859
Update docs/kafkacat-ssl.sample
fekete-robert Jan 11, 2022
51920c4
Fixes old version number
fekete-robert Feb 11, 2022
af11116
Updates cert-manager versions
fekete-robert Feb 11, 2022
14fe762
Updates cert-manager versions to 1.6.2
fekete-robert Feb 11, 2022
c74e16a
Clarifies cert-manager support
fekete-robert Feb 11, 2022
23d7a5b
Compatiblity matrix updates
fekete-robert Feb 11, 2022
d4781a0
Adds new jmx exporter image to the list
fekete-robert Feb 11, 2022
04c8420
Update docs/compatibility-matrix.md
fekete-robert Feb 11, 2022
3777df1
Version bumps for koperator 0.21.0
fekete-robert Mar 30, 2022
540ab36
Update docs/compatibility-matrix.md
fekete-robert Mar 30, 2022
cf2cd45
Version bumps for koperator 0.21.1
stoader Apr 14, 2022
6c4c770
Version bumps for koperator 0.21.1
stoader Apr 19, 2022
921165a
Update kafka image versions in sample commands
fekete-robert Apr 19, 2022
9814f1d
fix
kevin85421 Jun 20, 2022
1f33811
Update sasl oauth related doc to be in line with Kafka 3.1.0
stoader Jun 28, 2022
d45b3d0
Correction
stoader Jun 28, 2022
55666fc
Minor clarifications
fekete-robert Jun 28, 2022
28a565c
Add retrieve broker conf tip
bartam1 Jun 30, 2022
4465972
Fix review suggestion 1
bartam1 Jun 30, 2022
4971927
Fix suggestions
bartam1 Jul 13, 2022
ff67fa2
Update docs/monitoring.md
fekete-robert Jul 14, 2022
983aa94
docs(install): upped k8s range to 1.20+
pregnor Sep 15, 2022
2e0801c
Merge pull request #73 from banzaicloud/docs/k8s-1-22-rebranding
fekete-robert Sep 15, 2022
88bdf5a
SDM-SMM integration (#71)
fekete-robert Nov 28, 2022
5b550ba
Fixes image links
fekete-robert Nov 28, 2022
fba19da
Remove banzai logos from figures
fekete-robert Nov 28, 2022
b5f3913
Remove obsolete banzai references
fekete-robert Nov 28, 2022
9723d17
Updates support email
fekete-robert Nov 29, 2022
5c8202b
Cut unneeded border from figure
fekete-robert Nov 29, 2022
69f2bcc
Remove banzai cli references from the benchmark page
fekete-robert Nov 29, 2022
f7bb1cc
Style fixes
fekete-robert Dec 5, 2022
c607d27
Updates the compatibility matrix for koperator 0.22.0 (#77)
fekete-robert Dec 6, 2022
7fd60d1
Update kubectl command to use version 0.22.0 koperator CRDs for koper…
fekete-robert Jan 9, 2023
e908b8f
Add documentation about CruiseControlOperation feature (#76)
bartam1 Jan 21, 2023
521963e
Update custom certificate configuration for SSL (#69)
bartam1 Jan 21, 2023
1e9ded6
Typo corrections
fekete-robert Jan 23, 2023
1dfacae
Add warning for installing koeprator in EKS with 1.23+ (#84)
panyuenlau Jan 26, 2023
12cb9b6
Port warning to SDM/koperator docs
fekete-robert Jan 26, 2023
2982f21
Fix writing convention of ZooKeeper, and add some Apache references
fekete-robert Jan 26, 2023
ee37884
Fix koperator architecture image link (#85)
panyuenlau Jan 27, 2023
a028f85
Update Benchmarking Kafka docs (#87)
panyuenlau Feb 9, 2023
dc77ea9
Fix misspelled parameter `advertized.listeners`
fekete-robert Feb 14, 2023
933ec6e
Add configurations/kafkacluster (#82)
fekete-robert Feb 16, 2023
6196b89
Update zookeeper configuration to use 3 replicas
panyuenlau Feb 21, 2023
e02daa5
Typo correction
fekete-robert Mar 17, 2023
b683dad
Remove reference to an unsupported feature
fekete-robert Mar 17, 2023
84d9d9f
Added the features of Koperator to the main page (#101)
arati-1 Apr 12, 2023
b077fda
Update commands and wordings (#104)
panyuenlau Apr 13, 2023
0abd9a2
Update nits (#108)
panyuenlau Apr 19, 2023
307fdba
Syched from master 20230418 (#106)
fekete-robert Apr 28, 2023
d7a0f75
Fix duplicate container port (#109)
panyuenlau May 1, 2023
94e2a41
fixing xref error
arati-1 May 1, 2023
f038733
Removes username from sample output (#115)
fekete-robert May 3, 2023
bff572a
Update generated CRD references
fekete-robert May 5, 2023
964a19c
updated Koperator documentation (#107)
arati-1 May 13, 2023
142173d
Cherry-pick SSL from master (#118)
panyuenlau May 16, 2023
0b86b7e
Link update
fekete-robert May 17, 2023
4e7e361
fix(install): fixed GH version links (#113)
pregnor May 2, 2023
40f084e
Replace kafka-topics --zookeeper flags with --bootstrap-server flags …
pregnor May 13, 2023
5cd9917
Update _index.md (#122)
arati-1 Jun 6, 2023
b441461
Updates some broken links
fekete-robert Jul 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
.idea
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# koperator-docs
Documentation for Koperator - the operator for managing Apache Kafka on Kubernetes

This repository contains the documentation for the [Koperator](https://github.com/banzaicloud/koperator).

- The _master_ branch contains the public documentation of Koperator, published at https://banzaicloud.com/docs/supertubes/kafka-operator/
- The _supertubes-integration_ branch contains the Koperator docs for Calisti. The different branches are needed because of the different release schedule of Koperator and Calisti, and the path/url differences between the public Koperator docs and Calisti. The Calisti docs is published at [https://docs.calisti.app/sdm/koperator/](https://docs.calisti.app/sdm/koperator/).
126 changes: 113 additions & 13 deletions docs/_index.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,20 @@
---
title: Kafka operator
img: /docs/supertubes/kafka-operator/img/kafka-operator-arch.png
title: Koperator
img: /docs/koperator-docs/img/kafka-operator-arch.png
weight: 700
aliases:
- /sdm/koperator/features/
cascade:
module: kafka-operator
githubEditUrl: "https://github.com/banzaicloud/kafka-operator-docs/edit/master/docs/"
githubEditUrl: "https://github.com/banzaicloud/koperator-docs/edit/master/docs/"
operatorName: "Koperator"
---

The Banzai Cloud Kafka operator is a Kubernetes operator to automate provisioning, management, autoscaling and operations of [Apache Kafka](https://kafka.apache.org) clusters deployed to K8s.
The {{< kafka-operator >}} (formerly called Banzai Cloud Kafka Operator) is a Kubernetes operator to automate provisioning, management, autoscaling and operations of [Apache Kafka](https://kafka.apache.org) clusters deployed to K8s.

## Overview

[Apache Kafka](https://kafka.apache.org) is an open-source distributed streaming platform, and some of the main features of the **Kafka-operator** are:
[Apache Kafka](https://kafka.apache.org) is an open-source distributed streaming platform, and some of the main features of the **{{< kafka-operator >}}** are:

- the provisioning of secure and production-ready Kafka clusters
- **fine grained** broker configuration support
Expand All @@ -23,17 +26,114 @@ The Banzai Cloud Kafka operator is a Kubernetes operator to automate provisionin
- graceful rolling upgrade
- advanced topic and user management via CRD

![Kafka-operator architecture](./img/kafka-operator-arch.png)
![{{< kafka-operator >}} architecture](/sdm/koperator/img/kafka-operator-arch.png)

>We took a different approach to what's out there - we believe for a good reason - please read on to understand more about our [design motivations](features/) and some of the [scenarios](scenarios/) which were driving us to create the Banzai Cloud Kafka operator.

{{% include-headless "doc/kafka-operator-supertubes-intro.md" %}}
{{% include-headless "kafka-operator-supertubes-intro.md" "sdm" %}}

## Motivation

At [Banzai Cloud](https://banzaicloud.com) we are building a Kubernetes distribution, [PKE](/products/pke/), and a hybrid-cloud container management platform, [Pipeline](/products/pipeline/), that operate Kafka clusters (among other types) for our customers. Apache Kafka predates Kubernetes and was designed mostly for `static` on-premise environments. State management, node identity, failover, etc all come part and parcel with Kafka, so making it work properly on Kubernetes and on an underlying dynamic environment can be a challenge.
Apache Kafka predates Kubernetes and was designed mostly for `static` on-premises environments. State management, node identity, failover, etc all come part and parcel with Kafka, so making it work properly on Kubernetes and on an underlying dynamic environment can be a challenge.

There are already several approaches to operating Apache Kafka on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our customers' needs. At the same time, there is substantial interest within the Kafka community for a solution which enables Kafka on Kubernetes, both in the open source and closed source space.
>We took a different approach to what's out there - we believe for a good reason - please read on to understand more about our [design motivations](#features) and some of the [scenarios](scenarios/) which were driving us to create the {{< kafka-operator >}}.

Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator. We are long-term contributors and active community members of both Apache Kafka and Kubernetes, and we hope to recreate a similar community around this operator.

## Koperator features {#features}

### Design motivations
Kafka is a stateful application. The first piece of the puzzle is the Broker, which is a simple server capable of creating/forming a cluster with other Brokers. Every Broker has its own **unique** configuration which differs slightly from all others - the most relevant of which is the ***unique broker ID***.

All Kafka on Kubernetes operators use [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) to create a Kafka Cluster. Just to quickly recap from the K8s docs:

>StatefulSet manages the deployment and scaling of a set of Pods, and provide guarantees about their ordering and uniqueness. Like a Deployment, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains sticky identities for each of its Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that is maintained across any rescheduling.

How does this look from the perspective of Apache Kafka?

With StatefulSet we get:

- unique Broker IDs generated during Pod startup
- networking between brokers with headless services
- unique Persistent Volumes for Brokers

Using StatefulSet we **lose:**

- the ability to modify the configuration of unique Brokers
- to remove a specific Broker from a cluster (StatefulSet always removes the most recently created Broker)
- to use multiple, different Persistent Volumes for each Broker

{{< kafka-operator >}} uses `simple` Pods, ConfigMaps, and PersistentVolumeClaims, instead of StatefulSet. Using these resources allows us to build an Operator which is better suited to manage Apache Kafka.

With the {{< kafka-operator >}} you can:

- modify the configuration of unique Brokers
- remove specific Brokers from clusters
- use multiple Persistent Volumes for each Broker

## Features

### Fine Grained Broker Configuration Support

We needed to be able to react to events in a fine-grained way for each Broker - and not in the limited way StatefulSet does (which, for example, removes the most recently created Brokers). Some of the available solutions try to overcome these deficits by placing scripts inside the container to generate configurations at runtime, whereas the {{< kafka-operator >}}'s configurations are deterministically placed in specific Configmaps.

### Graceful Kafka Cluster Scaling with the help of our CruiseControlOperation custom resource

We know how to operate Apache Kafka at scale (we are contributors and have been operating Kafka on Kubernetes for years now). We believe, however, that LinkedIn has even more experience than we do. To scale Kafka clusters both up and down gracefully, we integrated LinkedIn's [Cruise-Control](https://github.com/linkedin/cruise-control) to do the hard work for us. We already have good defaults (i.e. plugins) that react to events, but we also allow our users to write their own.

### External Access via LoadBalancer

There are already several approaches to operating Kafka on Kubernetes, however, we did not find them appropriate for use in a highly dynamic environment, nor capable of meeting our customers' needs. At the same time, there is substantial interest within the Kafka community for a solution which enables Kafka on Kubernetes, both in the open source and closed source space.
>We took a different approach to what's out there - we believe for a good reason - please read on to understand more about our [design motivations](features/) and some of the [scenarios](scenarios/) which were driving us to create the Banzai Cloud Kafka operator.
The {{< kafka-operator >}} externalizes access to Apache Kafka using a dynamically (re)configured Envoy proxy. Using Envoy allows us to use **a single** LoadBalancer, so there's no need for a LoadBalancer for each Broker.

Finally, our motivation is to build an open source solution and a community which drives the innovation and features of this operator. We are long term contributors and active community members of both Apache Kafka and Kubernetes, and we hope to recreate a similar community around this operator.
![Kafka External Access](/sdm/koperator/img/kafka-external.png)

### Communication via SSL

The operator fully automates Kafka's SSL support.
The operator can provision the required secrets and certificates for you, or you can provide your own.

![SSL support for Kafka](/sdm/koperator/img/kafka-ssl.png)

### Monitoring via Prometheus

The {{< kafka-operator >}} exposes Cruise-Control and Kafka JMX metrics to Prometheus.

### Reacting on Alerts

{{< kafka-operator >}} acts as a **Prometheus Alert Manager**. It receives alerts defined in Prometheus, and creates actions based on Prometheus alert annotations.

Currently, there are three default actions (which can be extended):

- upscale cluster (add a new Broker)
- downscale cluster (remove a Broker)
- add additional disk to a Broker

### Graceful Rolling Upgrade

Operator supports graceful rolling upgrade, It means the operator will check if the cluster is healthy.
It basically checks if the cluster has offline partitions, and all the replicas are in sync.
It proceeds only when the failure threshold is smaller than the configured one.

The operator also allows to create special alerts on Prometheus, which affects the rolling upgrade state, by
increasing the error rate.

### Dynamic Configuration Support

Kafka operates with three type of configurations:

- Read-only
- ClusterWide
- PerBroker

Read-only config requires broker restart to update all the others may be updated dynamically.
Operator CRD distinguishes these fields, and proceed with the right action. It can be a rolling upgrade, or
a dynamic reconfiguration.

### Seamless Istio mesh support

- Operator allows to use ClusterIP services instead of Headless, which still works better in case of Service meshes.
- To avoid too early Kafka initialization, which might lead to unready sidecar container. The operator uses a small script to mitigate this behaviour. Any Kafka image can be used with the only requirement of an available **curl** command.
- To access a Kafka cluster which runs inside the mesh. Operator supports creating Istio ingress gateways.


---
Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries.
Loading