gafka

                   __   _            
                  / _| | |           
   __ _    __ _  | |_  | | __   __ _ 
  / _` |  / _` | |  _| | |/ /  / _` |
 | (_| | | (_| | | |   |   <  | (_| |
  \__, |  \__,_| |_|   |_|\_\  \__,_|
   __/ |                             
  |___/                              

A full ecosystem that is built around kafka powered by golang.

Hope it can help you.

Components

• ehaproxy

  Elastic haproxy that sits in front of kateway.

• kateway

  A fully-managed real-time secure and reliable RESTful Cloud Pub/Sub streaming message/job service.

• actord

  kateway job scheduler and webhook dispatcher.

• gk

  Unified multi-datacenter multi-cluster kafka swiss-knife management console.

• zk

  A handy zookeeper CLI that supports recursive operation without any dependency.

• kguard

  Kafka clusters body guard that emits health info to InfluxDB and exports key warnings to zabbix for alerting.

The Whole Picture

            +-----------+
            | VirtualIP |
            +-----------+                                
                  |
          +--------------+                                 Alert   SOS   Dashboard
          |              |                                    |     |       |              gk
 +-------------------------------------------------------------------------------------------+
 |        |              |                                    |     |       |                |
 |  +----------+    +----------+                              |     |       |                |
 |  | ehaproxy |    | ehaproxy |                              |     V       |                |
 |  +----------+    +----------+                              |     |       |                |
 |      |                |  | discovery                       |     |       |                |
 |      +----------------+  |                                 +-------------+                |
 |            | LB          |                                        |                       |
 |            |             |   +--------------------+           +--------+                  |
 |            |             +---|                    | election  |        | watch            |
 |            |keepalive        | zookeeper ensemble |-----------| kguard |-------------+    |
 |            |             +---|                    |           |        | aggragator  |    |
 |            |             |   +--------------------+           +--------+             |    |
 |            |     +-------+           |                                               |    |
 |            |     | registry          | orchestration                                 |    |      +- Pub
 |      +---------------+               |-----------+                      +---------+  |    | REST |
 |      |               |               |           |                      | kateway |--|----|------|
 |  +---------+    +---------+      +--------+    +--------+               +---------+  |    |      |
 |  | kateway |    | kateway |      | actord |    | actord |                            |    |      +- Sub
 |  +---------+    +---------+      +--------+    +--------+                            |    |
 |    | hh |         | hh |             | executor                                      |    |
 |    +----+         +----+          +--------------+                                   |    |
 |                      |            |              |                                   |    |  
 |                      |       +---------+    +---------+  push                        |    |  
 |             +--------+       | JobTube |    | Webhook |------------>-----------------|----|---Endpoints
 |             |        |       +---------+    +---------+                              |    |
 |             |        |           | scheduler     | sub                               |    |
 |        auth |        |job WAL    | dispatch      |                                   |    |
 |      +------+        +---------------------------+------------------+                |    |
 |      |               | tenant shard              | pubsub           | flush          |    |
 |  +----------+    +---------+                 +-------+           +------+            |    |
 |  | auth DB  |    | DB Farm |                 | kafka |           | TSDB |            |    |
 |  +----------+    +---------+                 +-------+           +------+            |    |
 |      |               |                           |                  |                |    |   
 |      |               +----------------------------------------------+                |    |  
 |      |                                   |                                           |    | 
 |      |                                   +-------------------------------------------+    |
 |      |                                                                                    |  
 |      |                                                                               zone |   
 +-------------------------------------------------------------------------------------------+
        |
    WebConsole 

Install

export PATH=$PATH:$GOPATH/bin

#========================================
# install go-bindata and go annotations
#========================================
go install github.com/jteeuwen/go-bindata/go-bindata
go install github.com/funkygao/goannotation

#========================================
# install gafka
#========================================
go get github.com/funkygao/gafka
cd $GOPATH/src/github.com/funkygao/gafka
./build.sh -a # build all components

#========================================
# try the gafka command 'gk'
#========================================
gk -h

Status

Currently gafka manages:

• 4 data centers
• 50+ kafka clusters
• 100+ kafka brokers
• 500+ kafka topics
• 2000+ kafka partitions
• 10Billion messages per day
• peak load
  • 1Million message per second
  • 8TB transfered per hour