Kafka Connect Elasticsearch

Stream data from Apache Kafka to Elasticsearch using Kafka Connect.

Usage

1. Install Elasticsearch 2.3.3

   cd elasticsearch-2.3.3
   bin/elasticsearch

2. Install Confluent Platform 2.0.1

   cd confluent-2.0.1

   terminal 1

   /bin/zookeeper-server-start ./etc/kafka/zookeeper.properties

   terminal 2

   /bin/kafka-server-start ./etc/kafka/server.properties

3. Clone this project and build

   git clone https://github.com/yuzutech/kafka-connect-elasticsearch.git
   cd kafka-connect-elasticsearch
   mvn clean package

4. Export CLASSPATH and launch Kafka connect in standalone mode

   export CLASSPATH=/path/to/kafka-connect-elasticsearch/workspace/target/kafka-connect-elasticsearch-all.jar
   cd confluent-2.0.1
   cp /path/to/kafka-connect-elasticsearch/workspace/samples/config/standalone.properties .
   cp /path/to/kafka-connect-elasticsearch/workspace/samples/config/connector.properties .
   bin/connect-standalone standalone.properties connector.properties

5. Start Kafka REST Proxy

   terminal 3

   bin/kafka-rest-start

6. Produce JSON messages

   curl -X POST -H "Content-Type: application/vnd.kafka.json.v1+json" \
         --data '{"records":[{"value":{"foo":"bar"}}]}' "http://localhost:8082/topics/baz"

7. View your data in Elasticsearch!

   curl -XGET http://localhost:9200/qux/_search