Save Kafka data on Windows host drive

[hhashim1]

Here is the docker info that you can use to host kafka data on the host drive in Windows. I am using deephaven with redpanda so some of the configuration details might be for deephaven.

The reason for saving data on the host drive as opposed to the docker image is that if you have a high-velocity stream (ex. stock market stream), you can fill up the space on the docker image in just a few days, and then you will have issues with redpanda not starting. To overcome this issue, you need to save data to the host drive.

The syntax for the drive path is as follows. This section would go under services->redpanda.

    volumes:
      - E:/redpanda:/var/lib/redpanda/data`

The path on the left of the colon above is your Windows directory where you want Kafka data to be saved (instead of in the docker image). This path can be changed depending on your preference but make sure the drive has plenty of space as the space can get filled very quickly in a high-velocity stream.

The path to the right of the colon is the path that redpanda/kafka needs for docker. DO NOT change this path as this is the path mapped to your Windows folder. Anything other than this path will result in data being saved on the docker image.

Here is the full docker-compose file.

version: "3.4"

services:
  redpanda:
    command:
      - redpanda
      - start
      - --check=false
      - --smp
      - '1'
      - --reserve-memory
      - 0M
      - --overprovisioned
      - --node-id
      - '0'
      - --kafka-addr
      - PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr
      - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    image: docker.vectorized.io/vectorized/redpanda:latest
    ports:
      - 9092:9092
      - 29092:29092
    volumes:
      - E:/redpanda:/var/lib/redpanda/data
      
  create_topic:
    depends_on:
      - redpanda
    entrypoint:
      - bash
      - -c
      - "sleep 30; rpk topic create example_topic --brokers=redpanda:29092 -v"
    image: docker.vectorized.io/vectorized/redpanda:latest

  server:
    image: ghcr.io/deephaven/server:${VERSION:-latest}
    environment:
      JAVA_TOOL_OPTIONS: -Xmx12g -Ddeephaven.console.type=python
    depends_on:
        - redpanda
    expose:
      - '8080'
    volumes:
      - ./data:/data
      - ./data/notebooks:/data/notebooks
      - api-cache:/cache

  web:
    image: ghcr.io/deephaven/web:${VERSION:-latest}
    environment:
      JAVA_TOOL_OPTIONS: -Xmx6g
    expose:
      - '80'
    volumes:
      - ./data:/data
      - ./data/layouts:/data/layouts
      - ./data/notebooks:/data/notebooks
      - web-tmp:/tmp

  grpc-proxy:
    image: ghcr.io/deephaven/grpc-proxy:${VERSION:-edge}
    environment:
      - BACKEND_ADDR=server:8080
    expose:
      - '8080'

  envoy:
    image: ghcr.io/deephaven/envoy:${VERSION:-edge}
    ports:
      - "${PORT:-10000}:10000"

volumes:
    web-tmp:
    api-cache: