CrateDB Prometheus Adapter

This is an adapter that accepts Prometheus remote read/write requests, and sends them on to CrateDB. This allows using CrateDB as long term storage for Prometheus.

Requires CrateDB 3.1.0 or greater.

Building

go get github.com/crate/crate_adapter
cd ${GOPATH-$HOME/go}/src/github.com/crate/crate_adapter
go build

Usage

Create the following table in your CrateDB database:

CREATE TABLE "metrics" (
    "timestamp" TIMESTAMP,
    "labels_hash" STRING,
    "labels" OBJECT(DYNAMIC),
    "value" DOUBLE,
    "value_raw" LONG,
    "day__generated" TIMESTAMP GENERATED ALWAYS AS date_trunc('day', "timestamp"),
    PRIMARY KEY ("timestamp", "labels_hash", "day__generated")
  ) PARTITIONED BY ("day__generated");

Depending on data volume and retention you might want to optimize your partitioning scheme and create hourly, weekly,... partitions.

Then run the adapter:

./crate_adapter

By default the adapter will listen on port 9268. This and more is configurable via command line flags, which you can see by passing the -h flag.

The CrateDB endpoints are provided in a configuration file, which defaults to config.yml (-config.file flag). The included example configuration file forwards samples to a CrateDB running on localhost on port 5432.

Adapter Crate Endpoint Configuration

The CrateDB endpoints that the adapter writes to are configured in a YAML-based configuration file provided by the -config.file flag. If multiple endpoints are listed, the adapter will load-balance between them. The options (for one example endpoint) are as below:

crate_endpoints:
- host: "localhost"         # Host to connect to (default: "localhost").
  port: 5432                # Port to connect to (default: 5432).
  user: "crate"             # Username to use (default: "crate")
  password: "mypass"        # Password to use (default: "").
  schema: "prometheus"      # Schema to use (default: "").
  max_connections: 5        # The maximum number of concurrent connections (default: 5).
  enable_tls: false         # Whether to connect using TLS (default: false).
  allow_insecure_tls: false # Whether to allow insecure / invalid TLS certificates (default: false).

Prometheus Configuration

Add the following to your prometheus.yml:

remote_write:
   - url: http://localhost:9268/write
remote_read:
   - url: http://localhost:9268/read

The adapter also exposes Prometheus metrics on /metrics, and can be scraped in the usual fashion.

Running with systemd

Copy config.yml to /etc/crate_adapter/config.yml and adjust as needed.

Copy systemd/crate_adapter.service to /etc/systemd/system/crate_adapter.service or just link the service file by running: sudo systemctl link $(pwd)/crate_adapter.service and run:

systemctl daemon-reload

Change flag-based configuration by changing the settings in /etc/default/crate_adapter based on the systemd/crate_adapter.default template. After that you can:

systemctl start crate_adapter
systemctl enable crate_adapter