HA Monolithic configuration

[TarekAS]

Hi!

I'm looking to deploy Grafana Mimir in monolithic mode on Kubernetes, horizontally scalable, with S3 as backend (without minio).

I made a simple chart that has one statefulset with -target=all, one service with all the relevant ports, and the configmap. (side-note: it would be nice if you provided an official monolithic-setup chart)

My question is, do you have a recommended/example configuration of a HA monolithic setup?

I rewrote the default one provided by the Helm chart. This is the best I could come up with:

activity_tracker:
  filepath: /data/metrics-activity.log

alertmanager:
  data_dir: /data
  enable_api: true
  external_url: /alertmanager

alertmanager_storage:
  backend: s3
  s3:
    bucket_name: "bucketname-ruler" #TODO
    endpoint: "s3.eu-west-1.amazonaws.com"

blocks_storage:
  backend: s3
  bucket_store:
    sync_dir: /data/tsdb-sync
  s3:
    bucket_name: "bucketname-tsdb" #TODO
    endpoint: "s3.eu-west-1.amazonaws.com"
  tsdb:
    dir: /data/tsdb

compactor:
  data_dir: "/data"

frontend:
  align_queries_with_step: true
  log_queries_longer_than: 10s

frontend_worker:
  frontend_address: {{ .Release.Name }}-mimir.{{ .Release.Namespace }}.svc:9095

ingester:
  ring:
    final_sleep: 0s
    num_tokens: 512
    unregister_on_shutdown: false

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600

limits: {}

memberlist:
  abort_if_cluster_join_fails: false
  compression_enabled: false
  join_members:
  - {{ .Release.Name }}-mimir

ruler:
  alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ .Release.Name }}-mimir.{{ .Release.Namespace }}.svc.cluster.local/alertmanager
  enable_api: true
  rule_path: /data

ruler_storage:
  backend: s3
  s3:
    bucket_name: "bucketname-ruler" #TODO
    endpoint: "s3.eu-west-1.amazonaws.com"

runtime_config:
  file: /var/mimir/runtime.yaml

server:
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

Are there any incorrect or missing options that I should address?

[pracucci]

The Mimir default configuration is already suited for HA and requires to run at least 3 Mimir replicas. I checked your config and you're not changing HA settings, so from an HA perspective it should be good.

That being said, I would remove the following config overrides and use the default values:

ingester:
  ring:
    final_sleep: 0s
    num_tokens: 512
    unregister_on_shutdown: false

ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600

memberlist:
  compression_enabled: false

server:
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

Finally, you could take a look at our "Play with Grafana Mimir" tutorial, which runs 3 replicas of Mimir in monolithic mode:
https://github.com/grafana/mimir/tree/main/docs/sources/tutorials/play-with-grafana-mimir

[SunSparc]

Looks like the link has changed to: https://github.com/grafana/mimir/tree/main/docs/sources/mimir/tutorials/play-with-grafana-mimir

[ghost]

Having checked both this config and the tutorial configs, I have an additional question.
For archiving purposes I have my metrics written to a grafana-cloud hosted instance and a local install using monolithic mimir on docker-compose (so no Kubernetes) using HA prometheus setup.
Is there a way to have the dedupe proces running using the gossip protocol instead of KV or etcd?

[pracucci]

Is there a way to have the dedupe proces running using the gossip protocol instead of KV or etcd?

Not as of today. We either require consul or etcd for HA deduplication.