prometheus remote write multi-zone replica issues

[jgutschon]

Hello, I am working on deploying version 3.0.0 of the mimir helm chart and I'm currently having some issues getting Prometheus remote write to work properly. When prometheus attempts to send metrics to mimir (through the mimir-nginx service), the distributor pods log this error repeatedly:

msg="POST /api/v1/push (500) 1.999681ms Response: \"at least 2 live replicas required across different availability zones, could only find 1\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 43721; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.32.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: anonymous;

This same error is also visible in the prometheus logs in the response to the remote write request. I'm not sure which component it's referring to here, but I've verified that both the ingester and distributor services are being replicated across two availability zones.

My helm values.yaml configuration is as follows:

---
mimir-distributed:
  fullnameOverride: "mimir"

  mimir:
    structuredConfig:
      multitenancy_enabled: false

      limits:
        accept_ha_samples: true

      distributor:
        ha_tracker:
          enable_ha_tracker: true
          kvstore:
            store: inmemory

      ingester:
        ring:
          zone_awareness_enabled: true

      store_gateway:
        sharding_ring:
          zone_awareness_enabled: true

      frontend:
        parallelize_shardable_queries: true

  nginx:
    ingress:
      tls: {}

  rbac:
    create: false

  compactor:
    replicas: 1

  distributor:
    replicas: 4

  ingester:
    replicas: 4
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - ingester
            topologyKey: topology.kubernetes.io/zone

  chunks-cache:
    enabled: true
    replicas: 2

  index-cache:
    enabled: true
    replicas: 3

  metadata-cache:
    enabled: true
    replicas: 1

  results-cache:
    enabled: true
    replicas: 1

  minio:
    enabled: false

  overrides_exporter:
    enabled: false

  alertmanager:
    enabled: false

  ruler:
    enabled: false

  store_gateway:
    replicas: 1

Any help would be greatly appreciated, thanks!

[dimitarvdimitrov]

there is an ongoing PR to enable multi-zone natively for the chart #2778. It think it will be ready or released in the next 2-3 weeks. If you can wait, that'd be easier for you.

Otherwise, I think you're not setting the CLI flags for which zone each instance is in. Have you seen this article https://grafana.com/docs/mimir/latest/operators-guide/configure/configuring-zone-aware-replication/ ?

[jgutschon]

That would definitely be helpful, I also noticed that topologySpreadConstraints are being added in #2722 which will also help.

The main problem turned out to be that the podAntiAffinity rules were based on the target label which isn't on the pods anymore, so I ended up just overwriting these rules with the topology.kubernetes.io/zone label instead which is now yielding the expected behaviour. We're running a fairly small cluster for now so this should be good enough until the next release.

I also didn't notice right away that mimir needs 3 zones and the node group this is being deployed into only has two zones atm, so fixing these affinity rules and adding an extra node in a third zone should fix it.

[dimitarvdimitrov]

That would definitely be helpful, I also noticed that topologySpreadConstraints are being added in #2722 which will also help.

I just released 3.1.0-weekly.203, which also has the TSC

The main problem turned out to be that the podAntiAffinity rules were based on the target label which isn't on the pods

yep, I think this was addressed by #2772 as well. The issue we have is #2681 and #2906 is the final PR to close it.

mimir needs 3 zones
👍 it should be the same as the replication factor, which is 3 by default

[mdvaliyullah]

how to resolve this error, I am also getting the same error in multicluster setup