Connecting Two Prometheus to One Mimir Instance

[bulmust]

We have a Mimir instance, Prometheus and Grafana in a k8s cluster. We can connect (I suppose) another Prometheus to Mimir in a different k8s cluster. In Mimir-Distributor UI, all_user_stats shows both tenant-ids. We can add Mimir data as a data source in Grafana.

Here is our struggling point. When we try to run query for Mimir data source in Grafana, we obtain only first Prometheus metrics which is in the same cluster with Mimir. Second Prometheus is "non-queryable". We try to add externalLabels into prometheuses instances but doesn't work. Also mimir-default dasboards does not work (it tries to obtain data from cortex_ metrics which do not exist).

Can you suggest anything?

Maybe, we don't using Mimir concept. Configure all external Prometheuses remote_write to single Mimir than all metrics will be stored and available to any query using Mimir. Do you have suggestions?

Mimir is installed with helm default values.

Prometheus config: (I can see kindtest-1 label in Grafana - Metric Query)

prometheus:
  prometheusSpec:
    externalLabels:
      cluster_name: kindtest-1
    remoteWrite:
      - name: centralized-mimir
        url: http://mimir-nginx.k8s.local/api/v1/push # Ingress
        headers:
          X-Scope-OrgID: kindtest-1

Possible duplication: #1656

[charleskorn]

@bulmust, I'm a little unclear what you're trying to achieve. Is this understanding correct?

• You have one Mimir instance
• You have one Grafana instance
• You have two Prometheus instances, each remote writing to the Mimir instance
• You'd like to be able to query metrics written by both Prometheus instances through one Grafana data source connected to the Mimir instance

If so, have you configured the Prometheus instances to remote write with the same X-Scope-OrgID? Or do they each have different values?

If you're using different values, have you configured the Mimir data source in Grafana with both org IDs? Or just one?

[bulmust]

If so, have you configured the Prometheus instances to remote write with the same X-Scope-OrgID? Or do they each have different values?

They have different X-Scope-OrgID. I did not write second prometheus configuration because it is same except all kindtest-1 strings become kindtest-2.

If you're using different values, have you configured the Mimir data source in Grafana with both org IDs? Or just one?

I think, I miss that part. Do I have to manually (or in Grafana yaml) add all X-Scope-OrgID values for every prometheus instances? Is there any easy way to do that?

"Grafana -> Connections -> Data sources -> Mimir Datasource -> Custom HTTP Headers"

[charleskorn]

Do I have to manually (or in Grafana yaml) add all X-Scope-OrgID values for every prometheus instances? Is there any easy way to do that?

Yep, for the Mimir datasource, include them as the tenant ID separated by pipe characters (|). For example, kindtest-1|kindtest-2.

[bulmust]

Thank you

[bulmust]

Note that, if you want to use pipe character in header, you have to enable mimir tenant federation in mimir helm chart.

mimir:
  structuredConfig:
    tenant_federation:
      enabled: true

[bulmust]

If you want to connect two Prometheus with same headers, keep X-Scope-OrgID value the same. You can distinguish metrics with external labels in Grafana.

First Prometheus instance:

prometheus:
  prometheusSpec:
    externalLabels:
      cluster_name: kindtest-1
    remoteWrite:
      - name: centralized-mimir
        url: http://mimir-nginx.k8s.local/api/v1/push # Ingress
        headers:
          X-Scope-OrgID: test

Second Prometheus instance:

prometheus:
  prometheusSpec:
    externalLabels:
      cluster_name: kindtest-2
    remoteWrite:
      - name: centralized-mimir
        url: http://mimir-nginx.k8s.local/api/v1/push # Ingress
        headers:
          X-Scope-OrgID: test