Mixin dashboards - cluster label

[djetelina]

Hi,

I have a mimir cluster deployed in our kubernetes through a helm chart. I've also found the mixin-compiled folder and got super excited, however while most of the dashboards worked out of the box, some don't.

I've identified the cause being cluster label being expected at certain places (everything to do with kube state metrics, e.g. max by(pod) (container_memory_working_set_bytes{cluster=~"$cluster", namespace=~"$namespace",container=~"alertmanager"})). It's possible I completely missed a part of documentation, but all of my searches left me emptyhanded.

So I have a few questions:

• Are the mixin compiled dashboards supposed to work out of the box?
• Is there a guide somewhere how to configure kube state metrics so that they are able to discover and expose the cluster label?
• Is the cluster label even necessary for use cases where there's only one cluster in a namespace?

I hope these questions aren't stupid and I that I didn't just fail at finding existing answers. Thank you for all your amazing work!

[pracucci]

Are the mixin compiled dashboards supposed to work out of the box?

The compiled dashboards have some requirements in terms of labels expected on your metrics (including the cluster label). The requirements are documented here. If you can't comply to the requirements, you can change the mixin config and recompile the dashboards: it's very simple, just run the command documented here.

Is there a guide somewhere how to configure kube state metrics so that they are able to discover and expose the cluster label?

The documentation we have is here. You're right that it doesn't explain how to set the cluster label (I would be glad if you could help improving it). To answer your question, the cluster label should be set to the Kubernetes cluster "name" (or identifier); typically, you run a Prometheus/Agent for each Kubernetes cluster, so you could just configure the cluster label as a external label of that Prometheus server.

Is the cluster label even necessary for use cases where there's only one cluster in a namespace?

The cluster label is not about Mimir cluster (which is identified by the namespace) but the Kubernetes cluster. The mixin is designed to support multiple Kubernetes clusters.

[dimitarvdimitrov]

To add to what @pracucci already said: the helm chart comes with metamonitoring, which takes care of putting the right labels so the mixin (rules & dashboards) works out of the box. You can see how to set that up in Collecting metrics and logs from Grafana Mimir

[djetelina]

Thank you very much! I feel embarassed that I did not find the doc pages linked by you 🤦 .

It seems that I even misunderstood the cluster value in the helm values. For some reason I thought it was supposed to be the name of the mimir cluster (we're temporarily running two side-by-side due to transitioning to multi tenancy). But it being the name of the kubernetes cluster makes a lot more sense for literally everyone else :D

Once we stop abusing the cluster label for separating between our two mimir clusters, we'll transition to identifying kube cluster and see if we can contribute in some way then. Although it seems like this was really mostly my/our blindness to already existing resources, rather than lacking documentation.

Thanks again and have a great day!

[pracucci]

I feel embarassed that I did not find the doc pages linked by you

Don't feel it! Can you tell me if you where using the "Search docs" feature at https://grafana.com/docs/mimir/latest/ or looking for it through the index? I would like to better understand how we could improve the discoverability. Thanks!

[djetelina]

Even though I've browsed the hosted docs before quite a bit (and used both search and index for other topics), it just didn't occur to me to look for mixin related information there, so I was mostly going through issues in this repository (and readme), I also tried some google searches with bad keywords (I wish I knew which, as I'm trying to replicate it the docs are coming up at the top for all the normal keywords). Retrospectively I can't find anything else to blame other than myself :D