How to mark data from multiple clusters

[EsDmitrii]

Hi!
I'm new to Mimir and I can't find any related docs about how to mark data from multiple clusters using ruler (may be it's not ruler's work).
I set up Mimir using these values and official docs.
I use s3 minio to store data and this is how it looks now:

As you can see, there is no structuring on any basis, folder names have incomprehensible names, and etc.

I created two VictoriaMetrics Agents in two clusters that push data to Mimir.
I want to mark data or create its own folders for every cluster something like cluster name == folder name where data is stored.
What should I use? I think I should configure ruler, but I can't find any documentation how to do it using helm chat.
Can I write all data ruler, tsdb, alermanager, etc to one s3 or do I need to create multiple buckets for these components?

And another thing that I found, that Mimir use user anonymous to push data (why?). How can I change the name of this user or change the folder name

May be someone has reference values to deploy production-ready Mimir for multiple k8s clusters? (in my case there are a lot of k8s, about 50 clusters)

Thank you in advance!

[pstibrany]

Hello,

I want to mark data or create its own folders for every cluster something like cluster name == folder name where data is stored.
What should I use? I think I should configure ruler, but I can't find any documentation how to do it using helm chat.

anonymous is simply a tenant name. You need to

1. enable multitenancy in Mimir
2. set up your vmagent (or Grafana Agent, or Prometheus, or anything that writes data to Mimir) to send X-Scope-OrgID HTTP header, when sending data to Mimir. Value of that header will be "tenant", Mimir will store data for different tenants into separate parts of the storage. (Looks like vmagent supports that feature: vmagent: custom http headers for remote write requests VictoriaMetrics/VictoriaMetrics#2805).

(Each tenant in Mimir can have separate set of alerting rules or configuration options)

[EsDmitrii]

Hi!
Thank you!
Will try.

Now I turned on tests (set remote write to Mimir from little prod cluster) and I faced a lot of 400/429 errors
As I understand there no deduplication enabled by default?
Cant find any related docs for helm.
I only found this https://grafana.com/docs/mimir/latest/operators-guide/mimir-runbooks/#err-mimir-tenant-max-ingestion-rate, but how to set -distributor.ingestion-rate-limit via Helm?

This agent writes ~15K samples per second. This is little prod cluster, not huge.. and this is only the beginning

[EsDmitrii]

thanks to the Medium that it is exists:)
Found this topic https://tech.loveholidays.com/grafana-mimir-our-journey-towards-infinite-wisdom-with-5m-active-time-series-7a262ba53a3f

Where can I find all values and options that I can set?

[pstibrany]

Where can I find all values and options that I can set?

You can check values.yaml file in Helm chart https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/values.yaml, and for complete list of command line and YAML options you can consult https://grafana.com/docs/mimir/latest/references/configuration-parameters/

[EsDmitrii]

thank you a lot
this is what I need