diff --git a/docs/sources/mimir/configure/configure-spread-minimizing-tokens/index.md b/docs/sources/mimir/configure/configure-spread-minimizing-tokens/index.md index 31ca35e75f2..bf0d62429ac 100644 --- a/docs/sources/mimir/configure/configure-spread-minimizing-tokens/index.md +++ b/docs/sources/mimir/configure/configure-spread-minimizing-tokens/index.md @@ -15,9 +15,14 @@ Using this guide, you can configure Mimir's ingesters to use the _spread-minimiz The ingester time series replication should be [configured with enabled zone-awareness](https://grafana.com/docs/mimir/latest/configure/configure-zone-aware-replication/#configuring-ingester-time-series-replication). -{{% admonition type="note" %}}Spread-mimizing tokens are recommended if [shuffle sharding](https://grafana.com/docs/mimir/latest/configure/configure-shuffle-sharding/#ingesters-shuffle-sharding) is disabled in your ingesters, or, if shuffle-sharding is enabled, but most of the tenants of your system use all available ingesters. +{{% admonition type="note" %}}Spread-mimizing tokens are recommended if [shuffle-sharding](https://grafana.com/docs/mimir/latest/configure/configure-shuffle-sharding/#ingesters-shuffle-sharding) is disabled on the [write path](https://grafana.com/docs/mimir/latest/configure/configure-shuffle-sharding/#ingesters-write-path) of your ingesters, or, if it is enabled, but most of the tenants of your system use all available ingesters. {{% /admonition %}} +{{% admonition type="note" %}}In order to prevent incorrect query results, [shuffle-sharding](https://grafana.com/docs/mimir/latest/configure/configure-shuffle-sharding/#ingesters-shuffle-sharding) on the [read path](https://grafana.com/docs/mimir/latest/configure/configure-shuffle-sharding/#ingesters-read-path) of your ingesters must be disabled before migrating ingesters to the spread-minimizing tokens. Shuffle-sharding on ingester's read path can be re-enabled at least `-querier.query-store-after` time after the last ingester zone was migrated to the spread-minimizing tokens. +{{% /admonition %}} + +If ingesters are configured with a non-empty value of `-ingester.ring.tokens-file-path`, this is the file where ingesters store the tokens at shutdown and restore them at startup. Keep track of this value, because you need it in the last step. + For simplicity, let’s assume that there are three configured availability zones named `zone-a`, `zone-b`, and `zone-c`. Migration to the _spread-minimizing token generation strategy_ is a complex process performed zone by zone to prevent any data loss. ## Step 1: Disable write requests to ingesters from `zone-a` @@ -103,6 +108,16 @@ If everything went smoothly, you should see something like this: Repeat steps 1 to 4, replacing all the occurrences of `zone-a` with `zone-b`. -## Step 6: migrate ingesters from `zone-c` +## Step 6: Migrate ingesters from `zone-c` Repeat steps 1 to 4, replacing all the occurrences of `zone-a` with `zone-c`. + +## Step 7: Delete the old token files + +If, before the migration, you configured ingesters to store their tokens under `-ingester.ring.tokens-file-path`, you must delete these files after migrating all ingester zones to spread-minimizing tokens. + +For example, if an ingester pod called `ingester-zone-a` from a namespace called `mimir-prod` used to store its tokens in a file called `/data/tokens`, you can run the following command to delete the `/data/tokens` file: + +``` +kubectl -n mimir-prod exec ingester-zone-a-0 -- rm /data/tokens +```