Redis Cluster Support

[zackman0010]

Describe the bug

When using Redis Cluster as a cache, the following log lines are frequently printed at a WARN level:

ts=2023-08-31T18:45:27.769115044Z caller=redis_client.go:242 level=warn name=metadata-cache msg="failed to mget items from redis" err="CROSSSLOT Keys in request don't hash to the same slot" items=0
ts=2023-08-31T18:47:01.966078844Z caller=redis_client.go:242 level=warn name=chunks-cache msg="failed to mget items from redis" err="CROSSSLOT Keys in request don't hash to the same slot" items=0
ts=2023-08-31T18:47:02.823316454Z caller=redis_client.go:242 level=warn name=index-cache msg="failed to mget items from redis" err="CROSSSLOT Keys in request don't hash to the same slot" items=0

To Reproduce

Steps to reproduce the behavior:

1. Configure Mimir to point to a Redis Cluster cache backend
2. Start Mimir version 2.9.0 (branch: release-2.9, revision: 761114d8b)

Expected behavior

No errors in the logs, functioning Redis cache

Environment

• Infrastructure: bare-metal
• Deployment tool: manual

Additional Context

This error is due to the way Redis Cluster hashes the keys. It can only do multi-key operations if all keys result in the same hash. Redis supports the concept of "Hash Keys", where it will only hash a subset of the key surrounded by {}. Any keys that will be used in a multi-key query should utilize the same hash key so they're guaranteed to fall in the same hash slot.
Example: key1 and key2 will hash to different slots, but {hashkey}.key1 and {hashkey}.key2 are guaranteed to hash to the same slot.

Alternatively, I saw that both Loki and Tempo have an isCluster check to only use multi-key operations if not operating in Cluster mode. This check is not present in Mimir.

[dimitarvdimitrov]

Mimir is using dskit for its redis client. I opened a dskit issue to track this grafana/dskit#373

[56quarters]

Redis support is deprecated now and being removed in the next major release (no ETA): #9453