We need to get alerted when Loki or Mimir cannot access object storage

[QuantumEnigmaa]

On my latest oncall shift, I got paged by a MimirCompactorFailedCompaction alert but after investigations, I found out that the root issue was that the compactor as well as other mimir components were unable to access the object storage. After further checks, I discovered that it was the same case for loki.

TL,DR the IRSA was broken and both loki and mimir couldn't access the object storage but we didn't get paged for this. Hence, we should create an alert that would page whenever any of these apps is in such a situation.

For more details, you can check the related incident.

[hervenicol]

Not sure which metrics we can rely on.
Logs are tempting, but if object storage access is broken, we won't have logs.

Mimir metrics: cortex_bucket_store_sent_chunk_size_bytes_count?
Loki equivalent: loki_rate_store_stream_rate_bytes_count?

[hervenicol]

Alert deployed and working.