Ingester service S3 storage.

[ldvy]

Hi folks,
I have deployed Mimir to my GKE cluster using the Mimir-Distributed helm chart, while configuring the GCS backend for "blocks_storage", "alertmanager_storage", and "ruler_storage":
mimir: structuredConfig: blocks_storage: storage_prefix: blocks backend: gcs gcs: bucket_name: my_bucket ruler_storage: backend: gcs gcs: bucket_name: my_bucket alertmanager_storage: backend: gcs gcs: bucket_name: my_bucket
Everything works well, and I can see data in the GCS bucket.

Recently I noticed that the ingester pods began crashing with the following error:
level=error ts=2022-10-18T16:10:48.390335341Z caller=ingester.go:1672 msg="unable to open TSDB" err="failed to compact TSDB: /data/tsdb/<some folder>: compact head: persist head block: 2 errors: populate block: adding series: asyncBlockWriter doesn't run anymore: write chunks: preallocate: no space left on device; write /data/tsdb/<something>: no space left on device" user=<something>

Since I am using a GCS bucket in my "blocks_storage" clause, shouldn't the time series also be stored in the GCS bucket?

I would appreciate any help on this matter!

[56quarters]

Ingesters use the local disk for the Write Ahead Log (WAL) and most recent few hours of samples before uploading. They also keep 24h of blocks on the local disk (by default) as a performance optimization. If the default PVC size (50G I think) isn't big enough, you'll have to grow it.

More information about the ingesters in the docs.