err-mimir-store-consistency-check-failed

[alfianabdi]

Describe the bug

Recently I keep getting error from querier 'err-mimir-store-consistency-check-failed'.
After looking into store-gateway, the requested block is not there, but the said block exists inside the storage ( S3 ) and inside the index.
Looking the logs, there is no error. It even says that the synchronization is successful

level=info ts=2022-10-28T11:10:49.850096057Z caller=gateway.go:294 msg="successfully synchronized TSDB blocks for all users" reason=periodic

To Reproduce

Steps to reproduce the behavior:

Expected behavior

Store gateway is able to download the required block.

Environment

• Infrastructure: Kubernetes ( eks 1.22 )
• Deployment tool: Helm ( helm distributed 3.0.0, migrated from cortex 1.4.0 )

Additional Context

Config

activity_tracker:
  filepath: /data/mimir/activity/metrics-activity.log
alertmanager:
  data_dir: /data
  external_url: /api/prom/alertmanager
  sharding_ring:
    kvstore:
      store: memberlist
alertmanager_storage:
  backend: s3
  s3:
    bucket_name: <REDACTED>
    endpoint: s3.ap-southeast-1.amazonaws.com
api:
  prometheus_http_prefix: /prometheus
blocks_storage:
  backend: s3
  bucket_store:
    sync_dir: data
  s3:
    bucket_name: <REDACTED>
    endpoint: s3.ap-southeast-1.amazonaws.com
  tsdb:
    dir: /data
compactor:
  data_dir: /data
  sharding_ring:
    kvstore:
      store: memberlist
distributor:
  ha_tracker:
    enable_ha_tracker: true
    kvstore:
      consul:
        acl_token: ${CONSUL_TOKEN}
        host: <REDACTED>
      prefix: cortex/ha-tracker/
      store: consul
  pool:
    health_check_ingesters: true
  ring:
    kvstore:
      store: memberlist
frontend:
  align_queries_with_step: true
  cache_results: false
  grpc_client_config:
    grpc_compression: snappy
  log_queries_longer_than: 10s
  results_cache:
    memcached:
      timeout: 1s
  split_queries_by_interval: 24h0m0s
frontend_worker:
  frontend_address: cortex-query-frontend-headless.monitoring.svc:9095
  grpc_client_config:
    grpc_compression: snappy
ingester:
  ring:
    kvstore:
      store: memberlist
    num_tokens: 512
    replication_factor: 2
ingester_client:
  grpc_client_config:
    grpc_compression: snappy
limits:
  accept_ha_samples: true
  ha_max_clusters: 3
  ingestion_burst_size: 500000
  ingestion_rate: 250000
  max_global_series_per_metric: 200000
  max_global_series_per_user: 0
memberlist:
  join_members:
  - cortex-gossip-ring
multitenancy_enabled: true
no_auth_tenant: fake
querier:
  query_ingesters_within: 8h5m0s
  query_store_after: 8h0m0s
ruler:
  alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.cortex-alertmanager-headless.monitoring.svc.cluster.local/alertmanager
  ring:
    kvstore:
      store: memberlist
  rule_path: /data/rules
ruler_storage:
  backend: s3
  s3:
    bucket_name: <REDACTED>
    endpoint: s3.ap-southeast-1.amazonaws.com
server:
  log_level: info

[pracucci]

👋 This custom configuration:

querier:
  query_ingesters_within: 8h5m0s
  query_store_after: 8h0m0s

Conflicts with the default store-gateway configuration:

  # (advanced) Blocks with minimum time within this duration are ignored, and
  # not loaded by store-gateway. Useful when used together with
  # -querier.query-store-after to prevent loading young blocks, because there
  # are usually many of them (depending on number of ingesters) and they are not
  # yet compacted. Negative values or 0 disable the filter.
  # CLI flag: -blocks-storage.bucket-store.ignore-blocks-within
  [ignore_blocks_within: <duration> | default = 10h]

Please set:

blocks_storage:
  bucket_store:
    ignore_blocks_within: 6h

[vikasbudhwat]

Thanks for suggestion, but could you plz suggest how to keep my store-gateway storage clean? any config which makes sure only limited duration blocks index-headers are synched so that store gateway will not filled 100%.?

[pracucci]

any config which makes sure only limited duration blocks index-headers are synched so that store gateway will not filled 100%.?

To get Mimir working correctly, all blocks index-header (older than ignore_blocks_within) must be kepts on store-gateway disks. If a store-gateway disk is filling up, you have the following options:

1. Increase disk size
2. Scale out store-gateways (add more replicas, since blocks are sharded)
3. Reduce the configured storage retention so that less blocks will be kept in the whole storage (but keep in mind that this mean any data older than the configured reletion will be deleted)

[vikasbudhwat]

mimir-config.txt

Attached complete mimir config file.
thought we have mentioned in config retention_period: 45m
but index header has indexes for more than 30 days blocks.

May be I am missing something but ultimately we need to have retention for blocks and index-headers so we can manage with given storage volume.

[pracucci]

May be I am missing something but ultimately we need to have retention for blocks and index-headers so we can manage with given storage volume.

I see you're setting:

blocks_storage:
  tsdb:
    retention_period: 45m

This retention is about the TSDB blocks stored in ingester. I discourage you from changing it and changing block_ranges_period: 15m (which we exposed only because of testing but shouldn't be really changed cause Mimir makes some assumptions on 2h blocks).

That being said, you mentioned you want to reduce the disk utilization of store-gateways. The TSDB retention only affect ingesters, not store-gateways. Store-gateways disk utilization is a function of the total size of blocks you have in the long-term storage. By default, Mimir keeps blocks in the long-term storage forever, but you can configure a retention using this configuration option:

# Delete blocks containing samples older than the specified retention period.
# Also used by query-frontend to avoid querying beyond the retention period. 0
# to disable.
# CLI flag: -compactor.blocks-retention-period
[compactor_blocks_retention_period: <duration> | default = 0s]

[vikasbudhwat]

Thanks for detail response. will revert to default config values..