Mimir query is slower then Prometheus

[kpinarci]

Mimir query is slower than Prometheus

Deployment Environment

• Os: Suse Linux 15
• Docker: 20.10.17-ce
• Docker-Compose: 1.25.1
• Grafana Mimir: 2.4.0
• Mimir deployment mode: monolith
• Count of mimir container: 1
• Reverse proxy was deliberately omitted

docker-compose

version: "3.5"
services: 
   mimir:
    build:
      context: .
      dockerfile: Dockerfile
    restart: always
    container_name: {{ mimir_container_name }}
    command: ["-config.file=/mimir/mimir.yaml"]
    hostname: {{ mimir_hostname }}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - {{ mimir_conf_dir }}/mimir.yaml:/mimir/mimir.yaml
      - {{ mimir_conf_dir }}/alertmanager-fallback-config.yaml:/mimir/alertmanager-fallback-config.yaml
      - {{ mimir_data_dir }}:/data
    ports:
      - "8080:8080"
      - "{{ mimir_port }}:9009"
    networks:
      {{ monitoring_network }}:
        ipv4_address: 192.11.9.8
networks:
 {{ monitoring_network }}:
    external: true

mimir config file

usage_stats:
  enabled: false
server:
  http_listen_port: 9009
  log_level: error

target: all
multitenancy_enabled: false
no_auth_tenant: "anonymous" #use when multitenancy is disabled.

blocks_storage:
  backend: s3
  s3:
    endpoint: {{ mimir_minio_entpoint }}
    bucket_name: {{ mimir_minio_common_bucket_name }}
    access_key_id: {{ mimir_minio_key_id }}
    secret_access_key: {{ mimir_minio_access_key }}
    insecure: true
  tsdb:
     dir: /data/tsdb
     retention_period: 24h0m0s #keep TSDB blocks in the ingesters
     ship_interval: 1m0s
  bucket_store:
      sync_dir: ./tsdb-sync/
#      sync_interval: 15m0s
#         metadata_cache:
#             backend: "memcached"  #####testen
#             memcached:
#                 addresses: "127.0.0.1"
        
compactor:
  data_dir: /tmp/mimir/compactor ## Directory to temporarily store blocks during compaction.
  sharding_ring:
    kvstore:
      store: memberlist
# cleanup_interval: 15m # How frequently compactor should run blocks cleanup and maintenance, as well as update the bucket index.

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1

ruler_storage:
 backend: s3
 s3:
   endpoint: {{ mimir_minio_entpoint }}
   bucket_name: {{ mimir_minio_bucket_name_ruler }}
   access_key_id: {{ mimir_minio_key_id }}
   secret_access_key: {{ mimir_minio_access_key }}
   insecure: true

store_gateway:
  sharding_ring:
    replication_factor: 1

query_scheduler:
    max_outstanding_requests_per_tenant: 1000

frontend:
    max_outstanding_per_tenant: 1000

limits:
  compactor_blocks_retention_period: 1y # Delete blocks containing samples older than the specified retention period. Blocks in the long-term storage will be kept forever, unless you configure the retention explicitly using:
  ingestion_rate: 250000
  max_label_names_per_series: 100
  max_global_series_per_user: 350000
  compactor_block_upload_enabled: true
  compactor_split_and_merge_shards: 1 # default 0
  compactor_split_groups: 1
  ingestion_tenant_shard_size: 1 # default 0

Our Purpose

Currently, we have a Prometheus instance that scrapes multiple targets (~300) it all works wonderfully until we set queries that have a larger time period. We would like to keep the metrics until 1 year, if we set the retention policy on Prometheus to 1 year it takes a very long time to display the metrics. That because of that, we want to test Grafana Mimir. According to the documentation, Grafana Mimir can display the long-term metrics much faster. We have installed Grana Mimir as a monolith mode as a docker container with one instance. We migrated the existing metrics from Prometheus to Mimir using the Mimirtool. After the migration, we gave Mimir enough time to process the data using the Compactor. In the process, we encountered the following issues.

Problem No. 1

The queries with Grafna Mimir take much longer than with Prometheus.

Problem No. 2

After migrating the data from Prometheus, the dashboards could not be displayed, in Grafana we get the following message:

internal: expanding series: the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The non-queried blocks are: 01GJRREK4PTRWP5DM1P2Y7W040 01GK5NDFS0C9MBYMBS75VVQBX5 01GHDC5STKXV0RNJCQPSJV7XMZ 01GJVB27MRF68N6X84M8JJV2FX 01GJYPCS133XC123D9Z0RSS200 01GK33G0MY9XYAAX0B9TFFS4Q0 01GK1GQNJX1ZDWB1VBDX34T7Z6 01GJG4HWJJ1Y89JRVM1TGQGYPD 01GK864NY463HWDDRS806QCF5S 01GK0ETHHNJBPJAFGGRSG49MAX 01GHYRBSN66E0MTMBEKRBB1CK3 01GK37MA0HCAF764ZAQ3ZA87RH 01GK6N0R5PK7BMZ5GFN5QR6KGQ 01GK8D4XZY2XV2TPVSRZWGTJ1X

Mimir container logs:

mimir | level=error ts=2022-12-02T12:33:38.368215612Z caller=retry.go:78 user=anonymous msg="error processing request" try=4 err="rpc error: code = Code(500) desc = \"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The non-queried blocks are: 01GJRREK4PTRWP5DM1P2Y7W040 01GK5NDFS0C9MBYMBS75VVQBX5 01GHDC5STKXV0RNJCQPSJV7XMZ 01GJVB27MRF68N6X84M8JJV2FX 01GJYPCS133XC123D9Z0RSS200 01GK33G0MY9XYAAX0B9TFFS4Q0 01GK1GQNJX1ZDWB1VBDX34T7Z6 01GJG4HWJJ1Y89JRVM1TGQGYPD 01GK864NY463HWDDRS806QCF5S 01GK0ETHHNJBPJAFGGRSG49MAX 01GHYRBSN66E0MTMBEKRBB1CK3 01GK37MA0HCAF764ZAQ3ZA87RH 01GK6N0R5PK7BMZ5GFN5QR6KGQ 01GK8D4XZY2XV2TPVSRZWGTJ1X\""

What exactly is going wrong here?

Stats

Queries	Range	Mimir	Prometheus
100-cpu_usage_idle{vm_svc="arti", host="vm01", cpu="cpu-total"}	24h	478 ms	184 ms
100-cpu_usage_idle{vm_svc="arti", host="vm01", cpu="cpu-total"}	6 months	372 ms	279 ms
sum(artifactory_usedSpaceInBytes)	24h	1.46 s	1.32 s
sum(artifactory_usedSpaceInBytes)	6 months	10.3 s	7.5 s

I look forward to hearing back from you! Thanks in advance! :)

[pracucci]

👋 Hi! Sorry for the late reply. This discussion slipped unnoticed.

Mimir is not magic. Under the hood it runs the same PromQL engine used by Prometheus (in order to guarantee 💯 query language compatibility). So, what's the Mimir strength? Mimir is a distributed system and can be scaled horizontally. You can distribute the workload across multiple CPU cores and machines, going beyond a single machine vertical scalability limit.

Where to start to squeeze out Mimir performance?

1. Make sure to adopt the production tips, in particular to configure memcached (results, index, chunks and metadata cache)
2. Make sure to run multiple Mimir replicas in a cluster. A 6 months query is split into daily query and each daily query can be processed concurrently (assuming the cluster has enough capacity, otherwise daily queries are enqueued)
3. To push it further (and assuming the cluster has enough capacity), consider enabling query sharding to further parallelise the query execution

[kpinarci]

Hey Marco, thanks for the feedback.

I have another question about load balancing. When one of the Mimir containers is unreachable the load balancing works, but when two containers are killed the GUI is reachable, but the metrics don't arrive. I get status code 502.
How many mimir containers must be present for the mimir to keep working?

In Grafana Explore, I get the following message:
internal: expanding series: too many unhealthy instances in the ring internal: expanding series: too many unhealthy instances in the ring

[pracucci]

Assuming you run Mimir with the default configuration:

When you deploy in monolithic mode, then Mimir supports 1 unhealthy instance. If you have more unhealthy instances, the service will be degraded. When you configure Mimir with zone-awareness replication enabled), then Mimir can support the failure of all instances in 1 zone.

When you deploy in microservices mode, it depends on the specific components. Many components are stateless (e.g. distributor, querier, ...). The most critical component is the ingester, which can support 1 unhealthy instance (single zone) or all instances in 1 zone (multi-zone). Store-gateway can support 2 unhealthy instances (single zone) or all instances in 2 zones (multi-zone).