Query returns too many (duplicated) data points

[ThommyH]

Describe the bug

When querying mimir over grafana we noticed that we get more data points (metrics) than the scrape interval suggests. We have a scrape interval of lets say 15m but we get every 15min 20 metrics back each 15s apart (the default query min step). If we query with a larger min step, we naturally get less data points back. If we query with a shorter interval, we get additional datapoints back.
When we drill in and set the query time to be after the first metric, we still get metrics back, which leads me to believe mimir "fills the gaps" for us and interpolates metric timestamps.

Is this a known bug or feature and can we somehow disable it, as it increases the network load and makes some dashboards barely usable when there is a lot of data. And changing all intervals in all dashboards feels a bit excessive.

To Reproduce

Steps to reproduce the behavior:

Query mimir with any query with an interval shorter than the scrape interval.

Expected behavior

We expect only one metric (or maybe one for each mimir replication factor) to be returned instead of one datapoint per min step.

Environment

Kubernetes, helm chart version 4.2.0 running mimir 2.6.0 with 21 ingestors and grafana chart 6.52.1 running version 9.4.3

Additional Context

[pstibrany]

Hello. This is how PromQL evaluation for range queries works -- it returns data for every defined step, while looking back max 5 minutes for the sample. It is explained very well for example in this blog post: https://promlabs.com/blog/2020/06/18/the-anatomy-of-a-promql-query/#how-does-time-come-in-range-and-instant-queries

To get raw samples instead, you can use "instant" query (select "Instant" query type in Grafana) like mongodb_dbstats_ok{...}[1h] -- then you will get raw samples from 1h window for matching series.

Alternatively you can use "remote read" endpoint to ask Mimir for data. Grafana however uses "query" and "query_range" endpoints and can't use "remote read".

[ThommyH]

thank you for the clarification and prompt response. We kind of expected something like that but were clearly missing the right key words for finding that information.