Mimirtool backfill fails with "blocktime(s) greater than the present"

[bjcubsfan]

We run Grafana Mimir as a monolith with local storage. We want to able to backfill with mimirtool to have a complete data set for our analysts. I picked a time well in the past and generated the TSDB blocks for some missing data in our set. I adjusted the mimir configuration to allow backfill as described in the documentation, but the backfill fails saying that "blocktime(s) greater than the present":

$ sudo -u mimir /usr/local/bin/mimirtool backfill --log.level debug --address=http://localhost:9009/cortex/ --id=anonymous /vagrant/data/mimir-tsdb-backfill/01HPEX1YYWB8JH4GTB3VPWFC75

INFO[0000] log level set to debug
INFO[0000] Backfilling                                   blocks=/vagrant/data/mimir-tsdb-backfill/01HPEX1YYWB8JH4GTB3VPWFC75 user=anonymous
DEBU[0000] New Mimir client created                      address="http://localhost:9009/cortex/" id=anonymous
INFO[0000] making request to start block upload          block=01HPEX1YYWB8JH4GTB3VPWFC75 file=meta.json path=/vagrant/data/mimir-tsdb-backfill/01HPEX1YYWB8JH4GTB3VPWFC75
DEBU[0000] sending request to Grafana Mimir API          method=POST url="http://localhost:9009/cortex/api/v1/upload/block/01HPEX1YYWB8JH4GTB3VPWFC75/start"
DEBU[0000] checking response                             status="400 Bad Request"
ERRO[0000] response                                      body="block time(s) greater than the present: minTime=1704915591000000, maxTime=1704915597000001\n" status="400 Bad Request"
ERRO[0000] failed uploading block                        error="request to start block upload failed: POST request 
        to http://localhost:9009/cortex/api/v1/upload/block/01HPEX1YYWB8JH4GTB3VPWFC75/start failed: 
        server returned HTTP status: 
        400 Bad Request, body: \"block time(s) greater than the present: 
        minTime=1704915591000000, maxTime=1704915597000001\\n\"
        " path=/vagrant/data/mimir-tsdb-backfill/01HPEX1YYWB8JH4GTB3VPWFC75
INFO[0000] finished uploading blocks                     already_exists=0 failed=1 succeeded=0
mimirtool: error: blocks failed to upload 1 block(s), try --help

$ date +%s
1709566773

This is not the case as shown by the date command which starts with 1709. The backfill min and max are both in the 1704 range.

Is there something I am missing or not understanding here?

[56quarters]

Hello!

The min/max times listed appear to be microseconds (1704915591000000). The min/max times for TSDB blocks are supposed stored in milliseconds (1704915591000). How were these blocks generated?

[bjcubsfan]

Ah, that's probably what I'm missing! I bet my code to generate the openmetrics accounts for that, but then promtool accounts for it again.

I will try again with better timestamps and see if that fixes the issue.

[bjcubsfan]

Thanks for the help! That fixed the issue.

[pstibrany]

We have seen this problem multiple times. Hopefully including converted timestamps in the error message will make this easier in the future. (PR)