Parameter to change the period of mimir sending data to object storage

[chesha1]

The default period of mimir sending data to object storage is 2h, but when it comes to 2h, the data will not be immediately sent to object storage.

The ingester log is below, from mimir-ingester-zone-a-0

ts=2024-02-28T09:14:27.983424388Z caller=head.go:723 level=info user=anonymous msg="On-disk memory mappable chunks replay completed" duration=1.974µs
ts=2024-02-28T09:14:27.983431752Z caller=head.go:731 level=info user=anonymous msg="Replaying WAL, this may take a while"
ts=2024-02-28T09:14:27.983840497Z caller=head.go:802 level=info user=anonymous msg="WAL segment loaded" segment=0 maxSegment=0
ts=2024-02-28T09:14:27.983881865Z caller=head.go:839 level=info user=anonymous msg="WAL replay completed" checkpoint_replay_duration=14.878µs wal_replay_duration=426.739µs wbl_replay_duration=191ns total_replay_duration=459.912µs
ts=2024-02-28T09:14:27.985462083Z caller=db.go:1885 level=info user=anonymous msg="Compactions disabled"
ts=2024-02-28T09:14:27.98548658Z caller=ingester.go:2260 level=info user=anonymous msg="Running compaction after WAL replay"
ts=2024-02-28T09:19:07.829991023Z caller=reporter.go:132 level=info msg="usage stats reporter initialized" cluster_id=391f1ce6-a347-4853-bb31-b7a93a250965
ts=2024-02-28T12:24:20.512765521Z caller=compact.go:716 level=info user=anonymous msg="write block" mint=1709111661949 maxt=1709114400000 ulid=01HQQTRFBTBCCCZPQWTE867AGC duration=1.446436827s
ts=2024-02-28T12:24:20.574093436Z caller=head.go:1340 level=info user=anonymous msg="Head GC completed" caller=truncateMemory duration=60.312198ms
ts=2024-02-28T12:24:20.574193156Z caller=checkpoint.go:100 level=info user=anonymous msg="Creating checkpoint" from_segment=0 to_segment=6 mint=1709114400000
ts=2024-02-28T12:24:23.205461152Z caller=head.go:1308 level=info user=anonymous msg="WAL checkpoint complete" first=0 last=6 duration=2.631339993s
ts=2024-02-28T12:24:37.062913708Z caller=shipper.go:158 level=info user=anonymous msg="uploading new block to long-term storage" block=01HQQTRFBTBCCCZPQWTE867AGC
ts=2024-02-28T12:24:37.907635996Z caller=shipper.go:166 level=info user=anonymous msg="finished uploading new block to long-term storage" block=01HQQTRFBTBCCCZPQWTE867AGC
ts=2024-02-28T13:09:20.918502739Z caller=compact.go:716 level=info user=anonymous msg="write block" mint=1709114400155 maxt=1709121600000 ulid=01HQQXAW2T7EZH92QNJGQSBKEC duration=1.852385587s
ts=2024-02-28T13:09:20.968358142Z caller=head.go:1340 level=info user=anonymous msg="Head GC completed" caller=truncateMemory duration=48.94704ms
ts=2024-02-28T13:09:20.978143524Z caller=checkpoint.go:100 level=info user=anonymous msg="Creating checkpoint" from_segment=7 to_segment=10 mint=1709121600000
ts=2024-02-28T13:09:23.185343802Z caller=head.go:1308 level=info user=anonymous msg="WAL checkpoint complete" first=7 last=10 duration=2.207282143s
ts=2024-02-28T13:09:23.63590634Z caller=shipper.go:158 level=info user=anonymous msg="uploading new block to long-term storage" block=01HQQXAW2T7EZH92QNJGQSBKEC
ts=2024-02-28T13:09:24.981416056Z caller=shipper.go:166 level=info user=anonymous msg="finished uploading new block to long-term storage" block=01HQQXAW2T7EZH92QNJGQSBKEC

It seems that period of every uploads is far more than 2 hours.

The old version of mimir has the parameter blocks-storage.tsdb.block-ranges-period, but now it disappears in the document and when I use the parameter, mimir cannnot correctly collect data from prometheus.

I would like to change the sending data frequency in mimir latest version, what is the solution of that?

Advice from anyone is appreciated.

[aknuds1]

Moving to discussions, as there's currently no evidence of a bug.

[pstibrany]

but when it comes to 2h, the data will not be immediately sent to object storage.

It seems that period of every uploads is far more than 2 hours.

Very first block for tenant is created 3h after tenant starts writing data to ingester. Subsequent block creations will happen on 2h boundaries (in UTC time it will be at 01:00, 03:00, 05:00, etc.).

So if tenant starts writing data to ingester at 01:35 and keeps writing more data, first block is created at 04:35 (this block will contain data between 01:35 and 02:00), then next block is created at 05:00 (with data 02:00 -- 04:00), then at 07:00 (with data 04:00 -- 06:00), and so on.

In upcoming not-yet-released Mimir version we are improving that so that in above situation very first block would be created at 03:00 already. (#7372)

[chesha1]

If I would like to increase the frequency to write blocks and upload to object storage, are there any parameters to control them? blocks-storage.tsdb.block-ranges-period is no longer valid.

By the way, if blocks are uploaded, will they be automatically deleted in local storage? If there are any parameters to control the delete frequency?

[dimitarvdimitrov]

If I would like to increase the frequency to write blocks and upload to object storage, are there any parameters to control them?

you can take a look at this answer #4696 (reply in thread)

By the way, if blocks are uploaded, will they be automatically deleted in local storage? If there are any parameters to control the delete frequency?

they will be cleaned up according to -blocks-storage.tsdb.retention-period