Block size vs data.parquet size

[simonsunday]

Can you please help me understand how the storage for tempo is working?
I'm getting around 150MB/s up to 200MB/s of traces.
Have block_retention set to 720h
Currently using around 120TB of s3 storage.
Compactor max_block_bytes is set to default 100GB.
When i'm looking on the s3 contents, the content of the block dir is more or less 100MB and there are so many blocks in the bucket that I cant even list them in reasonable time.
Is this expected? or is some other setting affecting compacted blocks size?

Just looked at the logs and one of them is saying:
level=info ts=2025-06-18T08:22:53.731612612Z caller=compactor.go:277 msg="wrote compacted block" version=vParquet4 tenantID=prd blockID=d08298c3-3d5e-461d-a549-18e87b3b3653 startTime="2025-06-18 07:22:41 +0000 UTC" endTime="2025-06-18 07:27:14 +0000 UTC" totalObjects=373444 size=451379070 compactionLevel=1 encoding=none totalRecords=373444 bloomShardCount=5 footerSize=84639 replicationFactor=0 dedicatedColumns=[]

430MB is still way less than 100GB

[joe-elliott]

Compactor max_block_bytes is set to default 100GB.

I would definitely recommend reducing this value. We used to create blocks this size with the v2 format, but parquet is far too costly to build blocks this large. Here are our settings:

compactor:
    compaction:
        compaction_window: 6m
        max_block_bytes: 4.84442112e+09
        max_compaction_objects: 3e+06

so many blocks in the bucket that I cant even list them in reasonable time.

how many? max(tempodb_blocklist_length) by (tenant) should give you a sense. Our largest clusters have 3M+ blocks across all tenants and 150K for one tenant.

150 MB/s * 3600s * 24h * 30d = 388800000 MB of retention

388800000 / 1000 / 1000 -> 388 TB? 120TB sounds like a deal to me :). Is my math wrong?

You are at a scale where Tempo can be difficult to operate. Feel free to ask Qs. Tempo 3.0 will include an RF1/queue based rearchitecture that should make your situation much better.

[yehaotian]

also need some guidance on the compactor config, what we have:

block_retention: 30d
other configs all take the default value (compacted_block_retention: 1h, compaction_window: 1h, max_compaction_objects: 6M, max_block_bytes: 100GB, ...)

max(tempodb_blocklist_length): ~3M
object storage: 13M object count
bucket size: 1.3PB

compactor:
compaction:
compaction_window: 6m
max_block_bytes: 4.84442112e+09
max_compaction_objects: 3e+06

will this work for the case?

[joe-elliott]

Your blocklist is enormous and is definitely causing you inflated memory and cpu resource usage. I think you've already identified the key settings to change. Your compaction_window is far too large given that number of blocks. I would even consider something smaller like 4 or 5m.

With a list that long you want your compactors to be as focused as possible on compacting level 0 to 1 blocks to keep it down. The settings you proposed will help. Watch a metric like sum(increase(tempodb_compaction_blocks_total{}[5m])) by (level) to have sense of things.

The RF1 rearchitecture will be very beneficial to you all.

[mdisibio]

Can you also check the ingester max_block_bytes setting? See comment below: #5301 (comment)

[mdisibio]

the content of the block dir is more or less 100MB

Can you check the ingester max_block_bytes setting? It defaults to 500MB, but this seems like it is configured for 100MB, maybe from an example or default somewhere else? Flushing 100MB blocks is too small for that volume, and it is putting too much pressure on polling and compaction to clean up. Flushing larger blocks from the ingester is the most effective way to reduce work since it is upstream of those components.