How does tempo uses parquet on object storage now?

[scalalang2]

Hello,
I have a question about how Tempo works. One of the things that impressed to me is that it utilizes object storage instead of traditional backend database such as ElasticSearch to provide distributed tracing at a lower cost.

I'm really curious about how this works, especially, how it uses Parquet with object storage.

Q1. Does Tempo query trace data directly from the object storage?
I assume that It must download parquet file from S3 to local storage.
As far as I know, It seems to need to download whole file from parquet to read data.

Q2. If tempo uses local storage. Do I need to allocate large amount of storage in production?

Q3. Does Compactor remove old locally downloaded files periodically?

[scalalang2]

Is there a document that describes whole Tempo architecture?
I wanna understand this project more deeply.

[joe-elliott]

Q1. Does Tempo query trace data directly from the object storage?

Tempo only pulls the columns it needs when executing a query from object storage. It does not download entire blocks locally when querying. It also has to pull the Parquet footer for each block so it can know where each column exists.

Q2. If tempo uses local storage. Do I need to allocate large amount of storage in production?

Only the ingesters require local storage. Upon receiving data they write it to a set of local parquet files that they then flush to the backend.

Q3. Does Compactor remove old locally downloaded files periodically?

Compactors take multiple input blocks from object storage and combine them together to create one output block. This serves 2 purposes:

• deduplicate traces in an RF=3 environment
• reduce the blocklist to reduce latency on trace by id queries
  Compactors are also responsible for removing blocks from object storage when they fall out of retention. The behavior of compactors is currently quite tuned to the old format. The team has not had a chance to revisit the strategy here and there is likely improvements to make.

Is there a document that describes whole Tempo architecture?

These are mostly up to date. They won't answer all of your questions but they cover the high level: https://grafana.com/docs/tempo/latest/operations/

I wanna understand this project more deeply.

Awesome :)! Please keep asking questions. I'm happy to answer.

[scalalang2]

Thanks :) It helped me a lot, i'll keep moving to understand it.