Block cutoff behaviour on Out of Order ingestion.

[manohar-koukuntla]

I would like to understand how block cutoff period works on enabling ooo ingestion.

What happens to OOO samples, that are ingested after block cut off period.
For example: If the block cut off was done at 02:00, what happens to the OOO samples that are ingested at 02:03 which has samples with timestamp 01:58 are earlier.

[jesusvazquez]

Hello @manohar-koukuntla

This doc may help answer this and some other doubts you may have https://docs.google.com/document/d/1Kppm7qL9C-BJB1j6yb6-9ObG3AbdZnFUBYPNNWwDBYM/edit#heading=h.unv3m5m27vuc

What happens to OOO samples, that are ingested after block cut off period.

The TSDB in-memory samples are typically in a single Head when OOO is disabled but when OOO is enabled there are two Heads. The in order one accepting new samples and the out of order one accepting any out of order samples up to the specified window.

When compaction happens, both heads and their respective memory mapped chunks are read and merged into blocks. The in order head typically generates a single 2hr block while the out of order head may generate as many 2 hour blocks as it needs depending on the samples it ingested.

For example if a compaction from 00:00-02:00 happens and during this period we had received OOO samples at 20:03, 22:03, 00:03, 01:03, the compaction will generate:

• 1 in-order block from 00:00-02:00
• 3 out of order blocks
  • 20:00-22:00 containing 1 sample at 20:03
  • 22:00-00:00 containing 1 sample at 22:03
  • 00:00-02:00 containing 2 samples at 00:03 and 01:03

So just remember that the ingesters will generate as many 2hr blocks as needed.

Then these blocks will be compacted by the compactor into bigger blocks. The reason why ingesters are not doing this is because we dont have to want the extra load there. Ingesters are typically very busy writing and reading so we don't want them to be compacting or optimizing blocks.

That said, there is room for improvement because in cases where we're talking about a few samples maybe we could just merge them. We have just not optimized this case yet.