Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

event cache: be more conservative when adding duplicated events to a linked chunk #4563

Open
bnjbvr opened this issue Jan 21, 2025 · 0 comments

Comments

@bnjbvr
Copy link
Member

bnjbvr commented Jan 21, 2025

When we're adding new events into the linked chunk, we try to minimize storing of back-pagination prev-batch tokens:

  1. when pushing to the tail, if all events have been duplicated, independently of ordering, we won't deduplicate/remove them, and we'll not store the prev-batch token.
  2. when inserting a chunk of events from a recent back-pagination, and they've all been deduplicated, we won't deduplicate/remove them, and we'll not store the prev-batch token.

Unfortunately, this is not as simple as that: we should also check the orderings of the events that have been deduplicated, in particular that they're indeed ordered the same way they're ordered in the linked chunk representation. If that's not the case, then we should keep the prev-batch token and keep on iterating until we get the same events in the same ordering.

Related to #3280.

cc @Hywan

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant