Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Let the fanout update instead of rewriting it entirely #281

Open
pascutto opened this issue Mar 17, 2021 · 1 comment
Open

Let the fanout update instead of rewriting it entirely #281

pascutto opened this issue Mar 17, 2021 · 1 comment

Comments

@pascutto
Copy link
Contributor

The fanout is currently built during the merge operation, by registering all addition in its internal array, then it is both written in the header of the file, and kept in memory for future find to use.

Whenever a merge occurs, the old fannout is dropped, and a new one is created from scratch. Although quite light, this introduces a lot of unnecessary computations (also keep in mind that almost all changes by calls to update are overwritten by the following call anyway, which could also be improved).

Since merges only introduce shifts in the data file, the Fan module should be able to update an existing fanout instead of rewriting it from scratch every time. Special care should be taken when the fanout requires resizing (see the size computed in Fan.v).

@pascutto
Copy link
Contributor Author

Reporting an offline discussion with @craigfe:
Given the current datastructure (a simple array) used for the fanout, it is unlikely that we can create a fanout from an existing one, and just update the log entries, especially when a resizing occurs (and accuracy is lost and can cumulate in time, resulting in very unhelpful states).
It appears that we will still have to register data entries, but we don't have to register them all. The bucket size used in the fanout is known, which means we could only register the last entry of each bucket in the merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant