Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twom database format and first implementation #5157

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

brong
Copy link
Member

@brong brong commented Dec 5, 2024

Throwing this open for wide reviews now! It's been rewritten a BUNCH of times, and optimised a fair bit - it's got a simpler internal structure and basically everything aligned with the right datastructures I think. It does MVCC reads nicely, can keep its transaction in foreach unless the callback writes, in which case it STILL does the right thing and re-locks and re-finds itself.

On disk format of records is:
image

lib/cyrusdb_twom.c Outdated Show resolved Hide resolved
@rjbs
Copy link
Collaborator

rjbs commented Dec 16, 2024

I've removed include-in-fm while tests are failing.

@brong brong force-pushed the twom branch 4 times, most recently from 7d3b303 to 5ffbf60 Compare January 5, 2025 11:37
@brong brong requested review from dilyanpalauzov, robmueller, rjbs, elliefm, wolfsage and rsto and removed request for dilyanpalauzov January 5, 2025 11:41
@brong brong force-pushed the twom branch 2 times, most recently from 981f5c7 to a0615b2 Compare January 5, 2025 12:14
@brong
Copy link
Member Author

brong commented Jan 5, 2025

Bah, CI failed because errno.h is required. I took it out because it's not needed in my machine and I was trying to trim back all the required headers

@brong brong requested a review from dilyanpalauzov January 5, 2025 12:33
Copy link
Contributor

@elliefm elliefm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't finished looking at this yet, but need to stop for lunch, so here's what I have so far.

imap/hammer_cyrusdb.c Outdated Show resolved Hide resolved
imap/hammer_cyrusdb.c Outdated Show resolved Hide resolved
lib/imapoptions Outdated Show resolved Hide resolved
lib/imapoptions Outdated Show resolved Hide resolved
lib/cyrusdb_twom.c Outdated Show resolved Hide resolved
lib/cyrusdb_twom.c Outdated Show resolved Hide resolved
@brong brong force-pushed the twom branch 5 times, most recently from 17f965c to 0e6615f Compare January 10, 2025 11:12
@brong
Copy link
Member Author

brong commented Jan 10, 2025

@brong I removed the include-in-fastmail label for this PR. The Rename test suite tests are failing.

Thanks! yeah, there were more issues but I believe it's all clean now!

@brong
Copy link
Member Author

brong commented Jan 10, 2025

OK! Major messing around.

  • I've added more comments
  • Standalone twom library is API made more consistent
  • It's actually passing tests (there were plenty of fun bugs)
  • Repack now happens in-object a bit more rather than creating a new DB.

@brong brong force-pushed the twom branch 2 times, most recently from 9d3e467 to 4592a4a Compare January 10, 2025 12:04
lib/imapoptions Outdated Show resolved Hide resolved
lib/imapoptions Show resolved Hide resolved
brong added 3 commits January 11, 2025 14:52
This is all awful and I'm going to ask Ken to add basecid in the next
index format change!
This contains the full implementation of the xxhash algorithm.  It's
only being used by twom database format, so it's being compiled straight
into that file.

This is an unmodified copy of xxhash.h from:

commit dd11140c2dc5d53a3c0a949d67af7f40f546878e

in the repository at [email protected]:Cyan4973/xxHash.git
@brong brong force-pushed the twom branch 5 times, most recently from e6cf33a to 1520f4e Compare January 12, 2025 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants