Skip to content

Conversation

@igor-sokolov
Copy link

@igor-sokolov igor-sokolov commented Jan 7, 2026

hi @lemire
nice to e-meet you. First of all I would like to say thank you for such great algorithms and implementation. This is literelly a treasure for my currrent work!

I want to transfer the XorBinaryFuse16 over the wires. Therefore I want to wrap the serialized filter data with an envolpe to support different underlying filters / algorithms (e.g. use RoaringBitmap serialized data if I really need in some edge cases exact filtering). So to avoid additional memory copying I thought do the logic around the given ByteBuffer. So the idea is the following

  1. the client first calls org.fastfilter.Filter#getSerializedSize and allocates a ByteBuffer of enough size, then it writes the header
  2. then the client calls org.fastfilter.Filter#serialize(ByteBuffer buffer) to write the serialized filter
  3. on the other side of wires the client reads the header and then calls org.fastfilter.xor.XorBinaryFuse16#deserialize(ByteBuffer buffer) to create the filter.

If this looks good to you I can expose the same approach to other filters implemented in the library.
Looking forward to hearing a feedback from you!

Copy link
Member

@lemire lemire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR makes sense but you are making unrelated changes like making a class final, commenting on the choice of the random generator.

Fixing our typos is fine, but please minimize changes.

@lemire
Copy link
Member

lemire commented Jan 8, 2026

This is nice and invited.

@igor-sokolov
Copy link
Author

igor-sokolov commented Jan 8, 2026

hi @lemire

thank you your quick response!

Fixing our typos is fine, but please minimize changes.

done. please take a look.

If this looks good to you I can expose the same approach to other filters implemented in the library.

do you want me to extend the serialization/deserialization to all filters? I can do that. I wasn't just sure if you have concerns about the taken approach for the serialization (there could be other options and preferences starting from the hopefully deprecated some day java vanilla serialization, serialization through Input/OutputStream and others). I have choosed to work with ByteBuffers because that opens up for me a chance to read from the direct buffers without the need to copy to heap.

@lemire
Copy link
Member

lemire commented Jan 10, 2026

Looks good !!! Could you extend it to Xor8, X16, XorBinaryFuse8, XorBinaryFuse16, XorBinaryFuse32?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants