Use hexane as the in memory representation of coln-store - #112
Open
incipit0 wants to merge 14 commits into
Open
Conversation
Coln tables tend to store lots of ids in memory. Storing all of them as plain (hash,counter) will blow up memory easily because each of them require 36 bytes. Instead use dictionary encoding for these ids so that we can represent each id with 8 bytes of memory, a 4x reduction in memory footprint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
We introduce three optimisations to optimise the in memory consumption
for large tables with lots of ids and hashes:
1. stream the importing from csv to coln-store in batch. This avoids a
big increase in peak memory consumption. This does not change the
stable memory consumption, but avoids the potential of OOM kicking
in when importing a large file.
2. cmt.resolved_ops now returns an iterator rather than a Vec. Again
this is to reduce the peak memory consumption because creating a
Vec that holds a copy of all the ids with hash in them is wasteful.
3. Drop Commit.pending field after we apply it, trading compute for
space. This is because the data would be accessed in tables and it
is wasteful to keep another copy in each commit, essentially
doubling the memory usage.
Hashcons data structure is disabled, and will be optimised next.
Use hexane as the in memory representation for data, and indexes. This helps reduce the memory footprint, especially for hashcons data structures. So we manage to go from 12 GB to about 6 GB. Also make the `id_packer` a store wide construction, which is shared by the index data structures (`HexaneIndex`) and the `Rowing` hashcons data structure.
Introduce ColumnMap as well.
A basic rollback support for tables and id_packer. Hashcons is broken right now and will be rewritten. Also add PackedOp so that tables no longer need to take id_packer.
It is not txn specific, so move to a more generic location.
Use egglog style canonicalisation: i.e. do fixpoint iteration until we get no more ids to update. Also add rollback support for rowing, for now this is doing some cloning. But we expect most of the data structures in rowing small. So should not be a big problem. This is a major rewrite of many of the internals of coln-store, to hopefully get better memory footprint. But for this commit, get to work first.
During rebuilding, add the option to do a full table scan rather than using index. Index scan is disabled for now until rebuilding becomes slow enough we need to do an index scan. But even then maybe we can just have a hexane index on each column than a HashMap<rowid, Vec<rowid>>. The full table scan right now reduces memory from 1.5 G to 350 MB. Also make finalize_handles canonicalise ids.
This saves some cloning around and reduces the peak memory usage by about 150 MB. Also reduces runtime by about 200s.
Just like duckdb -c
incipit0
force-pushed
the
coln-store/exp
branch
from
August 4, 2026 10:29
d3aabf3 to
ba485f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a major rewrite of the in memory representation of data. We now use hexane columns to represent data, which gives us nice compression properties, saving us lots of memory. This also provides us with a nice indexing data structure, like a btree index, but instead here we are using a sorted hexane column as indexes.
This PR also introduces a complete rewrite of the previous rowing data structures, which is used when structurally identical terms come up and we need to identify. I tried to follow egglog's style of canonicalisation. It also introduces memory savings by only adding ids that require canonicalisation to rowing.
Finally this PR introduces rollback functionalities to the store, which means we can rollback changes if a transaction aborts, without copying the entire store, which was a serious scalability issue.