Some RocksDB configuration #38
Conversation
| bopts.set_cache_index_and_filter_blocks(true); | ||
| bopts.set_pin_l0_filter_and_index_blocks_in_cache(true); | ||
| bopts.set_bloom_filter(10.0, false); | ||
| bopts.set_checksum_type(rocksdb::ChecksumType::XXH3); |
There was a problem hiding this comment.
I think the default is CRC32; XXH3 is much faster. Considering constellation is probably IO-bound (on the RasPi), most block reads come from the OS page cache or disk directly, so each block read will incur a checksum calculation.
| bopts.set_checksum_type(rocksdb::ChecksumType::XXH3); | ||
| opts.set_block_based_table_factory(&bopts); | ||
|
|
||
| opts.set_min_level_to_compress(1); |
There was a problem hiding this comment.
Constellation has a very steady input of new data, so we know L0 is short-lived. L0 tends to store <= 256 MB data, so not much space savings here, so disabling compression reduces CPU spent when reading from L0.
Could be expanded to L1 maybe.
| bopts.set_pin_top_level_index_and_filter(true); | ||
| bopts.set_index_type(rocksdb::BlockBasedIndexType::TwoLevelIndexSearch); | ||
| bopts.set_partition_filters(true); |
There was a problem hiding this comment.
I hope my intuition was right and TARGET_LINKERS_CF is the largest CF. The idea is to reduce cache movement when paging in and out the index/filter blocks.
| ); | ||
|
|
||
| let mut bopts = rocksdb::BlockBasedOptions::default(); | ||
| // bopts.set_block_size(/* 16 KiB */ 16 * 1_024); |
There was a problem hiding this comment.
Increasing block size may decrease space usage, but it really depends on the data that is stored in each CF. Prefix truncation already takes care of shortening keys to some degree, so if the values are not that compressible, the space savings may be subpar.
!!! I haven't benchmarked this !!!
This is mostly a configuration based on intuition because I don't know what data is stored exactly in each CF and what access pattern it has.