Memory efficient lookup data structure #266
akiradeveloper
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I think the memory efficiency is where dm-writeboost should be improved.
dm-writeboost (and likely other caching technologies too) needs lookup data structure to find a cache block from a block id in the backing device.
In dm-writeboost, I use a naive hash table but this isn't memory efficient. As a result, it consumes about 0.75% RAM against cache device size.
Why I chose this data structure? At the beginning, dm-writeboost, as it name implies, was implemented as a write back cache. It was hard to imagine to have a huge write back cache device and my experiment showed a small cache device (like 256GB) is enough to boost the performance of the storage: The point is the cache device needs to be larger enough than the DRAM. In this case, it only consumes only 2GB memory and it is quite reasonable.
I didn't think read caching will be a good technology for a block device because we can still cache data on RAM, which we could thought to increase its capacity, even 10 years ago. In fact, RAM's decreased its bit cost.
Later, I implemented read-caching to acquire users. I think it was a good decision. After introducing read-caching, I found some users tried to use gigantic cache device like 2TB. Well, this consumes about 20GB RAM which is too much even today where the RAM price is cheaper than before.
I have a plan to improve lookup data structure to be more memory efficient.
Beta Was this translation helpful? Give feedback.
All reactions