How to get uncommitted data size?

[anidotnet]

I am planning to use redb as a backend store for a database application where lots of read write operations will happen concurrently. Instead of committing each write, I am looking for a way to get the uncommitted data size. If that value reaches a threshold, I'll do a commit and open a new write transaction.

Is there any way I can achieve this functionality in it's current form? If not, do you have any plan to add this?

[cberner]

No, you will need to track that yourself at the moment. It is tracked internally though, so I think could be exposed

…

Sent from my phone

On Tue, Feb 11, 2025, 4:45 AM Anindya Chatterjee ***@***.***> wrote: I am planning to use redb as a backend store for a database application where lots of read write operations will happen concurrently. Instead of committing each write, I am looking for a way to get the uncommitted data size. If that value reaches a threshold, I'll do a commit and open a new write transaction. Is there any way I can achieve this functionality in it's current form? If not, do you have any plan to add this? — Reply to this email directly, view it on GitHub <#948>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGNXQARHFRW5PLIG5L5H432PHWHBAVCNFSM6AAAAABW46TDSOVHI2DSMVQWIX3LMV43ASLTON2WKOZSHA2DKMRZGQYDEMQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[anidotnet]

Any plan of exposing that in near future? It's a very crucial api for storage system.

[cberner]

Not at the moment. I think the best approach is for you to make a fork, add the calls you need, and once it's all working I'm happy to take a look and see if it make sense to upstream

[anidotnet]

I'll be happy to work on this. Could you please point me to where it is being tracked currently, as you mentioned earlier?

[cberner]

redb/src/tree_store/page_store/page_manager.rs

Line 86 in 51ca54c

allocated_since_commit: Mutex<HashSet<PageNumber>>,

[anidotnet]

I guess these is how I'll get the actual uncommitted data size in bytes right?

redb/src/tree_store/page_store/base.rs

Line 156 in 486f93b

pub(crate) fn page_size_bytes(&self, page_size: u32) -> u64 {

[cberner]

yep

[anidotnet]

I have raised the PR. Please review.

[cberner]

Have you integrated it into your database? I'd like to see how it performs before merging this feature. In particular, that's going to be somewhat expensive to call, so I'm curious whether that really works for your use case

[anidotnet]

Not yet. I'll do that in a couple of days and I'll let you know. In any case it will make my life easier. Currently for every modification (create, update, delete) I have to open a new transaction and commit it. For large data set it is taking ridiculous amount of time. Example, insertion of 10000 simple documents take 17 mins.