Caching

[christianreiss]

Hey folks,

Two suggestions, one topic: Caching.
I am using my CloudDrive (Amazon) as a store-backend for my linxu based sat receiver. The box records shows and rsyncs them in the night to the clouddrive. This works like a charm. During normal "Playback" the box connects via cifs to a server that has the clouddrive mounted with acd_cli's help.

The little box reads in 5 files per show (1 big data file, 4 metadata and information) so tons of small files. It would be awesome if...

• The entirety of the clouddrives files (filenames, paths) could be locally caches in acd_cli.
• An option to cache all files (data) that are smaller than X to a local directory.

I tried using a rsync --exclude $bigfiles and doing an unionfs (which worked great) but deleting a show only deletes the files in the local cache dir.

Think you could whack such features in?
-Chris.

[bgemmill]

@chrisidefix On your topic of caching, that's actually what the nodes.db in your ~/.cache folder contains after a sync. Are you seeing performance related issues querying it? I know there are a few PRs in that aim to improve that.

On the topic of data caching, that came up in discussion in my PR #374 , and it looks like the most straightforward way to do it would be to add a new table in nodes.db mapping node_id -> binary blob, and on a read with matching node_id, open up a WriteBuffer marked as non-dirty with the contents so it didn't upload again on close.

I'd be happy to look at patches there; I think the trick would be to cull this in some LRU fashion since lots of small files could overwhelm the local drive too.