Limit cache size

[sashahilton00]

Issue by giannello
Friday Jul 29, 2016 at 19:50 GMT
Originally opened as plietar/librespot#105

---

For people running librespot on hardware with limited resources, it would be nice to have an option to limit the cache size.

[sashahilton00]

Comment by plietar
Friday Jul 29, 2016 at 20:04 GMT

---

This is definitely something I'd like to have, but requires quite a bit of work.
At the moment, caching is just a matter of placing the file in the cache directory, and trying to open it when loading.

This requires some bookkeeping to keep track of what files are in the cache, their size, along with an eviction algorithm.

These could be used as inspiration, although librespot doesn't need something as sophisticated :
https://wiki.mozilla.org/Necko/Cache
https://www.chromium.org/developers/design-documents/network-stack/disk-cache

[sashahilton00]

Comment by mfonville
Tuesday Nov 29, 2016 at 14:16 GMT

---

For the moment, users that face issues with it, they could make a simple cron-job that checks the atime of files tp clean-up.

[sashahilton00]

Comment by juriansluiman
Wednesday Feb 22, 2017 at 13:22 GMT

---

@mfonville how is this possible? If I run find /tmp/librespot/files/* -atime +2 -exec rm {} \; it crashes librespot. The only way I can get it to work is to stop the librespot service, clear up the files and start it again. If I do this, the music will stop when playing, so it doesn't seem to be the most ideal solution yet :)

[sashahilton00]

Comment by plietar
Wednesday Feb 22, 2017 at 14:33 GMT

---

@juriansluiman Deleting files while librespot is running should work.
Can you run librespot with RUST_BACKTRACE=1 and --verbose and post the log please ? What platform are you running on ?

[sashahilton00]

Comment by juriansluiman
Wednesday Feb 22, 2017 at 15:10 GMT

---

@plietar sorry, you get forget my comment. I was having another issue with librespot so it stopped working, but just tried again and it does continue to play. My bad!

[sashahilton00]

Comment by michaelherger
Tuesday May 02, 2017 at 04:34 GMT

---

I've modified librespot to disable the audio file-cache. See pull request #181.

[shawaj]

Any progress on this @sashahilton00 ?

[Sojiro84]

Wondering the same. I just discovered that librespot is taking up 41.8GB of data.

That is a lot for something that should just stream audio!

[debdrup]

For a simple algorithm, would it make sense to evict the oldest cached files, either by creation or modification date, when the cache grows beyond a certain size?
As long some code gets added to fix this issue, I'm not sure it needs to be optimal or perfect.

It's definitely better than having a library be responsible for taking up any available diskspace on a long enough timescale.

[sashahilton00]

Nope, no progress. We did discuss it a little in another issue, I can't remember which one exactly, will check when I get home. Ideally we would separate the cache logic out, as there are a few use cases/requirements, eg. Limiting cache by size and/or date, or disable the cache all together.

I can appreciate that in the meantime though it would be better to have something rather than nothing, so if someone does want to throw together a PR that implements a basic cache eviction function, we'll look at getting it merged. If no one wants to, I'll try and give it a go at a later date, but currently I have limited time, and there are a couple of other features that take priority imo, such as retrieving files from the CDN or session reconnection.

[ashthespy]

I have an old branch with a simple LRU based file cache implementation with a prescribed limit, can hunt things up if someone would like to work on it..

[woodruffw]

I'm completely new to this project, but I'm happy to take a stab if someone has a partial implementation.

[azrdev]

I have deployed the following systemd units:

# /etc/systemd/system/spotifyd-cache-clean.timer
[Unit]
Description=Daily clean audio file cache of spotifyd/librespot

[Timer]
OnCalendar=4:00
Persistent=true

[Install]
WantedBy=timers.target

# /etc/systemd/system/spotifyd-cache-clean.service
[Unit]
Description=Clean audio file cache of spotifyd/librespot

[Service]
User=spotifyd
ExecStart=/usr/bin/find /var/cache/spotifyd/files -type f -atime +30 -print -delete

Obviously depends on systemd and the filesystem correctly tracking access times,
then deletes all cached files not accessed for the last 30 days

[debdrup]

While that's a good temporary hack for some, it's not really a solution.

Firstly, because it isn't cross-platform (even if it were to be included with librespot, which exists where systemd does not) and secondly, because it's not a POSIX-compatible invocation of find.

It's also not okay for libraries or programs to take up arbitrary disk space without ways of cleaning up after themselves.

[azrdev]

indeed, but may save s/o looking for a temporary workaround the hassle to write it themselves :)

[roderickvd]

I'm thinking: "there must be a crate for this", right?

A quick search on crates.io show some promising leads, though they seem to evict based on time and not on capacity?

https://docs.rs/futures-cache/0.10.0/futures_cache/
https://crates.io/crates/cached-path

[Johannesd3]

I created a PR that could resolve this issue: #675

[Johannesd3]

#675 was merged and release in 0.2.0. There's an option --cache-size-limit now.