Backing Up Many DBs to 2 or More Locations

[alteredorange]

Awesome project! I was wondering how performant it might be with thousands of smaller sqlite dbs, with somewhat infrequent updates (i.e. updates happen a few times a day per db at most, but in total there would be a few thousand updates per day). Would the litestream background process take up a ton of resources? How well does it scale?

Also, does it double the load if you go from one backup location to 2? Thanks for the help!

[benbjohnson]

Hi @alteredorange! Good question. Right now, Litestream does polling on the underlying WAL file, however, that's going to be changed in v0.4.0 to use something like fsnotify to only check the WAL file when it changes. In addition to using fewer resources, it will allow for low-latency read replicas of SQLite databases (#8).

As for load, Litestream doesn't incur much load generally but you'll probably see an increase in CPU for LZ4 compression. Unfortunately that can't be shared as two replicas may be at different positions in the WAL.

[spiffytech]

Would thousands of DBs affect how much Litestream hits the S3 API?

[benbjohnson]

The number of databases doesn't affect S3 usage so much as the number of writes. If you have a thousand databases but only 10 of them are getting writes in a given second then you'll only see 10 writes to S3.