Replication Failure Scenerios #90
-
Is there a scenario where the data written to SQLite by the application won't be durable? I don't know enough of SQLite functioning to be sure but my mental model is: Process 1 (Application): Write to SQLite -> SQLite appends entry in WAL If host died between the monitor interval and there were pending writes, the changes would be lost. Is this accurate? Thanks for the great project! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
Read more of the code and found that the WAL is continuously uploaded to S3, faster than the monitor interval. So the interval of missing out on WAL is shorter. |
Beta Was this translation helpful? Give feedback.
-
Yes, that's an accurate mental model. Litestream performs asynchronous replication similar to PostgreSQL's streaming replication so there is a small window for data loss in the event of a catastrophic failure. I plan to add synchronous replication in the future as well but that can incur a performance and availability penalty so it would need to be enabled explicitly. As for intervals, there are two main ones in Litestream. There's a monitor interval that moves WAL pages from the primary DB's WAL file to a shadow WAL file that can be read by replicas. This interval defaults to |
Beta Was this translation helpful? Give feedback.
Yes, that's an accurate mental model. Litestream performs asynchronous replication similar to PostgreSQL's streaming replication so there is a small window for data loss in the event of a catastrophic failure. I plan to add synchronous replication in the future as well but that can incur a performance and availability penalty so it would need to be enabled explicitly.
As for intervals, there are two main ones in Litestream. There's a monitor interval that moves WAL pages from the primary DB's WAL file to a shadow WAL file that can be read by replicas. This interval defaults to
1s
and is mainly to limit resource usage on the machine. Then there's also an interval for the S3 replica that sp…