External sqlite

[MalteMagnussen]

As far as I understand, meta-data about the files is stored in a sqlite in data/gokapi.sqlite.

That is meta-data such as:

• How many downloads allowed
• How long to store the files
• Connection between download url and S3 file

If Gokapi is deployed in Kubernetes, it loses all this data whenever the Pod is restarted.
Pods are considered to be relatively ephemeral.

So whenever the Pod is gone, any files uploaded via that Pod are also "gone". They're no longer tracked by Gokapi.
We created a S3 lifecycle policy to ensure that files are deleted after 30 days no matter what.

But if the Pod is restarted, then the link also "dies" to the files, and you get a page like this:

So based on all these assumptions, I assume that all this could be fixed by allowing the app to point to an external sqlite, instead of the internal auto-created one.

To sum up, we would like these things to persist across restarts, as it would generate issues for customers if we put files in gokapi, that they then can't access an hour later, because we lose persistence, if the pod restarts.

[Forceu]

The database file has to be stored in a persistent volume, you can refer to the example in the documentation: https://gokapi.readthedocs.io/en/stable/setup.html#id3

Would that work in your case?

[MalteMagnussen]

Sadly we do not have the ability to use persistent volumes in our Kubernetes cluster.

We have had a Mozilla/Send deployment (more like timvisee/send now).

"Send" had the option of saving the "meta-data" in an external redis (sqlite in gokapis case).

Just a suggestion / idea for a new feature.

Thanks again for making Gokapi.

[MalteMagnussen]

Been discussing this with the stakeholders.
It seems quite important to them that we don't lose the state too often.
Would be a hassle if the links die randomly, because the pod is restarted by kubernetes.

kubernetes makes everything much easier, right? anyway....

Options we have talked about:

• [Can be done entirely on our side]: Create a sidecar container running "Litestream": Backup the sqlite to S3
  • Create an ephemeral volume (we can make tiny non-persistant volumes) and mount it into ./data/ and similar in litestream container.
  • https://litestream.io/guides/kubernetes/
  • https://litestream.io/getting-started/
• [Has to be done in the codebase]: Change the code to allow connection to any SQL compatible backend.
  • We have a postgresql backend we could use.
  • Default to the current local sqlite behavior.
  • The code seems to be using the "generic golang database drivers": https://pkg.go.dev/database/sql so aside from a PRAGMA call here or there, it should be able to be interchangable?

---

Update: Working on the Litestream solution atm.

Update 2: Apparently our S3 implementation does not respond with a specific header that Litestream expects. And so it just crashes: benbjohnson/litestream#589

[Forceu]

I was considering support for Redis anyways, I might implement it next week.

[MalteMagnussen]

@Forceu - I managed to get the Litestream working.

1. Added it as a submodule
2. Build the image myself
3. Add it to the deployment

Now the state is persistent, despite the pod being ephemeral.

Docs used:

• https://litestream.io/guides/s3/
• https://litestream.io/guides/kubernetes/

And then the submodule + building it myself thing, to ensure I have the latest patches, which contained necessary fixes.