Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utilize streams #98

Open
Gerrit91 opened this issue Oct 8, 2024 · 0 comments
Open

Utilize streams #98

Gerrit91 opened this issue Oct 8, 2024 · 0 comments

Comments

@Gerrit91
Copy link
Contributor

Gerrit91 commented Oct 8, 2024

In the current implementation we pass around files: write a database backup to a directory, tar the backup directory contents, encrypt the tar and then upload. This requires a lot of disk space because we write everything to dedicated files.

To optimize this and also make the code a bit easier to understand (file path handling is not particularly beautiful), we could try utilizing streams, i.e. the io.Reader and io.Writer interface.

Our archiver library seems to offer a Compressor and Decompressor interface, which allows passing streams. If this does not work out for us, we could also try to compress using go std lib (+ pierrec/lz4 if we want to maintain lz4 compression support).

Then we could pipe the compressed backup into the encryption layer. AES encryption is also be possible with streams.

The backup providers also have no issues to handle streams for upload and download.

Only the databases cannot be enabled to pass streams as they depend on their own database tooling and os/exec. So, for databases the file-path interface has to be maintained, I assume. Please correct me if you think I am wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant