Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support very large files #48

Open
stephenh opened this issue Jan 26, 2020 · 1 comment
Open

Support very large files #48

stephenh opened this issue Jan 26, 2020 · 1 comment

Comments

@stephenh
Copy link
Owner

stephenh commented Jan 26, 2020

Files over ~500mb / ~1gb give mirror trouble b/c the RPC framework (grpc-java) we use for "really snappy bi-directional streaming" only supports in-memory / on-Java-heap byte[]s for its RPC messages, and doesn't support passing around zero-copy files / memory regions like netty's FileRegion.

So, for now, mirror just can't really send those sort of files unless/maybe you give it a huge heap.

We should probably detect the current size of heap, guess that we can do files ~half that size, and just ignore any file larger than that, which is basically #37.

@rleon
Copy link

rleon commented Jan 26, 2020

The best solution will be to change the transfer algorithm for such large files. For example, use rsync for delta-transfers.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants