Very large uploads fail

[richard-to]

If someone tries to upload a very large file, say 400MB, then the uploader will fail during proto serialization. This is because the file will exceed the maximum array length in Javascript which is 2^32 - 1 (4,294,967,295).

Currently best alternative is to use a Fast API endpoint to handle the upload and probably a custom web component to handle the upload to the Fast API endpoint.

In terms of fixes for this, could it be possible to implement some kind of chunked upload to handler larger file sizes? Or could there be a way to avoid using protos in this case? Or could there be another way to workaround the maximum array length issue here?

[wwwillchen]

I think avoiding protos and having a separate route handler would make sense (to avoid the overhead & limitations of protos), but tbh I find this to be a somewhat niche use case.

for example, could they instead have the user upload directly to a file storage service like GCS: https://cloud.google.com/storage/docs/xml-api/post-object-forms?

once you go down the route of large file uploads, then it opens the door to things like resumable uploads, etc.

I think the current suggestion of using FastAPI (or whatever framework) w/ web component SGTM. For example, if a user is uploading something that large, they will probably want some kind of progress indicator (e.g. 30% uploaded).