Context canceled on S3 upload when upload speed is throttled #1172
Labels
Comments
|
Can you give me more details to reproduce this? Which browser did you use and how did you throttle it? What tus-js-client configuration did you use? What are the full logs from tusd? |
|
Sure, I currently use Arc which is Chromium based. If you go to the network setting in the developer tools you have the possibility to throttle the bandwidth. Sorry, it's german. But I think you get the position of the dropdown. 
I have selected slow 4G and I upload 43 files with a total of 271,4 MB. My tusd client configuration is: Here are some more logs: The browser shows the following logs: I use caddy in front of tusd with the following settings: |
|
Thanks for the additional details. I will look into this in two weeks. Does this also happen if you upload directly to tusd without using Caddy as a reverse proxy? |
|
It needs a bit tweaking on my side to test that only with tusd, because my hook endpoints use authentication and that works only with my local domain. But I discovered another interesting case. Maybe this is expected, but I think this could lead to disk space issues on the tusd instance. During a throttled upload I have looked to my upload bucket and I have seen that only the info file gets stored and after looking at the docker metrics I have seen that tusd writes the data to its own storage. With a not throttled upload it sends it directly to minio over the network. In the first screenshot you can see that I first tested the upload with some small files and later then with a larger video (ca. 145mb). This is where I discovered the issue that there is no file in minio during the upload. Then I restarted tusd, to clear the stats. I disabled throttling and uploaded the video. You can see that there was no write to the disk but the networking IO shows that it has sent the file to minio. It's only a blue line, but if I hover over it, then you can see that it also has sent the data. 
Hovering over the data In the second screenshot you can see that with enabled throttling it writes the data to the disk and then sends it in three chunks over the network. But this is still strange, I can see now data in minio during upload. 
|
Tusd buffers data on disk before uploading it to S3. This happens in chunks, so the entire file isn't stored on disk. We have to do this because S3 expects us to known the size of the chunk we are uploading. See #1084 for more details and a potential alternative/improvement. |
S3 uploads are performing using S3 nultipart uploads, which don't shown up in the file explorer and ListObjects calls until the upload is done. |
|
Thanks for you explanation and the insides. Very helpful. Do you also have an explanation why it has no writes on the disk, if throttling is disabled as you can see in the first screenshot? |
|
Tested it without Caddy today and I couldn't reproduce it. So it seems it's an issue when using caddy. |
|
I think I'm getting the same when working with telnyx.com and backblaze.com I'm using embedded golang package, and testing it from local host. And this is what I get :/ For frontend I use uppy. Logs from telnyx.com Logs from backblaze.com 
I tried to find any way to configure some timeout maybe, but didn't find anything. My setup is |
|
I solved the issue by setting @akkie I'm sure there should be a similar command line param. |
|
@akkie I just had the chance to look more into this and I can shed additional light on the situation. I tried to reproduce the problem using https://uppy.io/examples/, which uploads to our public tusd instance at tusd.tusdemo.net/files/. Even with 30 files totaling to ~300 MB and a throttled connection (similar to your setup in #1172 (comment)), I was not able to reproduce this problem. So it does not seem to be caused inside tusd. The 502 Bad Gateway response (#1172 (comment)) that the browser receives is not from tusd, but Caddy instead. When a proxy forwards a request to tusd, it waits for a response from tusd. If no request is received in a certain period, the proxy stops the forwarded request and responds with a 502 Bad Gateway to the client/browser. On the other side, tusd sees that the request is cancelled and stops processing giving the It's possible that due to the throttling, the requests take longer than Caddy expects and thus exceeds its timeout for waiting for a response. Without throttling, the responses arrive quickly enough to not trigger the timeout. If you or anybody else is having this problem I would recommend you to look into your proxies settings and see if you can increase the timeout waiting for responses. Or even better, the proxies' timeout should only start when the full request has been sent to tusd. I hope this still helps! Since the original issue is a bit older, I will close this for now. If you still have this problem, feel free to leave a comment and we can continue the discussion here. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I locally test tusd with Minio. To test my frontend I throttle the upload speed in the browser. If I upload as example 30 image files, then after a certain time I get a lot of
operation error S3: CompleteMultipartUpload, context canceledmessages in the tusd log. Maybe it's a timeout on the writing side to S3?There is no such problem if I do not throttle. But this might be still an issue if a client uploads data with limited bandwidth.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
There should not be an error if the bandwidth is limited.
Setup details
Please provide following details, if applicable to your situation:
-s3-disable-ssl=true
-s3-endpoint=http://minio:9100
-port=9500
-max-size=1000000000
-cors-allow-origin=https://my.domain
-cors-allow-credentials=true
-behind-proxy=true
-show-greeting=false
-hooks-http=https://api.my.domain/hooks/tusd
-hooks-enabled-events=pre-create,pre-finish,post-finish
-hooks-http-forward-headers=Cookie
The text was updated successfully, but these errors were encountered: