Change upload to use 2 requests or http 100 code

[dilshat]

Currently we are having issues with upload of files to our CDN:

1. Token gets expired due to network latency
2. Clients fail late (in case of server errors only after a full upload is complete)

To solve these issues we can change the logic in the following way:

• Client makes initial request to server that contains the following:
  Header/Form parameters:
  token kurjun token
  md5sum of file to be uploaded
  size of file to be uploaded

• Server performs check of token and user quota (using the received size)
  a) in case checks don't pass, server returns appropriate 4xx code to let client fail early
  b) otherwise, server returns http 200 code with a new special one-time upload token which is stored on server side along with md5sum of file to be uploaded. This token has TTL of 24 hours

• Client receives the one time upload token and sends the file along with the upload token back to server
  or fails early if 4xx code is returned from the server

• Server receives the file, checks the md5sum of it along with the upload token (its ttl too), stores the file and removes the token. In case checks don't pass, the file is discarded and 4xx code is returned

• Client completes the operation with a proper result interpretation to user

[dilshat]

We can also use http 100 code where header is sent first and then body of request later within the same request.
We need to check how to support this on nginx and on client side.
If this works we can simplify the workflow:

• Client makes initial request to server that contains the following:
  Header:
  Expect:100-continue
  Header/Form parameters:
  token kurjun token
  md5sum of file to be uploaded
  size of file to be uploaded

• Server performs check of token and user quota (using the received size)
  a) in case checks don't pass, server returns appropriate 4xx code to let client fail early
  b) otherwise, server returns http 100 code

• Client proceeds with sending the request body (the file ) to server or fails early if 4xx code is returned from the server

• Server receives the file, checks the md5sum of it and stores the file. In case md5sums don't match, the file is discarded and 4xx code is returned

• Client completes the operation with a proper result interpretation to user

[dilshat]

Here we also can simplify the signing of uploaded file, we can join upload request with sign request:
Instead of sending just an md5sum , we will send a signed md5sum.
This way we can also get a signed upload in one shot

[samsonbek]

question: is md5 is reliable for this, or it's better to use sha1/256 checksum?

[dilshat]

yeah, we can use sha256

[emli]

After implementing this method, how clients of gorjun will upload to gorjun? I am wondering about clients of gorjun. How will you implement client side?

[dilshat]

To make NGINX support Expect:100-Continue header we need to add these directives to NGINX conf:

proxy_request_buffering off;
proxy_http_version 1.1;

From client side we need to send header Expect: 100-Continue

[dilshat]

Go http client supports Expect: 100-Continue
However Chrome browser prohibits using this header in client requests and fails with error:
Refused to set unsafe header "Expect"
So until we can workaround this problem we can not enable Hub uploads.
For this reason we freeze this issue

Also see https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=2&cad=rja&uact=8&ved=0ahUKEwjdncPpyqfYAhWKFZoKHeOeCcAQFggtMAE&url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F18029340%2Fexpect-100-continue-header-with-xmlhttprequest&usg=AOvVaw3wRbxoY9naYHkMapthyNsf