Downloading all.zip fails if larger than 2GB

[rrowlands]

When using acls other than "public-read", and downloading the all zip from S3, as per task/uuid/download/all.zip, this S3.GetObject

ClusterODM/libs/proxy.js

Line 540 in 7c77349

s3.getObject({ Bucket: s3Config.bucket, Key: key }, (err, data) => {

request fails if the size of the all.zip file is too large (larger than 2GB?), as per:

aws/aws-sdk-js#2916

This error can be found in the cluster odm logs as:

error: Error encountered downloading object RangeError: The value "..." is invalid for option "size".

The AWS team recommends using multi-part downloads to avoid this error.

[rrowlands]

This issue, when encountered, is actually pretty nefarious because it seems to also cause a memory leak (and I think it leaks an ec2 instance as well). Here's a sample of the server logs where the issue is occurring:

error: Error encountered downloading object RangeError: The value "5337046766" is invalid for option "size"
info: Trying to create machine... (1)
warn: Cannot update info for 10.120.10.207:3000: connect ECONNREFUSED 10.120.10.207:3000
info: Waiting for 10.120.10.207:3000 to get online... (1)
error: Error encountered downloading object RangeError: The value "6677566819" is invalid for option "size"
info: Trying to create machine... (1)
warn: Cannot update info for 10.120.10.239:3000: connect ECONNREFUSED 10.120.10.239:3000
info: Waiting for 10.120.10.239:3000 to get online... (1)
error: Error encountered downloading object RangeError: The value "11061465366" is invalid for option "size"
warn: Remove and cleanup failed: spawn ENOMEM
warn: Remove and cleanup failed: spawn ENOMEM
error: Error encountered downloading object RangeError: The value "11947370985" is invalid for option "size"
info: Trying to create machine... (1)
warn: Cannot create machine: Error: spawn ENOMEM
info: Trying to create machine... (2)
warn: Cannot create machine: Error: spawn ENOMEM
info: Trying to create machine... (3)
warn: Cannot create machine: Error: spawn ENOMEM
warn: Could not remove docker-machine, it's likely that the machine was not created, but double-check!
warn: Cannot create node via autoscaling: Cannot create machine (attempted 3 times)
warn: Cannot forward task 51c710a6-4f4f-4074-a828-670c324b1cb7 to processing node uasdm-nodeodm-prod:3000: No nodes available (attempted to autoscale but failed). Try again later.
error: Uncaught Exception: Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:408:11)
    at spawn (child_process.js:677:9)
    at Object.execFile (child_process.js:311:17)
    at Object.exec (child_process.js:212:25)
    at Object.rmfr (/var/www/libs/utils.js:177:34)
    at /var/www/libs/utils.js:166:22
    at suppressedCallback (fs.js:236:5)
    at FSReqCallback.oncomplete (fs.js:180:23)
error: Uncaught Exception: Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:408:11)
    at spawn (child_process.js:677:9)
    at Object.execFile (child_process.js:311:17)
    at Object.exec (child_process.js:212:25)
    at Object.rmfr (/var/www/libs/utils.js:177:34)
    at /var/www/libs/utils.js:166:22
    at suppressedCallback (fs.js:236:5)
    at FSReqCallback.oncomplete (fs.js:180:23)
info: Trying to create machine... (1)
warn: Cannot create machine: Error: spawn ENOMEM
info: Trying to create machine... (2)
warn: Cannot create machine: Error: spawn ENOMEM
info: Trying to create machine... (3)
warn: Cannot create machine: Error: spawn ENOMEM
warn: Could not remove docker-machine, it's likely that the machine was not created, but double-check!
warn: Cannot create node via autoscaling: Cannot create machine (attempted 3 times)
warn: Cannot forward task b76b6450-1771-4571-89d1-6888501dfa45 to processing node uasdm-nodeodm-prod:3000: No nodes available (attempted to autoscale but failed). Try again later.
error: Uncaught Exception: Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:408:11)
    at spawn (child_process.js:677:9)
    at Object.execFile (child_process.js:311:17)
    at Object.exec (child_process.js:212:25)
    at Object.rmfr (/var/www/libs/utils.js:177:34)
    at /var/www/libs/utils.js:166:22
    at suppressedCallback (fs.js:236:5)
    at FSReqCallback.oncomplete (fs.js:180:23)
info: Trying to create machine... (1)
warn: Cannot create machine: Error: spawn ENOMEM
info: Trying to create machine... (2)
warn: Cannot create machine: Error: spawn ENOMEM
info: Trying to create machine... (3)
warn: Cannot create machine: Error: spawn ENOMEM
warn: Could not remove docker-machine, it's likely that the machine was not created, but double-check!
warn: Cannot create node via autoscaling: Cannot create machine (attempted 3 times)
warn: Cannot forward task b4239813-1f06-440d-9914-7fcd3e032174 to processing node uasdm-nodeodm-prod:3000: No nodes available (attempted to autoscale but failed). Try again later.
error: Uncaught Exception: Error: spawn ENOMEM
    at ChildProcess.spawn (internal/child_process.js:408:11)
    at spawn (child_process.js:677:9)
    at Object.execFile (child_process.js:311:17)
    at Object.exec (child_process.js:212:25)
    at Object.rmfr (/var/www/libs/utils.js:177:34)
    at /var/www/libs/utils.js:166:22
    at suppressedCallback (fs.js:236:5)
    at FSReqCallback.oncomplete (fs.js:180:23)

I cannot at all recommend invoking task/uuid/download/all.zip on servers not using the public-read acl until this issue is fixed.

[diegoaces]

Hi @pierotofy we have the same problem, Is there any way to solve this problem?

[pierotofy]

I'm sure there's a way, we'd welcome a pull request.