Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch job submissions timing out #40

Open
dpwrussell opened this issue Sep 5, 2018 · 0 comments
Open

Batch job submissions timing out #40

dpwrussell opened this issue Sep 5, 2018 · 0 comments

Comments

@dpwrussell
Copy link
Contributor

Submissions of batch jobs have been exceeding the 6 second default lambda limit.

[INFO]	2018-09-05T01:17:24.391Z	Found credentials in environment variables.
[INFO]	2018-09-05T01:17:24.638Z	Starting new HTTPS connection (1): ssm.us-east-1.amazonaws.com
START RequestId: e50b2988-6e16-494c-af8e-6e50355d493d Version: $LATEST
Received event: {
"import_uuid": "848e5eea-35e9-412d-adec-a0c023579e96",
"files": [
"ashlar_examples/BP40.ome.tif"
],
"reader": "loci.formats.in.OMETiffReader",
"reader_software": "Bio-Formats",
"reader_version": "(unknown version)",
"fileset_uuid": "e7a6dbc8-a457-4d28-8b3b-65beed085716"
}
Parameters:{
"dir": "848e5eea-35e9-412d-adec-a0c023579e96",
"file": "ashlar_examples/BP40.ome.tif",
"reader": "loci.formats.in.OMETiffReader",
"reader_software": "Bio-Formats",
"reader_version": "(unknown version)",
"fileset_uuid": "e7a6dbc8-a457-4d28-8b3b-65beed085716",
"bucket": "minerva-test-cf-common-tilebucket-1su418jflefem"
}
[INFO]	2018-09-05T01:17:24.875Z	e50b2988-6e16-494c-af8e-6e50355d493d	Starting new HTTPS connection (1): batch.us-east-1.amazonaws.com
END RequestId: e50b2988-6e16-494c-af8e-6e50355d493d
REPORT RequestId: e50b2988-6e16-494c-af8e-6e50355d493d	Duration: 6006.21 ms	Billed Duration: 6000 ms Memory Size: 1024 MB	Max Memory Used: 79 MB	
2018-09-05T01:17:30.840Z e50b2988-6e16-494c-af8e-6e50355d493d Task timed out after 6.01 seconds

[INFO]	2018-09-05T01:17:31.967Z	Found credentials in environment variables.
[INFO]	2018-09-05T01:17:32.37Z	Starting new HTTPS connection (1): ssm.us-east-1.amazonaws.com

There should be no reason for this to time out, it looks like potentially an AWS issue. If this is more than a one time error then we can work around by increasing the lambda timeout, or with retry logic in the step function.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant