-
Notifications
You must be signed in to change notification settings - Fork 863
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Synchronous Lambda Timeout with Lambda SDK invocation #3227
Comments
@ThomasGaboriau Good afternoon. Kindly advise if you
Thanks, |
Hello @ashishdhingra , thanks for the quick answer !
So here is what I now have, just to be sure : _lambda = new AmazonLambdaClient(new AmazonLambdaConfig()
{
LogResponse = true,
MaxErrorRetry = 0,
Timeout = TimeSpan.FromMinutes(15),
ReadWriteTimeout = TimeSpan.FromMinutes(15),
});
.....
InvokeRequest lambdaRequest = new()
{
FunctionName = Constants.AWS.RecoveryTaskArn,
Payload = json,
};
InvokeResponse response = _lambda.InvokeAsync(lambdaRequest, stoppingToken).Result; The lambda still succeeded after 6 minutes (forced with a Any other ideas ? Thanks, |
@ThomasGaboriau Apologies for delayed reply. We would need to reproduce the issue to come to any conclusion. Could you please share your Lambda configuration, if there is any setting that might be causing the issue. Are you able to reproduce it when running locally, instead of from ECS hosted service? Thanks, |
Hey @ashishdhingra When running locally and calling the Lambda, I do not reproduce the issue. Here is the lambda configuration, please tell me if you need more infos :
And there is a VPC, but I don't know much about that one. Thanks, |
I am seeing this in a similar environment but I thought it might be worth highlighting the differences and what I have tried in case it helps to diagnose the issue. In my case I first saw the issue running the lambda client from an asp.net web application running .net 8.x in EKS on x86. AWSSDK.Lamda - 3.7.305.5 I've tried downgrading both the calling and lambda ends to .net 6 but I didn't see any improvement. |
I also built a quick client based on dotnet HttpClient and AwsSignatureVersion4 and I see the same results. |
I'm going to close this as both reports in this issue seem related to the networking setup (which the SDK doesn't have access to). If you have access to AWS Support, I recommend reaching out as they'll be able to troubleshoot your specific scenario. If you don't, one option is to use the Reachability Analyzer: VPC Reachability Analyzer determines network reachability for your specified network path when you run the analysis. If your network path is reachable, we display details of the components of the network path. If the path is not reachable, we identify the blocking components. For example, you can select the Lambda ENI as source and the ECS task ENI as destination, and the tool will display hop-by-hop details of the network path between them (note: there's a cost associated with each analysis, see |
Comments on closed issues are hard for our team to see. |
Describe the bug
I invoke a possibly long running synchronous Lamba (it can either run for 30 seconds or ~10 minutes) with the
AmazonLambdaClient
, and I wait the result.The invocation is done from a WebAPI HostedService, hosted in ECS.
Short runs are fine, but longer ones will always end after the configured timeout (15 minutes in our case), even though the lambda has finished running successfully.
Expected Behavior
The lambda runs fine.
The
InvokeAsync
methods returns the result after the lambda ends.Current Behavior
The lambda runs fine.
The
InvokeAsync
throws aTimeoutException
after 15 minutesAWSSDKUtils 10|2024-03-18T16:28:06.177Z|DEBUG|Double encoded /2015-03-31/functions/{FunctionName}/invocations with endpoint https://lambda.eu-west-1.amazonaws.com/ for canonicalization: /2015-03-31/functions/arn%{arn}
AmazonLambdaClient 11|2024-03-18T16:43:06.193Z|ERROR|An exception of type TimeoutException was handled in ErrorHandler. --> System.TimeoutException: The operation was canceled.
Reproduction Steps
At first I used the
AmazonLambdaConfig
Timeout property, and the tried to use a factory to add a keep-alive.Custom Http Factory :
Lambda client configuration, with AWS loggin config :
Retries are at 0 because the lambda would retry after a timeout that succeeded.
Lambda invocation :
Possible Solution
Did I miss or missused some information or configuration ?
Additional Information/Context
No response
AWS .NET SDK and/or Package version used
AWSSDK.Lambda 3.7.304.2
Targeted .NET Platform
.NET 8
Operating System and version
Debian 12 (docker image : 8.0-bookworm-slim-arm64v8)
The text was updated successfully, but these errors were encountered: