botocore client error Rate Exceeded

[nhartwic]

Basically, any tibanna stat command that would return more than about 75 results produces an error.

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/snake/bin/tibanna", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/__main__.py", line 576, in main
    subcommandf(*sc_args)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/__main__.py", line 502, in stat
    API().stat(sfn=sfn, status=status, verbose=long, n=nlines, job_ids=job_ids)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/core.py", line 483, in stat
    parse_exec_desc_and_ec2_desc(exc['executionArn'], verbose)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/tibanna/core.py", line 407, in parse_exec_desc_and_ec2_desc
    exec_desc = client.describe_execution(executionArn=exec_arn)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ubuntu/miniconda3/envs/snake/lib/python3.9/site-packages/botocore/client.py", line 915, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeExecution operation (reached max retries: 4): Rate exceeded

I'm guessing AWS made some change to table access rates or something?

[alexander-veit]

Which Tibanna version are you using? I assume this was working previously? Did you switch Tibanna versions?

I can see that you are using Python 3.9 which is not supported officially (although it most likely works). Did you install Tibanna from source?

[itenbergd]

@alexander-veit this error likely stems from not configuring the AWS SDK retry mechanism in the boto3 client. More info can be found here:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html
https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html

It is possible to utilize AWS SDK's built-in functionality of exponential backoffs for various types of exceptions (including ThrottlingException) to limit exceptions/failures back to the caller. This would be especially useful in higher scale applications of Tibanna.

Perhaps putting this config setting centrally somewhere in Tibanna would be useful to avoid these types of errors.

An example implementation below:

/tibanna/vars.py:

from botocore.config import Config

AWS_SDK_RETRY = Config(
   retries = {
      'max_attempts': 5,
      'mode': 'standard'
   }
)

/tibanna/check_task.py (line 118):

from .vars import AWS_SDK_RETRY

res = boto3.client('ec2', config=AWS_SDK_RETRY).describe_instances(InstanceIds=[instance_id])