Don't retry on 500 errors

[dsotirho-ucsc]

… only 503 & 429.

[dsotirho-ucsc]

@hannes-ucsc: "Spike to review past 500 errors returned by the service in order to determine if some of them should be retried. We also need to determine if 502 & 504 from the API Gateway should be retried."

[achave11-ucsc]

@hannes-ucsc: "When diagnosing https://ucsc-gi.slack.com/archives/C03TPJS54DC/p1761247675944979?thread_ts=1761246334.325919&cid=C03TPJS54DC, I noticed that 400 errors are also being retried. That shouldn't happen either. 400 means that the client sent and invalid request, and without modifying the request retries are futile."

[dsotirho-ucsc]

Spike to review past 500 errors returned by the service in order to determine if some of them should be retried

Over the last three months on prod and anvilprod, only one cause of a 500 error on prod could be found that would have benefited from the users (101 total requests) retrying the request. The cause elasticsearch.exceptions.NotFoundError lasted under a minute on Aug 28 due to the promotion of an update to OpenSearch.

The full list of 500 errors, sorted by deployment & number of occurrence:

Anvilprod service, 500 status, last 3 months

Count: 116
Date: Oct 4
Error: KeyError: "Key 'size' not defined in field_types"

Temporarily (~2 hour) broken due to promotion of
Rename files.url to files.azul_file_url for AnVIL #6549

---

Count: 92
Date: Aug 29
Error: elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f905c73b530>: Failed to establish a new connection: [Errno -2] Name or service not known)

Temporarily (~1 hour) down due to promotion of
Update to OpenSearch 2.x (#6779, #6911) #7360

Prod service, 500 status, last 3 months

Count: 2249
Date: Oct 20
Error: botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://dynamodb.us-east-1.amazonaws.com/\"

Temporarily broken due to AWS outtage (~1 hour)

---

Count: 599
Date: Aug 29
Error: elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f0f73470e30>: Failed to establish a new connection: [Errno -2] Name or service not known) caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f0f73470e30>: Failed to establish a new connection: [Errno -2] Name or service not known)

Temporarily (~30min) down due to promotion of
Update to OpenSearch 2.x (#6779, #6911) #7360

---

Count: 120
Date: Aug 11 - Oct 6
Error: chalice.app.BadRequestError: Error Parsing JSON

Fixed by 2025-10-14 promotion
GET request with empty JSON body causes 5xx (DataBiosphere/azul-private#289)

---

Count: 118
Date: Aug 11 - Sept 9
Error : ValueError: Expected exactly one item in iterable, but got {'sourceId': 'a4139ea7-84b8-449a-a62a-9a017bfbf0a1', 'sourceSpec': 'tdr:bigquery:gcp:datarepo-69bbc337:hca_prod_9c20a245f2c043ae82c92232ec6b594f__20220212_dcp2_20231102_dcp33:/0'}, {'sourceId': 'fa6ab844-0951-478f-9dd0-450f55ca445e', 'sourceSpec': 'tdr:bigquery:gcp:datarepo-50b00aaf:hca_prod_2079bb2e676e4bbf8c68f9c6459edcbb__20240327_dcp2_20240807_dcp41:/0'}, and perhaps more

Fixed when bad snapshots were deindexed Sept 9th
Non-disjunctive projects in dcp51 and dcp52 #7397

---

Count: 101
Date: Aug 29
Error: elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [azul_v2_prod_dcp52_files_aggregate]', azul_v2_prod_dcp52_files_aggregate, index_or_alias)

Temporarily (~1 min) down due to promotion of
Update to OpenSearch 2.x (#6779, #6911) #7360

---

Count: 20
Date: Sept 16 - Oct 24
Error: TypeError: unsupported operand type(s) for |: 'str' and 'dict'

Unresolved bug:
Tampering with paging links causes 5xx response #7418

---

Count: 23
Date: Sept 20-39
Error: Caught exception for path /index/{entity_type} KeyError: 'donors'

Unresolved bug:
Request for non-existing /index endpoints with size parameter yields 500 status #7430

---

Count: 9
Date: Oct 11
Error: TypeError: TerraClient._request() takes 3 positional arguments but 4 were given

Temporarily broken by attempted hotfix
Frequent hangups by TDR #7448

[dsotirho-ucsc]

We also need to determine if 502 & 504 from the API Gateway should be retried

Looking at the API Gateway on prod and anvilprod over the last 3 months, only one request that received a 504 on anvilprod would have likely succeeded on retry. The other 504 were due to an AWS outage that took a few minutes to resolve.

API Gateway Anvilprod

Count: 2
Error: 504

(One occurrence on Aug 29, and one due to Oct 20 AWS Outage)

INTEGRATION_TIMEOUT
Endpoint request timed out
Execution failed due to a timeout error

API Gateway Prod

Count: 379
Error: 502

INTEGRATION_FAILURE
Internal server error
Lambda Customer Function Error:Response payload size exceeded maximum allowed payload size (6291556 bytes).

---

Count: 527
Error: 504

(Oct 20 AWS Outage)

INTEGRATION_TIMEOUT
Endpoint request timed out
Execution failed due to a timeout error

---

Count: 1
Error: 502

INTEGRATION_FAILURE
Internal server error
Lambda Customer Function Error:Unable to import module 'app': No module named 'elasticsearch'

Race between layer and Lambda update DataBiosphere/azul#5927

[dsotirho-ucsc]

@hannes-ucsc:

Based on this, Azul should behave the following way:

• 429 Retry
• 400 No retry
• 500 No retry
• 502 No retry
• 503 Retry
• 504 No retry

All these response statuses should be documented in Swagger, the responses for retry-able statuses should include a retry-after header, and that header should be documented in the Swagger UI for each retry-able status. Continue spike to ensure that this work is tracked in Azul tickets, existing or new. These tickets should become blockers of this ticket. Once the Azul work is done, we need to communicate the desired behavior to CC as part of this ticket.

[dsotirho-ucsc]

@hannes-ucsc: "Daniel pointed out that the above contradicts a previous decision of mine. That's true, but given the evidence provided in this issue, the occurences of 504 would actually be aggravated by retries."