Skip to content

Adjust retries for EBS attachment checks - #30

Merged
antonipp merged 1 commit into
datadogfrom
ai/attachment-backoff-adjust
Aug 14, 2026
Merged

Adjust retries for EBS attachment checks#30
antonipp merged 1 commit into
datadogfrom
ai/attachment-backoff-adjust

Conversation

@antonipp

Copy link
Copy Markdown

Description

We noticed that EBS volume attachments are slower than the "1-4 seconds" that the polling was designed for. The current polling schedule is t = 0, 1.0, 2.8, 6.04, 11.87, 22.37s, ... which makes us waste time since the attachment times we have now are:

  • p50 = lands in the 1s-2.8s polling window
  • p75 = lands in the 2.8s - 6s polling window
  • p90 = lands in the 6s-11.9s polling window
  • p99 = lands in the > 22.37s polling window

This PR makes the retries a bit tighter to reduce the time it takes to detect an attached volume. This will slightly increase the amount of DescribeVolumes requests.

@github-actions

Copy link
Copy Markdown

Code Coverage Diff

This PR does not change the code coverage

@eakman-datadog eakman-datadog left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@antonipp This seems okay to me. Could we try just deploying it to the terrapin clusters though and confirming we see an improvement?

@eakman-datadog
eakman-datadog self-requested a review August 14, 2026 13:21
@antonipp
antonipp merged commit 3ff1961 into datadog Aug 14, 2026
8 checks passed
@antonipp
antonipp deleted the ai/attachment-backoff-adjust branch August 14, 2026 13:41
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants