Noisy alerts on client and server response time metrics

[AAkindele]

Alerts are firing A LOT. Initial investigation of the data suggests that the thresholds are set too low.
Example: HighClientResponseTimeWestUS2-cosmos has a threshold of 178ms. The data for the past month show that 98th percentile is around 178ms.

The alerts are the HighClientResponseTime, and HighServerResponseTime alerts for all 3 clusters.

Action Item:

• Discuss as a team. a criteria for identifying what is the acceptable threshold for these response time alerts.
• Use this criteria to periodically evaluate if alerts thresholds are still valid.