Pyrra causes instability of Prometheus #1153
-
|
Hi there, and thank you for Pyrra! I'm aware this is going to be a really vague issue report, but we've been plagued with Prometheus stability issues for the last month and have come to realise that Pyrra is causing this. We see our Prometheus pod being killed by Kubernetes and logging it received a SIGTERM. There are no OOM messages nor any probe issues on the container. This happens about every 10-40 minutes. You can see a graph here where we removed the entire Pyrra helm chart for a few days and then turned it back on today.
I'd like to be able to dig into why this might be, but I'm not really sure where to start. It took me several days of digging to even realise it was Pyrra at "fault". Perhaps you could point me in the right direction? |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 4 replies
-
|
We do see some errors in the Pyrra pods. The SLO is defined like so apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
name: inmusicprofile-authorised-devices
namespace: monitoring
labels:
prometheus: k8s
role: alert-rules
pyrra.dev/team: webservices
pyrra.dev/ns: inmusicprofile
pyrra.dev/service: AuthorisedDevicesService
pyrra.dev/tier: "4"
spec:
target: "99"
window: 4w
description: AuthorisedDevicesService public endpoints.
indicator:
ratio:
errors:
metric: traces_spanmetrics_latency_count{span_name=~"inmusicapi\\.v1\\.AuthorisedDevicesService\\/.*", status_code="STATUS_CODE_ERROR"}
total:
metric: traces_spanmetrics_latency_count{span_name=~"inmusicapi\\.v1\\.AuthorisedDevicesService\\/.*"} |
Beta Was this translation helpful? Give feedback.
-
|
It would be interesting to see what the final recording rules look like. That being said, I would investigate the cardinality of your |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your response. This sounds correct, of course! We'll look into this now. |
Beta Was this translation helpful? Give feedback.
-
|
@snikch Hello! Thanks in advance. |
Beta Was this translation helpful? Give feedback.
-
|
What exactly are you observing? As written above, it would be the same if similar recording and alerting rules would be added to your Prometheus. Especially the |
Beta Was this translation helpful? Give feedback.
-
|
@giz33 please don't hold off - this ultimately was an issue with the way we were deploying. When we deployed our monorepo we do an "slo deploy" which deletes and recreates all Pyrra SLOs. Because this change was over enough rulesets, Prometheus in k8s decides to restart the prometheus service instead of just updating the rules. If the number of changed rules was low enough, this happens without a restart but if a threshold is breached then Prometheus instead decides to restart the container with the new rules. So the be explicitly clear: There was no performance issues with Pyrra. We've updated our deploy to use The 10-40 minute occurrence interval was because we'd misconfigured our slo deploy job to run on most CI runs (instead of just when the SLOs were changed). |
Beta Was this translation helpful? Give feedback.
-
|
Hi @snikch and @metalmatze ! o/ |
Beta Was this translation helpful? Give feedback.
It would be interesting to see what the final recording rules look like.
Overall, I don't think the issue is too specific to Pyrra. It's more like whatever recording rules you feed into your Prometheus.
A similar recording rule that wouldn't come from Pyrra but elsewhere might have a similar impact.
That being said, I would investigate the cardinality of your
traces_spanmetrics_latency_countmetric and then how expensive it is to run aincrease(traces_spanmetrics_latency_count[4w])for that metric.