Computing SLI in the absence of errors and/or absence of activity

[multani]

Hello,

I'm in the process of testing out Pyrra to replace Sloth to monitor our SLOs.

I'm trying to define a very simple "availability" SLI, base on nginx requests. The spec looks like this:

      errors:
        metric: |
          nginx_ingress_controller_requests{ingress="my-service", status=~"^5.+"}

      total:
        metric: |
          nginx_ingress_controller_requests{ingress="my-service"}

That's a very simple example!

In Sloth, we used to have 2 "hacks":

• Add or on() vector(0) to the "error" query: when a service has no errors, the original "error" query doesn't return any numbers and it creates empty metrics (instead of error = 0), which in turn renders a lot of dashboard a bit useless.
  To compensate, we added that or on() vector(0) on all the Sloth error queries.

• In addition to the hack on the error query, we had (sometimes) the same hack on the "total" query, to compensate for some services which don't always have requests all the time. In this situation, the original "total" query doesn't return any number, thus creating holes in the final metrics and dashboards.
  We tried to compensate this by adding or on() vector(1) (describe a bit in NaN in SLO dashboard slok/sloth#531 (comment)), although this one is a bit more doubtful.

I initially tried to configure the same with Pyrra, but by default, Pyrra returns a ratio total metric is not a VectorSelector error.

• How do these 2 problems (absence of errors and/or absence of measurable activity) are handled in Pyrra?
• Are there any recommendations on how to write the metric queries?
• Is there something done automatically to compensate for the missing values?

Thanks for your help!

[metalmatze]

Hey, these are excellent questions.
Pyrra doesn't use vector(0) to fill the gap when there are no series. Pyrra, however, generates another alert called SLOMetricAbsent that alerts almost instantly if the underlying metrics are gone.
When using the UI to adjust for these gaps, Pyrra uses the or vector(0) to smooth things out in the graphs.

Let me know if this already clears things up.

[multani]

Thanks for the answer,

I was digging into this problem and how to address it properly. I'm testing a completely new approach, I'll report back in a couple of weeks if it's successful or not :D (instead of trying to hack the PromQL queries to fallback to specific values when the metric is absent, I'll try actually to generate a 0 metric in Prometheus)

[metalmatze]

Would you happen to have any updates?
I'm curious, seeing this again.

[multani]

hey @metalmatze,

I haven't hard the chance to try out pyrra yet, so I did some hacking around what we had with Sloth.

---

The first problem we had was when metrics report nothing for the error query (not 0, but really no values). This breaks a lot of further computations when we try to compute an error rate with null / X requests since this also returns null. I really wanted to get a 0% error rate in this case.
I fixed this problem by writing a small Prometheus exporter that always return something for the error query.

In the example I gave:

# this is a counter
nginx_ingress_controller_requests{ingress="my-service", status=~"^5.+"}

nginx, like many service, doesn't counter any 5xx when they are no 5xx and returns no values in this case.
So, my Prometheus exporter produces "fake" nginx metrics and would always return 0 here:

nginx_ingress_controller_requests{ingress="my-service", status="500"} 0

This metric is a counter, so returning 0 here shouldn't have any negative impact on the actual measurements. However!, it should prevent my error query to produce no value, since now there will always be a value.

I think this "trick" could also be used in pyrra, AFAIU.

---

For the second problem (some services don't have a lot of requests and may sometimes have 0 request/seconds). This is also a problem because if you try to compute 0 errors/s / 0 req/s, it also returns no values (note that it might be less a problem for pyrra than Sloth, since Sloth requires to have rate() PromQL query, which may produce 0 if there are no new requests).

In this case, I updated a bit our Sloth "total" query to be like this:

rate(nginx_ingress_controller_requests{ingress="my-service"}[{{.window}}]) + 0.0000000001

This query is a bit lying since it will report a extremely tiny amount of requests/second, even if there were none. This may move the SLI higher than what it really is, but it also prevents from having any "holes" inside the SLI reported, which (I think) helps to increase confidence in the SLO.

---

@metalmatze sorry, this is in the end not really Pyrra focused :( But hopefully it gives you an idea of what I was trying to achieve:

1. The first hack, using the "fake" metrics exported by the dummy Prometheus exporter, would be an idea that I could reuse with Pyrra I think.
2. For the second hack, I'm not sure how that would work, but I admittedly haven't re-tested yet Pyrra since :(