Example of a latency SLO?

[prologic]

Do you have an example of one?

[prologic]

I can see support for latency SLOs were added in #711 but I'm not sure how to use this just yet.

Ideally I'd want to take the measurements from Traefik which is already measuring latencies for backend services in buckets of 0.1, 0.3, 0.5, 1.2 and +Inf (seconds).

• traefik_service_request_duration_seconds_bucket
• traefik_service_request_duration_seconds_sum
• traefik_service_request_duration_seconds_count

I'm not sure yet how to specify the query for latencyNative.total.metric just yet...

[abestel]

These metrics look like “regular” histograms rather than native ones. I think you can use this example in your case: https://github.com/pyrra-dev/pyrra/blob/main/examples/caddy-response-latency.yaml

[metalmatze]

Here's a live demo for this:
https://demo.pyrra.dev/objectives?expr=%7B__name__=%22caddy-response-latency%22%7D&grouping=%7B%7D

[prologic]

Ahh I see. Thanks! I'll give this a go with my Traefik metrics for some of my services.

[prologic]

Hmmm, isn't this a counter though? At least it looks like it for me. I'm also not sure how to do this properly given you have different time series (counters) for each status code. See screenshot

I can see your example is using the normal ratio of success / total (or errors / total) which I assume expects a single time series right?

[prologic]

I'm hoping this works :D

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: Twtxt Latency
  namespace: twtxt
  labels:
    service: twtxt
    category: public
spec:
  target: '95.0'
  window: 28d
  indicator:
    ratio:
      errors:
        metric: sum(traefik_service_request_duration_seconds_bucket{service="twtxt_twtxt@docker", code!~"5..", le="0.1"})
      total:
        metric: sum(traefik_service_request_duration_seconds_count{service="twtxt_twtxt@docker", code!~"5.."})

[prologic]

Okay this looks right:

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  creationTimestamp: null
  labels:
    category: public
    service: twtxt
  name: Twtxt Latency
  namespace: twtxt
spec:
  alerting: {}
  description: ""
  indicator:
    latency:
      grouping: null
      success:
        metric: traefik_service_request_duration_seconds_bucket{service="twtxt_twtxt@docker",
          code!~"5..", le="0.1"}
      total:
        metric: traefik_service_request_duration_seconds_count{service="twtxt_twtxt@docker",
          code!~"5.."}
  target: "95.0"
  window: 28d
status: {}

[metalmatze]

Yes, this looks good! 👍
Pyrra uses sum(rate( and sum(increase( in various places on its own. All you have to do is tell Pyrra how to select the exact time series that should be part of the success and total sum.

[prologic]

I do have one that is currently reporting "No Data", but I haven't been able to figure out why yet. The errors returns no results in Prometheus, because there haven't been any errors. But the total returns two time-series for a bunch of GEt and POST requests.