All pods related to service not being scraped by service autodiscovery via annotations

[timharris777]

Using helm chart 1.4.3. I have had a service being scraped via annotation autodiscovery. Today we noticed that certain metrics were missing. The service has an underlying pod that can grow to up to 8 pods depending on load. If we only have 1 pod all the metrics are scraped. If we have more than one pod behind the service all the metrics are not scraped.

I was under the impression that if I add the k8s.grafana.com/scrape: "true" annotation to a service then alloy would get all the endpoints for that service and scrape them individually. Have I misunderstood how this works? Is there some other reason why this isn't working?

I switched and added the autodiscovery annotation to the deployment template and all the metrics are now being scraped properly via pod autodiscovery. I am so confused. Have I missed something?

Any help would be appreciated. Thanks!

[timharris777]

Interesting-- found this stating that discovery.kubernetes.services "discovers a target for each service port for each service." This seems to indicate that not all endpoints (pods) backing a service get set as targets to scrape, but only one.

My assumption was that it was working the same as prometheus.operator.servicemonitors documented here which states that all endpoints for a service are collected and scraped.

This would essentially mean that adding k8s.grafana.com/scrape: "true" annotation to a service with multiple pods backing it will not scrape all the pods for metrics, but only one.

Can someone from Grafana confirm this for me? And if so, is this expected behavior? I would think you would want discovery.kubernetes.services to scrape all underlying endpoints (pods) of a service.

[petewall]

Yeah, currently, services discovery will scrape once per service, pod discovery will scrape per pod.
One of the use cases for the current setup is for services where each pod will respond with the same output, but there might be multiple pods for load balancing or redundancy.

Let me think about this. Perhaps with a special annotation, we can trigger discovery by endpoint vs by service.