feat(example,metrics): kube-state-metrics to monitor custom resource …

[sebastiangaiser]

…state

In order to monitor the state of custom resources (CR) inside of the Kubernetes cluster, kube-state-metrics can be deployed. This describes the deployment using the prometheus-community Helm chart.

Issue: #10276

Type of change

Select the type of your PR

• Enhancement / new feature

Description

In order to monitor the state of custom resources (CR) inside of the Kubernetes cluster, kube-state-metrics can be deployed. This describes the deployment using the prometheus-community Helm chart.

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

Thanks for the PR. I think this is a good idea. I think we should consider some additional things here:

• Do we want to document this? (I guess we have some notes on what is int he examples, so we should mention it?)
• Do we want to have some no-Helm variant of this in the examples? As we in general have a pure YAML based installation as the primary source and as many users do not use Helm, so this useful only for some users.
• Should we provide this for all custom resources and deprecated (and later remove) the state metrics provided by Strimzi CO? (this might not be a task for this PR, but it should be considered when adopting this)
• Do we want to have some System Test coverage?

CC @strimzi/maintainers

[sebastiangaiser]

Should we provide this for all custom resources

I think this would be a good idea. E.g. reconcile for a Kafka resource fails, a KafkaRebalance has problems, ... But I didn't had any problems, yet. Neither I'm not sure which fields are relevant for all resources.

Do we want to have some no-Helm variant of this in the examples?

I think this depends on how you would like to deploy the monitoring. The Flux example uses the kube-prometheus-stack and injects the values there (into the Helm release). Personally I'm not a big fan of mixing my monitoring stack with some Flux/Strimzi/... specific monitoring. So I ended up deploying a second ksm for Flux and Strimzi (yes I use Flux).

[scholzj]

I think this depends on how you would like to deploy the monitoring. The Flux example uses the kube-prometheus-stack and injects the values there (into the Helm release). Personally I'm not a big fan of mixing my monitoring stack with some Flux/Strimzi/... specific monitoring. So I ended up deploying a second ksm for Flux and Strimzi (yes I use Flux).

Well, the Prometheus part seems to be just a custom resource that can have its own YAMl as well. I also assume that the confguration will end up in some ConfigMap or some environment variables to configure the Kube State Metrics? So that can be described or stored in separate YAMLs.

I think this would be a good idea. E.g. reconcile for a Kafka resource fails, a KafkaRebalance has problems, ... But I didn't had any problems, yet. Neither I'm not sure which fields are relevant for all resources.

I guess a start might be to replicate what the Cluster operator does -> a metric to indicate if the resource is ready or not. That would allow us to drop it from the Cluster operator. We can improve things later as some ideas pop-out. But we can wait for this for some discussion and have others chime in with their thoughts.