-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
istio merged metrics cannot be used #1060
Comments
Hello @vitalii-buchyn-exa !
I only know that this is needed for seamless workig |
hello @bartam1 ! Thank you for the suggestion, but that doesn't work. After applying that annotation it looks like:
Our goal is to use merged metrics that should be exposed on TCP port 15020 via /stats/prometheus endpoint |
Hello @vitalii-buchyn-exa ! Koperator reconcile the broker pods when the pod k8s resource is changed. |
hello @bartam1, If we remove broker annotations from kafkacluster CR, then istio will not setup an ISTIO_PROMETHEUS_ANNOTATIONS env var and pilot prometheus will not be aware where to scrape broker' metrics.
That's the issue i'm trying to emphasise. Appreciate your time and help! |
any update/help please? can koperator just ignore existing prometheus annotations? |
Description
Hello community,
We are experiencing an issue with scraping istio merged metrics with
enablePrometheusMerge
istio feature.In order to pull metrics by a pilot we need to add prometheus annotations to a brokerAnnotations spec of KafkaCluster object, like:
This should result in a following env var for istio-proxy sidecar:
The problem is that istio webhook overrides those annotations for a pod to make a merged metrics endpoint available for prometheus, like:
That webhook override causes koperator to instantly restart brokers.
Please let me know if any additional info is required.
Regards,
Vitalii
Expected Behavior
Maybe ignore istio specific prometheus annotations.
Yes, this will require manual brokers restart when prometheus config changes in kafkacluster object.
Actual Behavior
Continious pods restart.
Affected Version
kafka-operator:v0.22.0
banzaicloud-kafka:2.13-2.7.0-bzc.2
banzaicloud-cruise-control:2.5.101
jmx-exporter: banzaicloud-jmx-javaagent:0.14.0
cruise-control:2.5.101
Checklist
The text was updated successfully, but these errors were encountered: