kafka_2.13-3.7.1 without metrics

[qq359130530]

``OS: Linux
JDK：1.8.0_341
Kafka：kafka_2.13-3.7.1
jmx_exporter：1.0.1

edit kafka-server-start.sh

start Kafka, Kafka Server normal JMX port normal

But, metrics not running.

[dhoard]

Can you provide the output of netstat -an | grep LISTEN | grep -v LISTENING?

[qq359130530]

[dhoard]

The command looks incorrect.

• Do not place the JMX Exporter Jar in a directory that is in the Kafka classpath
• Add the JMX Exporter jar to the JVM opts before the classpath -cp definition

Please provide formatted text format in the future. Images are not searchable, etc.

[qq359130530]

I tried to run the Kafka & JMX Exporter console under Windows using IDEA to output the following error message:

Failed to start Prometheus JMX Exporter

java.lang.IllegalStateException: Can't register jmx_exporter_build because a metric with that name is already registered.
at io.prometheus.metrics.model.registry.PrometheusRegistry.register(PrometheusRegistry.java:26)
at io.prometheus.metrics.core.metrics.Metric$Builder.register(Metric.java:60)
at io.prometheus.metrics.core.metrics.MetricWithFixedMetadata$Builder.register(MetricWithFixedMetadata.java:47)
at io.prometheus.jmx.BuildInfoMetrics.register(BuildInfoMetrics.java:60)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:53)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)

Prometheus JMX Exporter exiting

[dhoard]

@qq359130530

For Linux

did you change the configuration on Linux as I described?

For Windows

Do you have the full command line when running in IntelliJ on Windows?

I don't use Windows, but I suspect this is a configuration issue as well... most likely the exporter Java agent is being loaded twice.

[qq359130530]

I originally wanted to run Kafka & JMX Exporter Agent on Linux, but Kafka started without exposing the /metrics service, so I tried debugging it on Windows only to see this error message with the following command-line arguments：

-XX:G1ConcRefinementThreads=13 -XX:GCDrainStackTargetSize=64 -XX:InitialHeapSize=263742592 -XX:MaxHeapSize=4219881472 -XX:+PrintCommandLineFlags -XX:ReservedCodeCacheSize=251658240 -XX:+SegmentedCodeCache -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseG1GC -XX:-UseLargePagesIndividualAllocation

[dhoard]

Can you provide the full command line that's being executed?

java.exe ...

[qq359130530]

this is Linux cmd:

/home/worker/java/jdk/bin/java -Xmx1G -Xms1G -javaagent:/home/worker/kafka/jmx_prometheus_javaagent-1.0.1.jar=12345:/home/worker/kafka/config.yaml -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/home/worker/kafka/kafka-2.12-2.0.1/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/worker/kafka/kafka-2.12-2.0.1/bin/../logs -Dlog4j.configuration=file:/home/worker/kafka/kafka-2.12-2.0.1/bin/../config/log4j.properties -cp /home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/activation-1.1.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/aopalliance-repackaged-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/argparse4j-0.7.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/audience-annotations-0.12.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/caffeine-2.9.3.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/checker-qual-3.19.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-beanutils-1.9.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-cli-1.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-collections-3.2.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-digester-2.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-io-2.11.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-lang3-3.12.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-logging-1.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/commons-validator-1.7.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-basic-auth-extension-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-json-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-mirror-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-mirror-client-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-runtime-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/connect-transforms-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/error_prone_annotations-2.10.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/hk2-api-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/hk2-locator-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/hk2-utils-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-annotations-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-core-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-databind-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-dataformat-csv-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-datatype-jdk8-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-jaxrs-base-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-jaxrs-json-provider-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-module-afterburner-2.16.2.jar/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-module-jaxb-annotations-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jackson-module-scala_2.13-2.16.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.activation-api-1.2.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.annotation-api-1.3.5.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.inject-2.6.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.validation-api-2.0.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.ws.rs-api-2.1.6.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jakarta.xml.bind-api-2.3.3.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javassist-3.29.2-GA.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.activation-api-1.2.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.annotation-api-1.3.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/javax.ws.rs-api-2.1.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jaxb-api-2.3.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-client-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-common-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-container-servlet-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-container-servlet-core-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-hk2-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jersey-server-2.39.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-client-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-continuation-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-http-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-io-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-security-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-server-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-servlet-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-servlets-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-util-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jetty-util-ajax-9.4.54.v20240208.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jline-3.25.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jopt-simple-5.0.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jose4j-0.9.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/jsr305-3.0.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka_2.13-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-clients-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-group-coordinator-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-group-coordinator-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-log4j-appender-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-metadata-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-raft-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-server-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-server-common-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-shell-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-storage-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-storage-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-examples-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-scala_2.13-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-streams-test-utils-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-tools-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-tools-api-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/kafka-transaction-coordinator-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/lz4-java-1.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/maven-artifact-3.9.6.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/metrics-core-4.1.12.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-buffer-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-codec-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-common-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-handler-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-resolver-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-classes-epoll-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-native-epoll-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/netty-transport-native-unix-common-4.1.110.Final.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/opentelemetry-proto-1.0.0-alpha.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/osgi-resource-locator-1.0.3.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/paranamer-2.8.jar:/home/worker/kafkakafka-2.12-2.0.1/bin/../libs/pcollections-4.0.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/plexus-utils-3.5.1.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/protobuf-java-3.23.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/reflections-0.10.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/reload4j-1.2.25.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/rocksdbjni-7.9.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-collection-compat_2.13-2.10.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-java8-compat_2.13-1.0.2.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-library-2.13.14.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-logging_2.13-3.9.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/scala-reflect-2.13.14.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/slf4j-api-1.7.36.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/slf4j-reload4j-1.7.36.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/snappy-java-1.1.10.5.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/swagger-annotations-2.2.8.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/trogdor-3.8.0.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/zookeeper-3.8.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/zookeeper-jute-3.8.4.jar:/home/worker/kafka/kafka-2.12-2.0.1/bin/../libs/zstd-jni-1.5.6-3.jar -Djava.security.krb5.conf=/home/worker/krb5/krb5/var/krb5kdc/krb5.conf -Djava.security.auth.login.config=/home/worker/kafka/kafka/config/kafka_client_jaas.conf kafka.Kafka /home/worker/kafkakafka-2.12-2.0.1/config/server.properties

[qq359130530]

JMX Exporter require JDK 11

[dhoard]

JMX Exporter requires JDK 11

Java 8 is supported and tested.

Here is the list of Java containers tested:

https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/java-docker-images.txt