Support for JMX filter pattern for kafka metrics #908

sreelu27 · 2024-01-24T15:17:25Z

I am trying to create a pattern for the below Kafka metrics to be filtered:
kafka_network_RequestMetrics_Count

I have tried the below pattern :

pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

Even after applying this filter, I can see other metrics related to jvm and jmx.

Below is the config :

lowercaseOutputName: false
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

The text was updated successfully, but these errors were encountered:

sreelu27 · 2024-01-25T11:26:03Z

Hi @dhoard
Can we get any help on the filtering issue, please?

dhoard · 2024-01-26T14:09:28Z

@sreelu27 are you wanting any of the metrics from the RequestMetrics MBean?

sreelu27 · 2024-01-30T10:05:58Z

Hi @dhoard ,

Do you want any of the metrics from the RequestMetrics MBean ? – No

The issue here is when I apply this filter, all network related metrics are coming but along with that JMX and JVM metrics also coming that's the problem.

**jvm_threads_deadlocked_monitor 0.0

HELP jvm_threads_state Current count of threads by state

TYPE jvm_threads_state gauge

jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 25.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 15.0
jvm_threads_state{state="TIMED_WAITING",} 36.0
jvm_threads_state{state="UNKNOWN",} 0.0**

HELP kafka_network_RequestMetrics_Count Attribute exposed for management kafka.network:name=LocalTimeMs,type=RequestMetrics,attribute=Count

TYPE kafka_network_RequestMetrics_Count untyped

kafka_network_RequestMetrics_Count{name="LocalTimeMs",request="Fetch",} 0.0

**# HELP jmx_scrape_duration_seconds Time this JMX scrape took, in seconds.

TYPE jmx_scrape_duration_seconds gauge

jmx_scrape_duration_seconds 0.15442887**

sreelu27 · 2024-02-01T10:20:17Z

Hi @dhoard,
Can we get any help on the issue, please?

sreelu27 · 2024-02-02T10:47:03Z

Hi @dhoard,
Can we get any help on the previous comment, please ?
#908 (comment)

sreelu27 · 2024-02-02T13:38:51Z

Hi @dhoard,

I tried to add blacklistObjectNames as below, but still I am getting the jvm metrics,
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="TERMINATED",} 0.0
jvm_threads_state{state="RUNNABLE",} 21.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="WAITING",} 17.0
jvm_threads_state{state="TIMED_WAITING",} 39.0
jvm_threads_state{state="UNKNOWN",} 0.0

    lowercaseOutputName: false
    blacklistObjectNames: ["jvm.threads*"]
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

dhoard · 2024-02-02T16:44:20Z

@sreelu27 if you don't want any request metrics, then you should use...

excludeObjectNames: ["kafka.network:type=RequestMetrics,*"]

... which will filter all all MBean's with ObjectNames that match.

sreelu27 · 2024-02-02T16:46:02Z

Hi @dhoard ,

No I want to exclude the jvm metrics

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1580928.0
jvm_memory_pool_bytes_used{pool="Metaspace",} 6.7503056E7
jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 1.213376E7
jvm_memory_pool_bytes_used{pool="Compressed Class Space",} 8137312.0
jvm_memory_pool_bytes_used{pool="G1 Eden Space",} 3.3554432E7
jvm_memory_pool_bytes_used{pool="G1 Old Gen",} 1.7213184E8
jvm_memory_pool_bytes_used{pool="G1 Survivor Space",} 2097152.0
jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 2563200.0

Is there any way I can do that?

We are using jmx version 0.18.0

dhoard · 2024-02-02T17:26:44Z

@sreelu27 some questions... since this is different than what you originally requested...

I am trying to create a pattern for the below Kafka metrics to be filtered:
kafka_network_RequestMetrics_Count

Are you wanting to exclude all JVM metrics?
Are you running the exporter as a Java agent?

sreelu27 · 2024-02-02T17:30:41Z

Hi @dhoard ,
apologies for the confusion.
Are you wanting to exclude all JVM metrics? yes
Are you running the exporter as a Java agent? yes

dhoard · 2024-02-02T17:36:58Z

@sreelu27 you should be able to use the configuration excludeObjectNames to exclude the MBean that is providing the metrics.

sreelu27 · 2024-02-06T12:45:07Z

Hi @dhoard,

I was trying to apply the configuration excludeObjectNames using the below configmap

kafka_metrics_config.yaml: |
    lowercaseOutputName: true
    excludeObjectNames: ["java.lang:name=G1 Young Generation,type=GarbageCollector","java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool","java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool"]
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

curl localhost:9404 | grep CodeHeap                         

jvm_memory_pool_bytes_used{pool="CodeHeap 'non-nmethods'",} 1538688.0
261jvm_memory_pool_bytes_used{pool="CodeHeap 'profiled nmethods'",} 8927744.0
5jvm_memory_pool_bytes_used{pool="CodeHeap 'non-profiled nmethods'",} 2490880.0
  jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-nmethods'",} 2555904.0
 jvm_memory_pool_bytes_committed{pool="CodeHeap 'profiled nmethods'",} 1.1730944E7
 0     0   169k      0 --:--:-- --:--:-jvm_memory_pool_bytes_committed{pool="CodeHeap 'non-profiled nmethods'",} 2752512.0
- jvm_memory_pool_bytes_max{pool="CodeHeap 'non-nmethods'",} 5828608.0
-jvm_memory_pool_bytes_max{pool="CodeHeap 'profiled nmethods'",} 1.22912768E8
jvm_memory_pool_bytes_max{pool="CodeHeap 'non-profiled nmethods'",} 1.22916864E8
-jvm_memory_pool_bytes_init{pool="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_pool_bytes_init{pool="CodeHeap 'profiled nmethods'",} 2555904.0
:jvm_memory_pool_bytes_init{pool="CodeHeap 'non-profiled nmethods'",} 2555904.0
--jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 1.0053888E7
:--jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 2187776.0
 jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 1537152.0
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'profiled nmethods'",} 1.707223262247E9
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-profiled nmethods'",} 1.70722326225E9
 171k
jvm_memory_pool_allocated_bytes_created{pool="CodeHeap 'non-nmethods'",} 1.70722326225E9

I have tried, but didnt work,

https://github.com/prometheus/jmx_exporter/blob/f96796d5b242ffa9b0ec53e30091429901c8c52a/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/ExcludeObjectNamesTest/JavaAgent/exporter.yaml

kafka_metrics_config.yaml: |
    lowercaseOutputName: false
    excludeObjectNames: ["java.lang:type=ObjectName"]
    excludeObjectNameAttributes:
        "java.lang:name=Survivor Space,type=MemoryPool":
            - "ObjectName"
        "java.lang:type=OperatingSystem":
            - "ObjectName"
        "java.lang:type=Threading":
            - "ObjectName"
        "java.nio:name=direct,type=BufferPool":
            - "ObjectName"
        "java.lang:type=Compilation":
            - "ObjectName"
        "java.lang:name=Tenured Gen,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=CodeCacheManager,type=MemoryManager":
            - "ObjectName"
        "java.util.logging:type=Logging":
            - "ObjectName"
        "java.lang:type=ClassLoading":
            - "ObjectName"
        "java.lang:name=Metaspace Manager,type=MemoryManager":
            - "ObjectName"
        "java.lang:name=Metaspace,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Eden Space,type=MemoryPool":
            - "ObjectName"
        "java.lang:name=CodeHeap 'profiled nmethods',type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Copy,type=GarbageCollector":
            - "ObjectName"
        "java.lang:name=MarkSweepCompact,type=GarbageCollector":
            - "ObjectName"
        "java.nio:name=mapped - 'non-volatile memory',type=BufferPool":
            - "ObjectName"
        "java.lang:name=CodeHeap 'non-nmethods',type=MemoryPool":
            - "ObjectName"
        "java.lang:name=Compressed Class Space,type=MemoryPool":
            - "ObjectName"
        "java.nio:name=mapped,type=BufferPool":
            - "ObjectName"
        "java.lang:type=Memory":
            - "ObjectName"
        "java.lang:name=CodeHeap 'non-profiled nmethods',type=MemoryPool":
            - "ObjectName"
        "com.sun.management:type=HotSpotDiagnostic":
            - "ObjectName"
        "java.lang:type=Runtime":
            - "ObjectName"
            - "ClassPath"
            - "SystemProperties"
    rules:
    - pattern : kafka.network<type=RequestMetrics, name=LocalTimeMs, request=Fetch><>Count

Also tried wild card pattern :

https://github.com/prometheus/jmx_exporter/blob/f96796d5b242ffa9b0ec53e30091429901c8c52a/integration_test_suite/integration_tests/src/test/resources/io/prometheus/jmx/test/ExcludeObjectNamesTest/JavaAgent/exporter.yaml

excludeObjectNames: ["java.lang:*"]
rules:
  - pattern: ".*"

All of the above didn't work.

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

Could you please help me here @dhoard ? Thanks :)

sreelu27 · 2024-02-08T14:00:01Z

HI @dhoard

Can you please help me with the query #908 (comment)

Thanks,
Sree

dhoard · 2024-02-08T14:16:27Z

@sreelu27 please upgrade to the latest version.

sreelu27 · 2024-02-08T14:18:46Z

Hi @dhoard ,

I am using 0.20.0 version. Uplifted from 0.18.0 to 0.20.0 for configuration excludeObjectNames

Thanks,
Sree

sreelu27 · 2024-02-12T10:06:04Z

Hi @dhoard,

Could you please help me with the query #908 (comment)
I am using 0.20.0 jmx exporter version

Thanks,
Sree

dhoard · 2024-02-12T14:39:14Z

@sreelu27 excludeObjectNames is a global setting for an MBean. If you exclude the MBean there is no need to exclude specific attributes.

excludeObjectNames: ["java.lang:*"]

Should exclude the MBean.

sreelu27 · 2024-02-12T17:13:10Z

Hi @dhoard,

I have tried the above, but it didn't work as mentioned in my previous comment?

sreelu27 · 2024-02-16T10:04:59Z

Hi @dhoard,

Could you please help me with the above query #908 (comment)

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

sreelu27 · 2024-02-20T16:16:51Z

Hi @dhoard,

Could you please help me with the above query #908 (comment)

Is the MBean, that I am using is incorrect ? or is it that we cant exclude these default metrics (jvm & jmx) ?

dhoard · 2024-02-20T22:12:18Z

@sreelu27 apologies, I've been extremely busy with work.

After doing some testing and research, it appears what you are trying to filter is not supported. I should have caught it earlier.

#696

sreelu27 · 2024-02-21T10:59:20Z

Hi @dhoard,

Thank you for the confirmation.Will this be supported in the future version, the enhancement that you have linked in the previous comment : #696

Thanks,
Sree

dhoard · 2024-02-22T13:59:22Z

@sreelu27 Currently I'm are working on migration to the new Prometheus Java client_library.

I feel filtering JVM metrics is low priority since most uses/use cases I have seen want to monitor these metrics.

dhoard · 2024-02-22T19:28:36Z

Working as designed. Filtering of JVM metrics is an enhancement tracked via #696

github-actions bot added the status: new label Jan 24, 2024

dhoard added status: waiting on feedback and removed status: new labels Jan 26, 2024

dhoard added status: working as designed theme: configuration and removed status: waiting on feedback labels Feb 22, 2024

dhoard closed this as completed Feb 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for JMX filter pattern for kafka metrics #908

Support for JMX filter pattern for kafka metrics #908

sreelu27 commented Jan 24, 2024

sreelu27 commented Jan 25, 2024

dhoard commented Jan 26, 2024

sreelu27 commented Jan 30, 2024

sreelu27 commented Feb 1, 2024

sreelu27 commented Feb 2, 2024

sreelu27 commented Feb 2, 2024 •

edited

Loading

dhoard commented Feb 2, 2024

sreelu27 commented Feb 2, 2024 •

edited

Loading

dhoard commented Feb 2, 2024

sreelu27 commented Feb 2, 2024

dhoard commented Feb 2, 2024

sreelu27 commented Feb 6, 2024 •

edited

Loading

sreelu27 commented Feb 8, 2024

dhoard commented Feb 8, 2024

sreelu27 commented Feb 8, 2024 •

edited

Loading

sreelu27 commented Feb 12, 2024

dhoard commented Feb 12, 2024

sreelu27 commented Feb 12, 2024

sreelu27 commented Feb 16, 2024

sreelu27 commented Feb 20, 2024

dhoard commented Feb 20, 2024

sreelu27 commented Feb 21, 2024

dhoard commented Feb 22, 2024

dhoard commented Feb 22, 2024

Support for JMX filter pattern for kafka metrics #908

Support for JMX filter pattern for kafka metrics #908

Comments

sreelu27 commented Jan 24, 2024

sreelu27 commented Jan 25, 2024

dhoard commented Jan 26, 2024

sreelu27 commented Jan 30, 2024

HELP jvm_threads_state Current count of threads by state

TYPE jvm_threads_state gauge

HELP kafka_network_RequestMetrics_Count Attribute exposed for management kafka.network:name=LocalTimeMs,type=RequestMetrics,attribute=Count

TYPE kafka_network_RequestMetrics_Count untyped

TYPE jmx_scrape_duration_seconds gauge

sreelu27 commented Feb 1, 2024

sreelu27 commented Feb 2, 2024

sreelu27 commented Feb 2, 2024 • edited Loading

dhoard commented Feb 2, 2024

sreelu27 commented Feb 2, 2024 • edited Loading

dhoard commented Feb 2, 2024

sreelu27 commented Feb 2, 2024

dhoard commented Feb 2, 2024

sreelu27 commented Feb 6, 2024 • edited Loading

sreelu27 commented Feb 8, 2024

dhoard commented Feb 8, 2024

sreelu27 commented Feb 8, 2024 • edited Loading

sreelu27 commented Feb 12, 2024

dhoard commented Feb 12, 2024

sreelu27 commented Feb 12, 2024

sreelu27 commented Feb 16, 2024

sreelu27 commented Feb 20, 2024

dhoard commented Feb 20, 2024

sreelu27 commented Feb 21, 2024

dhoard commented Feb 22, 2024

dhoard commented Feb 22, 2024

sreelu27 commented Feb 2, 2024 •

edited

Loading

sreelu27 commented Feb 2, 2024 •

edited

Loading

sreelu27 commented Feb 6, 2024 •

edited

Loading

sreelu27 commented Feb 8, 2024 •

edited

Loading