The jvm_threads_daemon_threads indicator will gradually become negative

[liuxuzxx]

We introduced pyroscope.jar as an agent. After the process worked for about 20 minutes, we found that the jvm_threads_daemon_thread indicator obtained from spring-boot-actuator gradually changed from positive to negative. When we removed the pyroscope.jar agent, we found that it was normal.

[liuxuzxx]

The first three monitoring data in this figure are configured with pyroscope.jar as the agent. It can be seen that the jvm_threads_daemon_thread indicator shows a linear function property, gradually decreasing to a negative number. The last book removes pyroscope.jar as the agent monitoring, and it is normal.

[liuxuzxx]

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cpaas-msg-hub-deployment
spec:
  template:
    spec:
      containers:
        - env:
            - name: JAVA_TOOL_OPTIONS
              value: >-
                ......
                -javaagent:/skywalking/agent/skywalking-agent.jar
                -javaagent:/shared-data/pyroscope/pyroscope.jar #use -javaagent
                -Dcom.rocketmq.remoting.client.worker.size="10"
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: SW_AGENT_NAME
              value: cpaas-msg-hub
            - name: PYROSCOPE_AGENT_ENABLED
              value: 'true'
            - name: PYROSCOPE_APPLICATION_NAME
              value: cpaas-msg-hub
            - name: PYROSCOPE_PROFILING_INTERVAL
              value: 30ms
            - name: PYROSCOPE_FORMAT
              value: jfr
            - name: PYROSCOPE_PROFILER_EVENT
              value: itimer
            - name: PYROSCOPE_PROFILER_ALLOC
              value: 32k
            - name: PYROSCOPE_PROFILER_LOCK
              value: 10ms
            - name: PYROSCOPE_SERVER_ADDRESS
              value: 'http://172.16.15.161:30424'
            - name: PYROSCOPE_LABELS
              value: 'env=test,zone=cloud,podname=$(POD_NAME)'
          image: 'xxxxxx/cpaas-test/cpaas-msg-hub:v1.8.3-8eecaaff295'
          name: cpaas-msg-hub
      initContainers:
        - command:
            - sh
            - '-c'
            - cp /opt/pyroscope.jar /shared-data/pyroscope/
          image: 'xwharbor.wxchina.com/cpaas/pyroscope:v2.1.2'
          name: pyroscope
          volumeMounts:
            - mountPath: /shared-data/pyroscope
              name: pyroscope
      volumes:
        - emptyDir: {}
          name: pyroscope

[korniltsev]

Can you check if it happens with just using async-profiler https://github.com/async-profiler/async-profiler ? Just to make sure the issue is in the pyroscope jar and not the async-profiler itself.

[liuxuzxx]

OK, let me try it out and see if there is any problem when using only async-profile

[liuxuzxx]

When using pyroscope.jar as the agent, will the memory usage of the Java process increase compared to not using pyroscope.jar? Currently, it has been verified that under the current configuration, the performance is basically not affected.

[liuxuzxx]

My environment is

openjdk version "1.8.0_342"
OpenJDK Runtime Environment (build 1.8.0_342-b07)
OpenJDK 64-Bit Server VM (build 25.342-b07, mixed mode)

[liuxuzxx]

I see the requirements for async-profiler are as follows:

make
GCC 7.5.0+ or Clang 7.0.0+
Static version of libstdc++ (e.g. on Amazon Linux 2023: yum install libstdc++-static)
JDK 11+

Is JDK8 not working?

@korniltsev

[korniltsev]

It requires jdk11 to build the profiler.

It should work on jdk8 if you already have binaries: https://github.com/async-profiler/async-profiler/blob/981619680ea6c93768df6253e3a6d87d4c33cd2b/.github/workflows/test-and-publish-nightly.yml#L108

[liuxuzxx]

It requires jdk11 to build the profiler.

It should work on jdk8 if you already have binaries: https://github.com/async-profiler/async-profiler/blob/981619680ea6c93768df6253e3a6d87d4c33cd2b/.github/workflows/test-and-publish-nightly.yml#L108

OK, thanks

[liuxuzxx]

We have a module based on JDK21. After following the same configuration, we found that the jvm_threads_daemon_threads indicator is normal, but a little more than before.

From the monitoring chart above, we can see that the jvm_threads_daemon_threads indicator has increased by about 3 daemon threads before and after using pyroscope.jar. It may be that pyroscope.jar needs some threads to report, so it is understandable that daemon is set.

[liuxuzxx]

JDK21 is OK!

[liuxuzxx]

My java process id is: 1448183. When I use async-profiler./asprof start 1448183, and then wait for 10 minutes, execute./asprof stop 1448183. During this period, I see that the jvm_threads_daemon_threads monitoring data has not changed at all, but has changed from 28 to 29. The extra threads should be other monitoring data acquisition. However, when I use pyroscope.jar (version 2.1.2) as javaagent on the vm, I see that it will decrease by about 3-4 in 1 minute. However, I have tested the jdk21 version and it is normal.

@korniltsev

[liuxuzxx]

The statistics of other threads are normal, and there is no difference with not adding the pyroscope.jar javaagent.

[liuxuzxx]

My java environment:

liuxu@liuxu-B560M-AORUS-ELITE /media/liuxu/data/code/github > java -version

openjdk version "1.8.0_452"
OpenJDK Runtime Environment (build 1.8.0_452-8u452-ga~us1-0ubuntu1~24.04-b09)
OpenJDK 64-Bit Server VM (build 25.452-b09, mixed mode)

[liuxuzxx]

I used the jstack command to obtain the thread stack of the corresponding Java process at intervals of 1 minute, and found that the daemon thread did not change, but the indicator exposed by spring-boot-starter-actuator: jvm_threads_daemon_threads was gradually decreasing.