[BUG] Deadlock / BLOCKED threads starting tomcat with dd-agent

[karlhenselin]

Full thread dump OpenJDK 64-Bit Server VM (21.0.4+7-LTS mixed mode, sharing):

Threads class SMR info:
_java_thread_list=0x00007f72a4c28530, length=13, elements={
0x00007f72a402f9f0, 0x00007f72a42bf910, 0x00007f72a42c0f60, 0x00007f72a42c2a40,
0x00007f72a42c4080, 0x00007f72a42c5620, 0x00007f72a42c7140, 0x00007f72a42c8840,
0x00007f72a42e24c0, 0x00007f72a4986fc0, 0x00007f72a4bfd540, 0x00007f72a4bfeb10,
0x00007f72a4c27990
}

"main" #1 [2034618] prio=5 os_prio=0 cpu=2320.56ms elapsed=16567.14s tid=0x00007f72a402f9f0 nid=2034618 waiting for monitor entry [0x00007f72adedd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:573)
- waiting to lock <0x000000073dc217b0> (a java.lang.Object)
at datadog.trace.bootstrap.DatadogClassLoader.loadClass(DatadogClassLoader.java:114)
at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
at com.datadog.debugger.util.ClassFileHelper.extractSourceFile(ClassFileHelper.java:11)
at com.datadog.debugger.agent.SourceFileTrackingTransformer.transform(SourceFileTrackingTransformer.java:34)
at java.lang.instrument.ClassFileTransformer.transform([email protected]/ClassFileTransformer.java:244)
at sun.instrument.TransformerManager.transform([email protected]/TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform([email protected]/InstrumentationImpl.java:610)
at java.lang.ClassLoader.findBootstrapClass([email protected]/Native Method)
at java.lang.ClassLoader.findBootstrapClassOrNull([email protected]/ClassLoader.java:1277)
at java.lang.System$2.findBootstrapClassOrNull([email protected]/System.java:2397)
at jdk.internal.loader.ClassLoaders$BootClassLoader.loadClassOrNull([email protected]/ClassLoaders.java:140)
at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:700)
at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull([email protected]/BuiltinClassLoader.java:676)
- locked <0x000000073df9f028> (a java.lang.Object)
at jdk.internal.loader.BuiltinClassLoader.loadClass([email protected]/BuiltinClassLoader.java:639)
at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:580)
- locked <0x000000073df9eff8> (a java.lang.Object)
at datadog.trace.bootstrap.DatadogClassLoader.loadClass(DatadogClassLoader.java:114)
at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
at com.datadog.debugger.agent.DebuggerAgent.run(DebuggerAgent.java:62)
- locked <0x000000073df81ba0> (a java.lang.Class for com.datadog.debugger.agent.DebuggerAgent)
at java.lang.invoke.LambdaForm$DMH/0x00007f722c001800.invokeStatic([email protected]/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00007f722c002c00.invoke([email protected]/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f722c003000.invokeExact_MT([email protected]/LambdaForm$MH)
at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl([email protected]/DirectMethodHandleAccessor.java:155)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke([email protected]/DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke([email protected]/Method.java:580)
at datadog.trace.bootstrap.Agent.startDebuggerAgent(Agent.java:1121)
- locked <0x000000073cc9a988> (a java.lang.Class for datadog.trace.bootstrap.Agent)
at datadog.trace.bootstrap.Agent.maybeStartDebugger(Agent.java:1107)
at datadog.trace.bootstrap.Agent.access$900(Agent.java:67)
at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:541)
at datadog.trace.bootstrap.Agent.start(Agent.java:355)
at java.lang.invoke.DirectMethodHandle$Holder.invokeStatic([email protected]/DirectMethodHandle$Holder)
at java.lang.invoke.LambdaForm$MH/0x00007f722c006c00.invoke([email protected]/LambdaForm$MH)
at java.lang.invoke.Invokers$Holder.invokeExact_MT([email protected]/Invokers$Holder)
at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl([email protected]/DirectMethodHandleAccessor.java:157)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke([email protected]/DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke([email protected]/Method.java:580)
at datadog.trace.bootstrap.AgentBootstrap.agentmainImpl(AgentBootstrap.java:155)
at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:72)
at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:60)
at java.lang.invoke.LambdaForm$DMH/0x00007f722c001800.invokeStatic([email protected]/LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/0x00007f722c002c00.invoke([email protected]/LambdaForm$MH)
at java.lang.invoke.LambdaForm$MH/0x00007f722c003000.invokeExact_MT([email protected]/LambdaForm$MH)
at jdk.internal.reflect.DirectMethodHandleAccessor.invokeImpl([email protected]/DirectMethodHandleAccessor.java:155)
at jdk.internal.reflect.DirectMethodHandleAccessor.invoke([email protected]/DirectMethodHandleAccessor.java:103)
at java.lang.reflect.Method.invoke([email protected]/Method.java:580)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent([email protected]/InstrumentationImpl.java:560)
at sun.instrument.InstrumentationImpl.loadClassAndCallPremain([email protected]/InstrumentationImpl.java:572)

"Reference Handler" #9 [2034626] daemon prio=10 os_prio=0 cpu=1.13ms elapsed=16567.11s tid=0x00007f72a42bf910 nid=2034626 waiting on condition [0x00007f72a92ea000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList([email protected]/Native Method)
at java.lang.ref.Reference.processPendingReferences([email protected]/Reference.java:246)
at java.lang.ref.Reference$ReferenceHandler.run([email protected]/Reference.java:208)

"Finalizer" #10 [2034627] daemon prio=8 os_prio=0 cpu=0.20ms elapsed=16567.11s tid=0x00007f72a42c0f60 nid=2034627 in Object.wait() [0x00007f72a91e9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait0([email protected]/Native Method)
- waiting on <0x000000073cc080f0> (a java.lang.ref.NativeReferenceQueue$Lock)
at java.lang.Object.wait([email protected]/Object.java:366)
at java.lang.Object.wait([email protected]/Object.java:339)
at java.lang.ref.NativeReferenceQueue.await([email protected]/NativeReferenceQueue.java:48)
at java.lang.ref.ReferenceQueue.remove0([email protected]/ReferenceQueue.java:158)
at java.lang.ref.NativeReferenceQueue.remove([email protected]/NativeReferenceQueue.java:89)
- locked <0x000000073cc080f0> (a java.lang.ref.NativeReferenceQueue$Lock)
at java.lang.ref.Finalizer$FinalizerThread.run([email protected]/Finalizer.java:173)

"Signal Dispatcher" #11 [2034628] daemon prio=9 os_prio=0 cpu=0.18ms elapsed=16567.11s tid=0x00007f72a42c2a40 nid=2034628 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Service Thread" #12 [2034629] daemon prio=9 os_prio=0 cpu=0.27ms elapsed=16567.11s tid=0x00007f72a42c4080 nid=2034629 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Monitor Deflation Thread" #13 [2034630] daemon prio=9 os_prio=0 cpu=1930.60ms elapsed=16567.11s tid=0x00007f72a42c5620 nid=2034630 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #14 [2034631] daemon prio=9 os_prio=0 cpu=1394.51ms elapsed=16567.11s tid=0x00007f72a42c7140 nid=2034631 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task

"C1 CompilerThread0" #15 [2034632] daemon prio=9 os_prio=0 cpu=821.83ms elapsed=16567.10s tid=0x00007f72a42c8840 nid=2034632 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task

"Common-Cleaner" #16 [2034633] daemon prio=8 os_prio=0 cpu=29.55ms elapsed=16567.08s tid=0x00007f72a42e24c0 nid=2034633 waiting on condition [0x00007f72a8b13000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x000000073cc08418> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:1847)
at java.lang.ref.ReferenceQueue.await([email protected]/ReferenceQueue.java:71)
at java.lang.ref.ReferenceQueue.remove0([email protected]/ReferenceQueue.java:143)
at java.lang.ref.ReferenceQueue.remove([email protected]/ReferenceQueue.java:218)
at jdk.internal.ref.CleanerImpl.run([email protected]/CleanerImpl.java:140)
at java.lang.Thread.runWith([email protected]/Thread.java:1596)
at java.lang.Thread.run([email protected]/Thread.java:1583)
at jdk.internal.misc.InnocuousThread.run([email protected]/InnocuousThread.java:186)

"dd-task-scheduler" #17 [2034635] daemon prio=5 os_prio=0 cpu=149.84ms elapsed=16564.67s tid=0x00007f72a4986fc0 nid=2034635 waiting on condition [0x00007f72a880e000]
java.lang.Thread.State: RUNNABLE
at com.datadog.debugger.util.ClassFileHelper.extractSourceFile(ClassFileHelper.java:11)
at com.datadog.debugger.agent.SourceFileTrackingTransformer.transform(SourceFileTrackingTransformer.java:34)
at java.lang.instrument.ClassFileTransformer.transform([email protected]/ClassFileTransformer.java:244)
at sun.instrument.TransformerManager.transform([email protected]/TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform([email protected]/InstrumentationImpl.java:610)
at java.lang.ClassLoader.defineClass1([email protected]/Native Method)
at java.lang.ClassLoader.defineClass([email protected]/ClassLoader.java:1027)
at java.security.SecureClassLoader.defineClass([email protected]/SecureClassLoader.java:150)
at datadog.trace.bootstrap.DatadogClassLoader.findClass(DatadogClassLoader.java:135)
at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:593)
- locked <0x000000073dc217b0> (a java.lang.Object)
at datadog.trace.bootstrap.DatadogClassLoader.loadClass(DatadogClassLoader.java:114)
at java.lang.ClassLoader.loadClass([email protected]/ClassLoader.java:526)
at jnr.ffi.provider.jffi.AsmLibraryLoader.generateInterfaceImpl(AsmLibraryLoader.java:97)
at jnr.ffi.provider.jffi.AsmLibraryLoader.loadLibrary(AsmLibraryLoader.java:87)
at jnr.ffi.provider.jffi.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:44)
at jnr.ffi.LibraryLoader.load(LibraryLoader.java:420)
at jnr.unixsocket.Native.(Native.java:80)
at jnr.unixsocket.UnixSocketChannel.(UnixSocketChannel.java:101)
at jnr.unixsocket.UnixSocketChannel.open(UnixSocketChannel.java:60)
at datadog.common.socket.UnixDomainSocketFactory.createSocket(UnixDomainSocketFactory.java:38)
at datadog.okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:241)
at datadog.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
at datadog.okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
at datadog.okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
at datadog.okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
at datadog.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at datadog.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at datadog.okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at datadog.okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at datadog.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at datadog.okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
at datadog.okhttp3.RealCall.execute(RealCall.java:93)
at datadog.communication.ddagent.DDAgentFeaturesDiscovery.doDiscovery(DDAgentFeaturesDiscovery.java:149)
at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:134)
- locked <0x000000073e842c70> (a datadog.communication.ddagent.DDAgentFeaturesDiscovery)
at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discover(DDAgentFeaturesDiscovery.java:118)
at datadog.communication.ddagent.SharedCommunicationObjects$$Lambda/0x00007f722c28a4a8.run(Unknown Source)
at datadog.trace.util.AgentTaskScheduler$RunnableTask.run(AgentTaskScheduler.java:41)
at datadog.trace.util.AgentTaskScheduler$RunnableTask.run(AgentTaskScheduler.java:36)
at datadog.trace.util.AgentTaskScheduler$PeriodicTask.run(AgentTaskScheduler.java:311)
at datadog.trace.util.AgentTaskScheduler$Worker.run(AgentTaskScheduler.java:266)
at java.lang.Thread.runWith([email protected]/Thread.java:1596)
at java.lang.Thread.run([email protected]/Thread.java:1583)

"dd-trace-monitor" #21 [2034637] daemon prio=5 os_prio=0 cpu=0.59ms elapsed=16564.18s tid=0x00007f72a4bfd540 nid=2034637 waiting on condition [0x00007f72a8613000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x000000073e969880> (a datadog.jctools.queues.MpscBlockingConsumerArrayQueue)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
at datadog.jctools.queues.MpscBlockingConsumerArrayQueue.parkUntilNext(MpscBlockingConsumerArrayQueue.java:536)
at datadog.jctools.queues.MpscBlockingConsumerArrayQueue.take(MpscBlockingConsumerArrayQueue.java:482)
at datadog.trace.agent.core.PendingTraceBuffer$DelayingPendingTraceBuffer$Worker.run(PendingTraceBuffer.java:180)
at java.lang.Thread.runWith([email protected]/Thread.java:1596)
at java.lang.Thread.run([email protected]/Thread.java:1583)

"dd-trace-processor" #20 [2034638] daemon prio=5 os_prio=0 cpu=6680.41ms elapsed=16564.17s tid=0x00007f72a4bfeb10 nid=2034638 waiting on condition [0x00007f72a8512000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x000000073e8d6dc8> (a datadog.jctools.queues.MpscBlockingConsumerArrayQueue)
at java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:269)
at datadog.jctools.queues.MpscBlockingConsumerArrayQueue.parkUntilNext(MpscBlockingConsumerArrayQueue.java:536)
at datadog.jctools.queues.MpscBlockingConsumerArrayQueue.poll(MpscBlockingConsumerArrayQueue.java:512)
at datadog.trace.agent.common.writer.TraceProcessingWorker$TraceSerializingHandler.consumeFromPrimaryQueue(TraceProcessingWorker.java:215)
at datadog.trace.agent.common.writer.TraceProcessingWorker$TraceSerializingHandler.runDutyCycle(TraceProcessingWorker.java:184)
at datadog.trace.agent.common.writer.TraceProcessingWorker$TraceSerializingHandler.run(TraceProcessingWorker.java:172)
at java.lang.Thread.runWith([email protected]/Thread.java:1596)
at java.lang.Thread.run([email protected]/Thread.java:1583)

"dd-data-streams-monitor" #22 [2034639] daemon prio=5 os_prio=0 cpu=461.37ms elapsed=16564.11s tid=0x00007f72a4c27990 nid=2034639 waiting for monitor entry [0x00007f72a8411000]
java.lang.Thread.State: BLOCKED (on object monitor)
at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:131)
- waiting to lock <0x000000073e842c70> (a datadog.communication.ddagent.DDAgentFeaturesDiscovery)
at datadog.communication.ddagent.DDAgentFeaturesDiscovery.discoverIfOutdated(DDAgentFeaturesDiscovery.java:123)
at datadog.trace.agent.core.datastreams.DefaultDataStreamsMonitoring$InboxProcessor.run(DefaultDataStreamsMonitoring.java:332)
at java.lang.Thread.runWith([email protected]/Thread.java:1596)
at java.lang.Thread.run([email protected]/Thread.java:1583)

"GC Thread#1" os_prio=0 cpu=9.86ms elapsed=16564.64s tid=0x00007f722000c050 nid=2034636 runnable

"VM Thread" os_prio=0 cpu=517.59ms elapsed=16567.12s tid=0x00007f72a42ae2f0 nid=2034625 runnable

"VM Periodic Task Thread" os_prio=0 cpu=4765.19ms elapsed=16567.12s tid=0x00007f72a4294d30 nid=2034624 runnable

"G1 Service" os_prio=0 cpu=615.22ms elapsed=16567.13s tid=0x00007f72a42833a0 nid=2034623 runnable

"G1 Refine#0" os_prio=0 cpu=0.09ms elapsed=16567.13s tid=0x00007f72a42823f0 nid=2034622 runnable

"G1 Conc#0" os_prio=0 cpu=0.08ms elapsed=16567.14s tid=0x00007f72a40aab40 nid=2034621 runnable

"G1 Main Marker" os_prio=0 cpu=0.14ms elapsed=16567.14s tid=0x00007f72a40a9ba0 nid=2034620 runnable

"GC Thread#0" os_prio=0 cpu=9.80ms elapsed=16567.14s tid=0x00007f72a40985e0 nid=2034619 runnable

JNI global refs: 43, weak refs: 0

Heap
garbage-first heap total 1050624K, used 44865K [0x0000000700000000, 0x0000000800000000)
region size 2048K, 21 young (43008K), 3 survivors (6144K)
Metaspace used 19912K, committed 20288K, reserved 1114112K
class space used 2921K, committed 3072K, reserved 1048576K

-->

Agent Environment
jdk-21.0.4p7

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[dd.trace 2025-02-07 08:31:58:024 -0600] [main] INFO datadog.common.socket.SocketUtils - Detected /var/run/datadog/apm.socket. Using it to send trace data.
[dd.trace 2025-02-07 08:31:59:130 -0600] [dd-task-scheduler] INFO datadog.trace.agent.core.StatusLogger - DATADOG TRACER CONFIGURATION {"version":"1.45.2~400277e123","os_name":"Linux","os_version":"5.4.17-2136.326.6.el8uek.x86_64","architecture":"amd64","lang":"jvm","lang_version":"21.0.4","jvm_vendor":"Microsoft","jvm_version":"21.0.4+7-LTS","java_class_version":"65.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service":"org.apache.catalina.startup.Bootstrap","agent_url":"http://localhost:8126","agent_error":false,"debug":false,"trace_propagation_style_extract":["datadog","tracecontext"],"trace_propagation_style_inject":["datadog","tracecontext"],"analytics_enabled":false,"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":false,"remote_config_enabled":true,"debugger_enabled":false,"debugger_exception_enabled":false,"debugger_span_origin_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","rasp_enabled":true,"telemetry_enabled":true,"telemetry_dependency_collection_enabled":true,"telemetry_log_collection_enabled":true,"dd_version":"","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"d1db1716-255d-4326-82f4-883f9dbabc4e","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":false,"datadog_profiler_safe":true,"datadog_profiler_enabled_overridden":false,"data_streams_enabled":false}

I ran with these java args passed in to catalina"-XX:-OmitStackTraceInFastThrow -javaagent:/u01/flexdeploy/datadog/dd-java-agent.jar -Ddd.profiling.enabled=false -XX:FlightRecorderOptions=stackdepth=256"

Describe what happened:
Installed a new war and restarted tomcat, and tomcat didn't start due to dd-agent BLOCKED threads that prevented startup.
Thread dump is above.

Describe what you expected:
Tomcat starts.

Steps to reproduce the issue:
Since it usually works, it isn't easy to reproduce.

Stopping and Starting tomcat resolved the issue.

Additional environment details (Operating System, Cloud provider, etc):
OEL 8 on Azure VM.