When Opentelemetry performs dynamic injection (using the agentmain method entry point), the injection reports a java.lang.VerifyError

[shenlvcheng]

Describe the bug

When our project starts up, we found that it's not generating trace metrics data. After adding the -Dotel.javaagent.debug=true parameter, we discovered this issue.

[otel.javaagent 2024-12-12 09:39:34:954 +0800] [Attach Listener] DEBUG io.opentelemetry.javaagent.tooling.AgentInstaller$RedefinitionLoggingListener - Exception while retransforming 76 classes: [class org.springframework.core.task.SimpleAsyncTaskExecutor, class org.springframework.beans.factory.support.DisposableBeanAdapter, class org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter, class org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter, class org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter, class org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter, class org.springframework.context.support.AbstractApplicationContext$1, class org.springframework.context.support.ContextTypeMatchClassLoader, class org.springframework.core.DecoratingClassLoader, class org.springframework.web.context.support.XmlWebApplicationContext, class org.springframework.web.context.support.AbstractRefreshableWebApplicationContext, class ch.qos.logback.classic.spi.LoggingEvent, class ch.qos.logback.classic.Logger, interface org.springframework.web.servlet.HandlerAdapter, class org.springframework.web.servlet.DispatcherServlet, class org.springframework.web.servlet.FrameworkServlet, class org.springframework.web.servlet.HttpServletBean, class org.apache.catalina.connector.CoyoteOutputStream, interface javax.servlet.AsyncContext, class org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState, class org.apache.tomcat.util.net.SocketWrapperBase$OperationState, class org.apache.tomcat.util.net.Acceptor, class org.apache.tomcat.util.threads.TaskThread$WrappingRunnable, class org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker, class org.apache.tomcat.util.threads.ThreadPoolExecutor, class org.apache.tomcat.util.threads.TaskThread, class org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor, class org.apache.tomcat.websocket.server.WsFilter, interface javax.servlet.Filter, class org.apache.jasper.servlet.JspServlet, class org.apache.catalina.servlets.DefaultServlet, class javax.servlet.http.HttpServlet, class javax.servlet.GenericServlet, class org.apache.catalina.loader.ParallelWebappClassLoader, class org.apache.catalina.loader.WebappClassLoaderBase, interface javax.servlet.Servlet, class org.apache.catalina.startup.HostConfig$DeployWar, class javax.servlet.ServletOutputStream, class org.apache.catalina.core.ContainerBase$StartChild, class org.apache.catalina.connector.CoyoteAdapter, class org.apache.tomcat.util.net.NioEndpoint$Poller, class org.apache.coyote.AbstractProtocol$AsyncTimeout, class org.apache.catalina.connector.Response, interface javax.servlet.http.HttpServletResponse, class org.apache.catalina.connector.Request, interface javax.servlet.http.HttpServletRequest, interface javax.servlet.ServletRequest, class org.apache.tomcat.util.net.NioEndpoint$SocketProcessor, class org.apache.tomcat.util.net.SocketProcessorBase, class org.apache.catalina.startup.Catalina$CatalinaShutdownHook, class org.apache.juli.AsyncFileHandler$1, class org.apache.juli.ClassLoaderLogManager$Cleaner, class org.apache.juli.ClassLoaderLogManager$RootLogger, class java.util.concurrent.CompletableFuture, class java.util.concurrent.ForkJoinTask, class java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask, interface java.util.concurrent.RunnableScheduledFuture, class java.util.concurrent.ScheduledThreadPoolExecutor, class java.util.concurrent.Executors$RunnableAdapter, class java.util.concurrent.FutureTask, interface java.util.concurrent.RunnableFuture, class java.lang.invoke.InnerClassLambdaMetafactory, class java.util.concurrent.ThreadPoolExecutor$Worker, interface java.util.concurrent.Callable, class java.net.HttpURLConnection, class java.util.logging.LogManager$RootLogger, class java.util.concurrent.ThreadPoolExecutor, class java.util.concurrent.Executors$FinalizableDelegatedExecutorService, class java.util.concurrent.Executors$DelegatedExecutorService, class java.util.concurrent.AbstractExecutorService, class java.util.logging.Logger, class sun.reflect.Reflection, class sun.misc.Launcher$AppClassLoader, class java.net.URLClassLoader, class java.lang.ClassLoader, class java.lang.Class]
java.lang.VerifyError
at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.bytebuddy.utility.Invoker$Dispatcher.invoke(Unknown Source)
at net.bytebuddy.utility.dispatcher.JavaDispatcher$Dispatcher$ForNonStaticMethod.invoke(JavaDispatcher.java:1033)
at net.bytebuddy.utility.dispatcher.JavaDispatcher$ProxiedInvocationHandler.invoke(JavaDispatcher.java:1163)
at net.bytebuddy.agent.builder.$Proxy28.retransformClasses(Unknown Source)
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector$ForRetransformation.doApply(AgentBuilder.java:8611)
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy$Collector.apply(AgentBuilder.java:8426)
at net.bytebuddy.agent.builder.AgentBuilder$RedefinitionStrategy.apply(AgentBuilder.java:6123)
at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11737)
at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11637)
at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:202)
at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:109)
at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:59)
at io.opentelemetry.javaagent.OpenTelemetryAgent.agentmain(OpenTelemetryAgent.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)

Steps to reproduce

No

Expected behavior

The data can be exported

Actual behavior

The trace data cannot be exported

Javaagent or library instrumentation version

2.10.0

Environment

JDK:1.8.
OS: 1.8.0_421
spring framework:4.0.0.RELEASE

Additional context

[
spring-mvc-1.6.zip
](url)

test demo

[steverao]

According to your demo, it seems everything is fine in my environment. It can create spans and print them out.

[shenlvcheng]

According to your demo, it seems everything is fine in my environment. It can create spans and print them out.

Are you using the -javagent method? Don't use this approach. I just tested it again and it won't export the log data. You should use the dynamic approach instead:
VirtualMachine vm = VirtualMachine.attach(pid);
vm.loadAgent(javaagentPath);

[steverao]

You should use the dynamic approach instead: VirtualMachine vm = VirtualMachine.attach(pid); vm.loadAgent(javaagentPath);

As far as I know, we don't support it so far.

[shenlvcheng]

You should use the dynamic approach instead: VirtualMachine vm = VirtualMachine.attach(pid); vm.loadAgent(javaagentPath);

As far as I know, we don't support it so far.

agentmain is the entry point for attach injection, while premain is the entry point for -javaagent injection. The source code clearly provides this entry point, indicating that attach injection is supported

[steverao]

agentmain is the entry point for attach injection, while premain is the entry point for -javaagent injection. The source code clearly provides this entry point, indicating that attach injection is supported

We don't support or test it officially, you can refer to previous discussion #2198.
Hi @trask, is it necessary to add some comments on agentmain?

[laurit]

@shenlvcheng if you wish for someone to take a look then please provide a minimal application that reproduces the issue, the sample app that you provided does not include the dynamic attach code as far as I can tell, along with any necessary instructions. Generally our advice is that you perform the dynamic attach as early as possible, from the stack trace you provided it looks like you are not doing this. We are open to reasonable PRs that fix issues when agent is attached later, but we can not commit to fixing issues when attach is not called immediately when the application starts.