"SupportabilityMetrics - Suppressed Spans by " since version 2.2.0

[fede843]

Hi, we have a Java 17 app, auto instrumented using the agent. It uses Spring Boot 3.1/3.2. Until agent version 2.1.0 it was working just fine, no much tuning.
When tried updating to 2.2.0+ it started missing all traces, or at least most of the important ones that we care.
Funny enough we have another app in old Java 8, with an older Spring Boot and it just took it fine.

After some debugging I find this in logs (debug mode on):

satapp | [otel.javaagent 2024-06-19 15:06:33:769 +0000] [supportability_metrics_reporter] DEBUG io.opentelemetry.javaagent.shaded.instrumentation.api.internal.SupportabilityMetrics - Suppressed Spans by 'io.opentelemetry.tomcat-10.0' (SERVER) : 1

That gave a clue, my spans were being suppressed. Then changed the strategy by adding:

   -Dotel.instrumentation.experimental.span-suppression-strategy=none

and we started receiving the spans as before.

What happened in 2.2.0 that the default is not working anymore? I went trough the change list but cannot see any.

I am not sure how much this will increase the tracing volume in our backend and if it will be worth it. But in any case I'd like to understand cause of the issue and the implications of this workaround.

I appreciate any help on this matter.
Thanks!

[trask]

it sounds like a bug, can you confirm the issue still exists in 2.5.0, and if so, create a small repro and open an issue for it?

[fede843]

I can confirm it is working in 2.1.0 and not working in 2.2.0, 2.3.0, 2.4.0 nor 2.5.0.

[trask]

What happened in 2.2.0 that the default is not working anymore? I went through the change list but cannot see any.

yeah, I looked at the change log and it's not clear to me the cause either, can you bisect the commits to find out? https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/CONTRIBUTING.md#building-from-source

[fede843]

I've never done it but I can try.
It's is a long holiday weekend ahead here. I'll give it a try on Monday. Thanks !

[fede843]

OK, I managed to compile and create the jar for the current version. But I am struggling to build at version 2.2.0 checkout.

> Task :custom-checks:compileJava FAILED
/home/neo/Proyectos/opentelemetry-java-instrumentation/custom-checks/src/main/java/io/opentelemetry/javaagent/customchecks/OtelPrivateConstructorForUtilityClass.java:24: error: cannot access Symbol
public class OtelPrivateConstructorForUtilityClass extends BugChecker
       ^
  class file for com.sun.tools.javac.code.Symbol not found
1 error

[laurit]

Try building with jdk17 if you were using jdk21.

[fede843]

Yes I did try Java 17 for the check out version. Gave me that error though. Maybe I need to clear some cache or gradle related.

[NicolasBissig]

I encountered a similar issue and saw the same debug message about the suppressed span on agent version 2.5.0

When I set OTEL_INSTRUMENTATION_EXPERIMENTAL_SPAN_SUPPRESSION_STRATEGY=none the HTTP span was there again, but not as a root span, and the root span was missing.
There also was this warning logged:

[otel.javaagent 2024-06-28 15:20:53:452 +0200] [http-nio-8081-exec-1] WARN io.opentelemetry.javaagent.shaded.instrumentation.api.internal.ContextPropagationDebug - Unexpected non-root current context found when extracting remote context!
[otel.javaagent 2024-06-28 15:20:53:453 +0200] [http-nio-8081-exec-1] WARN io.opentelemetry.javaagent.shaded.instrumentation.api.internal.ContextPropagationDebug - It contains this span: SdkSpan{traceId=caab354b087eef062bbc75fe68a5a7f4, spanId=4b2c8cf2a7cf7514, parentSpanContext=ImmutableSpanContext{traceId=00000000000000000000000000000000, spanId=0000000000000000, traceFlags=00, traceState=ArrayBasedTraceState{entries=[]}, remote=false, valid=false}, name=PUT, kind=SERVER, attributes=AttributesMap{data={url.scheme=http, thread.name=http-nio-8081-exec-1, server.address=localhost, client.address=127.0.0.1, user_agent.original=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36, url.path=[REDACTED], server.port=8081, http.request.method=PUT, thread.id=50}, capacity=128, totalAddedValues=9}, status=ImmutableStatusData{statusCode=UNSET, description=}, totalRecordedEvents=0, totalRecordedLinks=0, startEpochNanos=1719580853446606411, endEpochNanos=0}

[NicolasBissig]

@laurit I was not able to find a way to replicate the issue yet. Could this be some kind of dependency collision? Can I provide other information? Thanks :)

[NicolasBissig]

@laurit I invested more time into the issue and gradually removed all dependencies in one of the services, and found that one dependency in particular causes the spans to go missing in our case. However, the service and the dependency are both internal and cannot be shared. I can however share the mvn dependency:tree of the dependency:

[INFO] +- [[REDACTED]]
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.15.4:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.15.4:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.15.4:compile
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.4:compile
[INFO] |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.4:compile
[INFO] |  +- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.15.4:compile
[INFO] |  +- [[REDACTED]]
[INFO] |  |  +- org.slf4j:slf4j-api:jar:2.0.13:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-guava:jar:2.15.4:compile
[INFO] |  |  \- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- [[REDACTED]]
[INFO] |  +- io.github.openfeign:feign-jackson:jar:13.2.1:compile
[INFO] |  |  \- io.github.openfeign:feign-core:jar:13.2.1:compile
[INFO] |  +- io.github.openfeign:feign-slf4j:jar:13.2.1:compile
[INFO] |  +- io.github.openfeign:feign-okhttp:jar:13.2.1:compile
[INFO] |  |  \- com.squareup.okhttp3:okhttp:jar:4.12.0:compile
[INFO] |  |     +- com.squareup.okio:okio:jar:3.6.0:compile
[INFO] |  |     |  \- com.squareup.okio:okio-jvm:jar:3.6.0:compile
[INFO] |  |     |     \- org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.9.23:compile
[INFO] |  |     \- org.jetbrains.kotlin:kotlin-stdlib-jdk8:jar:1.9.23:compile
[INFO] |  |        +- org.jetbrains.kotlin:kotlin-stdlib:jar:1.9.23:compile
[INFO] |  |        |  \- org.jetbrains:annotations:jar:13.0:compile
[INFO] |  |        \- org.jetbrains.kotlin:kotlin-stdlib-jdk7:jar:1.9.23:compile
[INFO] |  +- com.auth0:java-jwt:jar:4.4.0:compile
[INFO] |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  \- [[REDACTED]]
[INFO] |     +- [[REDACTED]]
[INFO] |     |  +- org.springframework.boot:spring-boot-actuator:jar:3.2.5:compile
[INFO] |     |  +- org.jetbrains.kotlin:kotlin-reflect:jar:1.9.23:compile
[INFO] |     |  +- javax.servlet:javax.servlet-api:jar:4.0.1:compile
[INFO] |     |  \- io.micrometer:micrometer-core:jar:1.12.5:compile
[INFO] |     |     +- io.micrometer:micrometer-commons:jar:1.12.5:compile
[INFO] |     |     +- io.micrometer:micrometer-observation:jar:1.12.5:compile
[INFO] |     |     +- org.hdrhistogram:HdrHistogram:jar:2.1.12:runtime
[INFO] |     |     \- org.latencyutils:LatencyUtils:jar:2.0.3:runtime
[INFO] |     \- org.springframework.boot:spring-boot-autoconfigure:jar:3.2.5:compile
[INFO] |        \- org.springframework.boot:spring-boot:jar:3.2.5:compile
[INFO] |           +- org.springframework:spring-core:jar:6.1.6:compile
[INFO] |           |  \- org.springframework:spring-jcl:jar:6.1.6:compile
[INFO] |           \- org.springframework:spring-context:jar:6.1.6:compile
[INFO] |              +- org.springframework:spring-aop:jar:6.1.6:compile
[INFO] |              +- org.springframework:spring-beans:jar:6.1.6:compile
[INFO] |              \- org.springframework:spring-expression:jar:6.1.6:compile
[INFO] \- org.projectlombok:lombok:jar:1.18.32:compile

Is there a dependency in there which is known to cause issues?
What is to be noticed:

• The dependency is an API client that utilizes openfeign
• It is written in Kotlin, while the service itself is written in Java
• it brings with it the actuator and micrometer

Is this helpful?

[NicolasBissig]

OK after some more testing, I found the dependency in the imported dependency that caused this. javax.servlet:javax.servlet-api

I added it as an exclusion when declaring the dependency, and now all our HTTP spans are back. (The guava exclusion was there before, but left it in the snippet for completeness)

<dependency>
	<groupId>[[REDACTED]]</groupId>
	<artifactId>[[REDACTED]]</artifactId>
	<version>[[REDACTED]]</version>
	<exclusions>
		<exclusion>
			<groupId>com.google.guava</groupId>
			<artifactId>guava</artifactId>
		</exclusion>
		<exclusion>
			<!-- This dependency caused all HTTP spans to not get created -->
			<groupId>javax.servlet</groupId>
			<artifactId>javax.servlet-api</artifactId>
		</exclusion>
	</exclusions>
</dependency>

[laurit]

without reproducing it is hard to tell what exactly causes this

[NicolasBissig]

@laurit got it replicated: https://github.com/NicolasBissig/otel-javaagent-replicate-11636

[fede843]

OK, I have just tried version 2.7.0 and it seems fixed, at least in my case. 2.6.0 was still broken. Anyone else?