Minimum build example for Quarkus OTel-LGTM

[jfslin]

Trying to learn a bit about telemetry and logging, and poking around the Quarkus guides seems to indicate https://quarkus.io/guides/observability-devservices-lgtm is a good place to start. I tried to directly incorporate these instructions into my production project and wasn't able to get it working, so decided to start a minimum build on the side first, with hope that this might eventually be useful for quarkus-quickstarts. If following guides like https://quarkus.io/guides/opentelemetry is recommended instead, I will switch over to that.

My ultimate goal is to either use something like Grafana Cloud or Sentry to facilitate telemetry and logging. Not particularly attached to any given provider at the moment.

Current project: https://github.com/jfslin/quarkus-observability

Procedure:

quarkus create app --extension='rest'

Then manually added to pom.xml

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-observability-devservices-lgtm</artifactId>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>io.quarkiverse.micrometer.registry</groupId>
            <artifactId>quarkus-micrometer-registry-otlp</artifactId>
            <version>${micrometer-registry-otlp.version}</version>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-opentelemetry</artifactId>
        </dependency>

Dev services properly loads the grafana endpoint, as noted in the documents:

 [io.qua.obs.dep.ObservabilityDevServiceProcessor] (build-13) Dev Service Lgtm started, config: {grafana.endpoint=http://localhost:52352, quarkus.otel.exporter.otlp.endpoint=http://localhost:52351, otel-collector.url=localhost:52351, quarkus.micrometer.export.otlp.url=http://localhost:52351/v1/metrics, quarkus.otel.exporter.otlp.protocol=http/protobuf}

Now, if I attempt to go to the endpoint http://localhost:8080/hello leads to the following error:

2024-09-01 22:37:56,634 INFO  [io.quarkus] (Quarkus Main Thread) org.acme 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.14.1) started in 9.059s. Listening on: http://localhost:8080
2024-09-01 22:37:56,636 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2024-09-01 22:37:56,637 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [cdi, micrometer, observability, opentelemetry, rest, smallrye-context-propagation, vertx]
2024-09-01 22:39:01,611 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-4) First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-01 22:39:01,617 WARN  [io.ver.cor.htt.imp.HttpClientConnection] (vert.x-eventloop-thread-4) Connection evicted
2024-09-01 22:39:01,749 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-4) First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-01 22:39:01,751 WARN  [io.ver.cor.htt.imp.HttpClientConnection] (vert.x-eventloop-thread-4) Connection evicted
2024-09-01 22:39:01,919 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-4) First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-01 22:39:01,921 WARN  [io.ver.cor.htt.imp.HttpClientConnection] (vert.x-eventloop-thread-4) Connection evicted
2024-09-01 22:39:02,138 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-4) First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-01 22:39:02,141 WARNING [io.qua.ope.run.exp.otl.sen.VertxGrpcSender] (vert.x-eventloop-thread-4) Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-01 22:39:02,143 WARNING [io.ope.exp.int.grp.GrpcExporter] (vert.x-eventloop-thread-4) Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-01 22:39:02,145 WARN  [io.ver.cor.htt.imp.HttpClientConnection] (vert.x-eventloop-thread-4) Connection evicted

Some searching suggest that I sent HTTP 2 messages or pointing to the wrong server type, as per here and here, but I'm not sure how to modify this.

Then, attempting the implement the next part of the example (Testing full Grafana OTel LGTM stack - example), which I am assuming is needed to hook up the data transmission (since the Grafana dev-endpoint tells me there's no data), I am not able to find:

• arr in GreetingResources:22
• QueryResult in GrafanaClient:85
• login in LgtmTestBase:22
• data in LgtmTestBase:25

[cescoffier]

\CC @alesj

[alesj]

This should explain a few things

• Add LGTM traces test / check #42901

Atm tracing doesn't work ootb -- we have some config issues.
But if you would set the right endpoints and protocol explicitly it should work.

The "missing" classes are here (see also "support" sub-folder)

• https://github.com/quarkusio/quarkus/tree/main/integration-tests/observability-lgtm/src/test/java/io/quarkus/observability/test

HTH

[jfslin]

@alesj Thank you. Using the classes, pom, and application.properties settings from your links, I was able to compile. Other than the span/trace errors (ie Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f) that you noted doesn't work OOTB, it seems to be okay. That is, when I implement some of the contents from these 3 tutorials:

• https://quarkus.io/guides/opentelemetry-metrics
• https://quarkus.io/guides/telemetry-micrometer
• https://quarkus.io/guides/opentelemetry

I get (with quarkus.otel.metrics.exporter=logging):

2024-09-03 17:51:29,586 DEBUG [io.qua.obs.tes.Lgt.docker] (docker-java-stream--193521764) The OpenTelemetry collector and the Grafana LGTM stack are up and running.
2024-09-03 17:51:29,588 DEBUG [io.qua.obs.tes.Lgt.docker] (docker-java-stream--193521764) Open ports:
2024-09-03 17:51:29,590 DEBUG [io.qua.obs.tes.Lgt.docker] (docker-java-stream--193521764)  - 4317: OpenTelemetry GRPC endpoint
2024-09-03 17:51:29,591 DEBUG [io.qua.obs.tes.Lgt.docker] (docker-java-stream--193521764)  - 4318: OpenTelemetry HTTP endpoint
2024-09-03 17:51:29,594 DEBUG [io.qua.obs.tes.Lgt.docker] (docker-java-stream--193521764)  - 3000: Grafana. User: admin, password: admin
2024-09-03 17:51:34,326 INFO  [io.ope.exp.log.LoggingMetricExporter] (PeriodicMetricReader-1) Received a collection of 1 metrics for export.
2024-09-03 17:51:34,328 INFO  [io.ope.exp.log.LoggingMetricExporter] (PeriodicMetricReader-1) metric: ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={host.name="DESKTOP-530TG09", service.name="org.acme", service.version="1.0.0-SNAPSHOT", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.39.0", webengine.name="Quarkus", webengine.version="3.14.1"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.sdk.trace, version=null, schemaUrl=null, attributes={}}, name=queueSize, description=The number of items queued, unit=1, type=LONG_GAUGE, data=ImmutableGaugeData{points=[ImmutableLongPointData{startEpochNanos=1725400284309975200, epochNanos=1725400294322138700, attributes={processorType="BatchSpanProcessor"}, value=0, exemplars=[]}]}}

2024-09-03 17:51:44,323 INFO  [io.ope.exp.log.LoggingMetricExporter] (PeriodicMetricReader-1) Received a collection of 1 metrics for export.
2024-09-03 17:51:44,324 INFO  [io.ope.exp.log.LoggingMetricExporter] (PeriodicMetricReader-1) metric: ImmutableMetricData{resource=Resource{schemaUrl=null, attributes={host.name="DESKTOP-530TG09", service.name="org.acme", service.version="1.0.0-SNAPSHOT", telemetry.sdk.language="java", telemetry.sdk.name="opentelemetry", telemetry.sdk.version="1.39.0", webengine.name="Quarkus", webengine.version="3.14.1"}}, instrumentationScopeInfo=InstrumentationScopeInfo{name=io.opentelemetry.sdk.trace, version=null, schemaUrl=null, attributes={}}, name=queueSize, description=The number of items queued, unit=1, type=LONG_GAUGE, data=ImmutableGaugeData{points=[ImmutableLongPointData{startEpochNanos=1725400284309975200, epochNanos=1725400304323406500, attributes={processorType="BatchSpanProcessor"}, value=0, exemplars=[]}]}}
2024-09-03 17:52:45,489 ERROR [io.ver.cor.net.imp.ConnectionBase] (vert.x-eventloop-thread-4) First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-03 17:52:45,494 WARNING [io.qua.ope.run.exp.otl.sen.VertxGrpcSender] (vert.x-eventloop-thread-4) Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-03 17:52:45,496 WARNING [io.ope.exp.int.grp.GrpcExporter] (vert.x-eventloop-thread-4) Failed to export spans. Server responded with gRPC status code 2. Error message: Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2024-09-03 17:52:45,498 WARN  [io.ver.cor.htt.imp.HttpClientConnection] (vert.x-eventloop-thread-4) Connection evicted

Next, I will try to get the trace/span stuff working... I'm not sure if I follow the content of the LGTM traces test yet but I will give it another shot shortly.

[alesj]

The tracing config bug fix is now upstream, so it should all work now -- if you use upstream.