How to emit traces for two services from a single java application with manual instrumentation?

[MacNale]

I have some traces captured in a csv file for two different microservices (e.g ServiceA and ServiceB)

In my spring boot application, I would like to read this csv file and convert it to OLTP and emit. I initialized the SDK as follows -

@Bean
public OpenTelemetrySdk getOpenTelemetry (){
    Resource serviceNameResource =
            Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "demo-service"));
    return OpenTelemetrySdk.builder()
            .setTracerProvider(
                    SdkTracerProvider.builder()
                            .addSpanProcessor(SimpleSpanProcessor.create(getJaegerExporter()))
                            .addSpanProcessor(SimpleSpanProcessor.create(LoggingSpanExporter.create()))
                            .setSampler(Sampler.alwaysOn())
                            .setResource(Resource.getDefault().merge(serviceNameResource))
                            .build())
            .setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
            .buildAndRegisterGlobal();
}

Now read the CSV file and converted to spans as follows -

SpanContext spanContext = SpanContext.createFromRemoteParent(traceId, spanId,
        TraceFlags.getDefault(),TraceState.getDefault());
Context context = Context.current().with(Span.wrap(spanContext));
Tracer tracer = GlobalOpenTelemetry.get().getTracer()

SpanBuilder spanBuilder = tracer.spanBuilder("your-span-name")
    .setSpanKind(SpanKind.SERVER).setAttribute("service.name", service_name)
    .setStartTimestamp(startTime)
    .setParent(context);

try (Scope scope = context.makeCurrent()) {
    Span span = spanBuilder.startSpan();
    span.end(endTime);
}

This code emits the traces in Jaeger, but I see only one Service ("demo-service") in the drop down.

1. How can I associate appropriate "service_name" in the trace, so that Jaeger shows me two different microservices and respective traces?
2. Do I need to create multiple OpenTelemetrySdks? Is that recommended?

[jkwatson]

Yes, you would need to create more than one instance of the SDK. I don't think there's another way to do this at the moment, and it should work fine. Just avoid calling buildAndRegisterGlobal because there can only be one global and the 2nd call would fail.

[MacNale]

Let me make sure if I understood your suggestion correctly - Pseudo code -

Option 1 - Using sdk

SpanData mySpanData = MySpanData.builder()
        .spanId(myspanId)
        .spanContext(myspancontext)
        .transaction(transaction).build();

Collection<SpanData> spanData = new ArrayList<SpanData>();
spanData.add(mySpanData);

OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
        .setEndpoint("your-endpoint-url")
        .build();

exporter.export(spanData);

Option 2 : Generate ExportTraceServiceRequest using protobuf

ExportTraceServiceRequest.Builder builder = ExportTraceServiceRequest.newBuilder();
....
dump in a kafka topic to be picked up by Opentelemetry collector

Which one are you suggesting?

[jkwatson]

I don't understand Option 2. I was suggesting Option 1.

[MacNale]

Alright.. I did not find the any spanBuilder. I implemented custom Spandata and used exporter. I can see the trace in Jaeger now. Yea!!

    Resource resource = Resource.getDefault()
            .merge(Resource.create(Attributes.of(ResourceAttributes.SERVICE_NAME, "mac_logical-service-name")));

    String traceID = TraceId.fromLongs(111121212 ,121212);
    String spanID =  SpanId.fromLong(1213213);
    String parentId = SpanId.getInvalid();

    SpanContext mySpanContext = SpanContext.create(traceID, spanID, TraceFlags.getDefault(), TraceState.getDefault());

    CustomSpanData customSpanData = new CustomSpanData(mySpanContext,
            "custom_span_data",
            InstrumentationScopeInfo.create("custom_span_instrumentation"),
            SpanKind.CLIENT,
            SpanContext.getInvalid(),
            resource,
            AttributesMap.create(10, 10),
            Collections.emptyList(),
            Instant.now().toEpochMilli() * 1_000_000,
            StatusData.unset(),
            Instant.now().toEpochMilli() * 1_000_000 + 10,
            Collections.emptyList(),
            true
    );

    SpanExporter spanExporter = ExampleConfiguration.getJaegerExporter();
    CompletableResultCode completableResultCode = spanExporter.export(Collections.singletonList(customSpanData));

public class CustomSpanData implements SpanData {
    // Contains the identifiers associated with this Span.
    private SpanContext context = SpanContext.getInvalid();
    // The parent SpanContext of this span. Invalid if this is a root span.
    private SpanContext parentSpanContext = SpanContext.getInvalid();
    ...

Let me know if there is an easy way to create SpanData/Span.

One of the drawback of this approach is it bypasses the span processor(s) hence I will loose the sampling/batching processors. :(

[jkwatson]

Yes, this is not a use-case that we really support, so I think this is the best you can do for now.

[MacNale]

Thank you for all the pointers. I could successfully complete the POC.

[MacNale]

SpanData mySpanData = MySpanData.builder()
.spanId(myspanId)
.spanContext(myspancontext)
.transaction(transaction).build();

Collection spanData = new ArrayList();
spanData.add(mySpanData);

OtlpGrpcSpanExporter exporter = OtlpGrpcSpanExporter.builder()
.setEndpoint("your-endpoint-url")
.build();

exporter.export(spanData);