What is the motivation behind only allowing a single TraceProvider in the IServiceCollection?

[solidyellowstudio]

The opentelemetry documentation for customizing the sdk has the following note.

In the same documentation, another area mentions the Sdk.CreateTraceProviderBuilder() is available in scenarios where multiple providers are required.

The motivation for my questions is that I want to add multiple trace providers to a .NET Aspire application, so I can send a specific set of traces and logs to a different OTEL application for analysis, while still maintaining the .NET Aspire standalone dashboard experience.

Are the statements in the documentation in conflict with each other or am I interpreting them incorrectly ?

Is there a different approach I should consider to send traces to multiple or different OTEL backends ?

[TimothyMothra]

Hi @solidyellowstudio, you can create multiple TracerProviders

There are two APIs available get configuring OpenTelemetry in your application. Both of these are in that doc.
https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/customizing-the-sdk/README.md#building-a-tracerprovider

• OpenTelemetry.Extensions.Hosting provides the API services.AddOpenTelemetry()

  You can think of this as a convince method, it's creating the TracerProvider (and providers for other signals) and adding them to your application's ServiceCollection which handles the lifecycle of the Provider.

• SDK.CreateTracerProviderBuilder

  You can still use this API to create an additional provider, but you'll be responsible for storing it somewhere that it won't be disposed until your application shuts down.

[juarola]

That sounds promising! Are you aware of any samples/how-tos/blog posts on how to accomplish that in practice? My current issue is that i cannot seem to get span processors to apply to only a single "pipeline". Maybe i'm approaching this completely wrong, but i'm using processors to redact sensitive data from trace entries.

[cijothomas]

I am not aware of any full examples in this repo, but the concepts are here : https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/extending-the-sdk
The general idea is you do the below:

1. RedactionProcessor(Wrapping the exporting processor) -- send to backend A
2. ExportingProcessor - send to backend B

Since processors are invoke in the same order, and changes made to Activity are visible to next processors (i.e pipelines are not independent), you'd want to orchestrate the ordering to avoid "side-effect" from redaction affecting other pipelines.

[cijothomas]

@martinjt Does a lot of amazing blogs/videos on OTel, so maybe have pointers to external blogs showing this?

[martinjt]

I'm slightly confused by the question, but perhaps it's to do with understanding the relationships.

Aspire is "just" a destination for the OpenTelemetry SDK to send data too.

Right now, the OOTB Aspire ServiceDefaults uses the .UseOtlpExporter() pattern allows only a single OTLP exporter destination to be configured, and it should use the Environment Variables of the application select the destination.

You shouldn't (in 99.9999999% of scenarios) add multiple tracer providers. If you goal is pushing to multiple destinations and adding pipelines for filtering redaction, that's literally the reason the OpenTelemetry Collector exists. This is an application that sits between your app and your multiple destinations and duplicates, filters and redacts telemetry as you need it.

I don't have content specifically on what you're looking for, but I have an Aspire extension which adds a collector that you can configure.

https://github.com/practical-otel/opentelemetry-aspire-collector

I would suggest looking at that over trying to move away from the servicedefaults if you're not confident with opentelemetry and how Aspire works under the hood.

[juarola]

...and immediately after a month i pick this up again, sorry 😅

Thank you @cijothomas and @martinjt for the insights, now i think i got it: Point is to configure all the intricacies of what to send where on collector level, not in the application code. It makes 100% sense to decouple app from this stuff.

But. Lets consider a situation where i need to export telemetry to two backends:

1. On-prem Seq, as some of the telemetry data contains sensitive data, so that should not end up stored outside private data centers.
2. Application Insights, generic logs, traces, metrics about usage, exception stack traces can be exported here

Exporting to Seq is a no-brainer, but what about Application Insights (or Azure Monitor). It looks like OpenTelemetry collector approach is not supported, so am i basically back in square one, or am i missing something important here?

[AlexandruRus23]

I have a similar requirement.

I needed to have an additional OpenTelemetry destination, but apply special filtering (only a subset of spans).

I've went the route of building multiple trace providers

var tracer1 = Sdk.CreateTracerProviderBuilder()
  .AddSource(...)
  .AddOtlpExporter(destination1)
  .Build()

var tracer2 = Sdk.CreateTracerProviderBuilder()
  .AddSource(...)
  .AddProcessor<>()
  .SetSampler(...)
  .AddOtlpExporter(destination2)
  .Build()

I'm flushing and disposing them myself at application shutdown.

I have, however stumbled on some oddities, namely:

• couldn't get the right data across the two tracers initially, after some reordering, it looks good
• my sampler doesn't seem to be applied to the second tracer (the first one doesn't have one). Even if I set the AlwaysOffSampler, traces are still being sent. Worth mentioning that the Processor also "samples" by looking at tags and setting

activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
activity.IsAllDataRequested = false;

Maybe someone from the OpenTelemetry team could comment on if there are any issues with this setup or if what I'm seeing is expected, for a tracer setup to be influenced by another's.

[martincostello]

I can't comment on the original design of the .NET SDK in this regard, but this use case sounds like one that's typically dealt with in a collector rather than in the application. The application would send all the data to a collector, then rules in the collector would be configured to forward the data to different endpoints with additional sampling as required.

[martinjt]

The tracers act on the same activity, therefore the sampler would only be applied once to both.

Its not advised to use 2 exporters like this. The recommended approach is to use the OpenTelemetry to apply the sampling and filtering. Alternatively, you would need to apply that within a custom exporter.

The issue you have is that you're acting on the same activities since they're not scoped to the tracers (they're created by the application), therefore if you decide to discard a trace during sampling, it would apply the flags to the activity that both tracer providers receive.

The other issue is about trace parents, since they're a chain, if you don't export one span in the middle of the trace, you get a broken trace.

[AlexandruRus23]

Makes sense, thank you for the quick replies.

I know this is typically a job for an Otel Collector but I'm (at least initially) trying to avoid the additional infrastructure. I'm also using the AzureMonitor integration so I would still send the telemetry to two places from the app and the otel collector would have one receiver and one exporter (which feels like it's defeating the purpose of having the collector in the first place).

Doing a custom exporter sounds like an interesting solution. I'd want in this case to build a wrapper over an Otlp Exporter. Do you know of any samples of such a thing?

Much appreciated!

[martinjt]

For AzureMonitor, you'd replace the .NET libraries and use the AzureMonitor exporter in the collector along with your OTLP destinations.

I don't know of any examples I'm afraid.

This is ultimately one of the reasons that any vendor requiring a processor to be added isn't really supporting OpenTelemetry properly as it makes it impossible to do dual backends and therefore brings lock-in back to the application.

In terms of infrastructure, you could sidecar the collector if its ACA, AppService or k8s instead of adding more infrastructure.