Fix issue with metrics exporter in otelcol.exporter.loadbalancing

[ptodev]

PR Description

In #5529, we upgraded Agent from OTel 0.85 to 0.87. One of the recent changes in OTel is the ability of the LB exporter to work with metrics. Unfortunately, this also means that its metrics exporter rejects configs which use the "traceID" routing key. Due to the way otelcol components are intialised, we always create all three exporters (logs, metrics, traces) for each otelcol exporter.

Agent does not need to support the metrics exporter in the LB exporter. That's because metrics exporting is experimental, and otelcol.exporter.loadbalancing is labeled in Flow as beta. For this reason metrics routing keys were explicitly disabled during #5529, and the ability to export metrics was not documented in the Flow docs for otelcol.exporter.loadbalancing.

Notes to the Reviewer

This issue was reported on the community Slack.

This PR is a temporary workaround so that otelcol.exporter.loadbalancing will work again with a routing key of "traceID". We will need to find a permanent solution to this problem soon. And remove this workaround.

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[rfratto]

I understand that this is to stop the bleeding, so I think this is fine.

However, I don't understand the underlying problem here, and why it's not an issue for OpenTelemetry Collector.

Are they using different default configs for different telemetry types? Do we need to copy the same pattern?

[ptodev]

FYI, I tested this locally using the k6 traces creator and this config:

otelcol.receiver.otlp "from_k6" {
    grpc {
        endpoint = "0.0.0.0:4320"
    }

    output {
    traces  = [otelcol.exporter.loadbalancing.default.input]
    }
}

otelcol.exporter.loadbalancing "default" {
  routing_key = "traceID"
    resolver {
        static {
            hostnames = ["localhost:55690"]
        }
    }
    protocol {
        otlp {
            client {
              tls {
                insecure = true
              }
            }
        }
    }
}

otelcol.receiver.otlp "from_lb" {
    grpc {
        endpoint = "0.0.0.0:55690"
    }

    output {
    traces  = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
    }
}


otelcol.exporter.otlp "grafana_cloud_tempo" {
    client {
        endpoint = "tempo-prod-06-prod-gb-south-0.grafana.net:443"
        auth     = otelcol.auth.basic.grafana_cloud_tempo.handler
    }
}

otelcol.auth.basic "grafana_cloud_tempo" {
    username = ""
    password = ""
}

[rfratto]

I think adding a disableMetricsExporter parameter feels a little too targeted for this specific bug. Since the exporter package is not where the bug originates, I think it can make adding this specific parameter a big confusing and vague.

I'd like to see this fix transform into a more general improvement to the exporter package's API, such as some kind of bitfield for which telemetry types to create exporters for, such as calling:

• exporter.New(opts, fact, args.(Arguments), exporter.TypesAll), or
• exporter.New(opts, fact, args.(Arguments), exporter.TypeLogs|exporter.TypeTraces).

This will be more work, but I think it leaves the API design feeling more intentional rather than having the ability to disable specifically metrics.

[ptodev]

TBH I'm hoping we just get rid of this workaround soon altogether. We shouldn't be initialising a metrics exporter if other components are not configured to send metrics to the LB exporter. I think that's the real problem.

[ptodev]

I added a bitfield just now, and also tested it locally by running the agent with an LB exporter and an otlp one. PTAL.

[ptodev]

However, I don't understand the underlying problem here, and why it's not an issue for OpenTelemetry Collector.

Are they using different default configs for different telemetry types? Do we need to copy the same pattern?

I think Collector would only create a metrics exporter if it's explicitly requested. I think a metrics exporter for an LB would only be created it it's in a metrics pipeline. There's an example of a traces pipeline in the otel docs.

We had to make changes to component/otelcol/processor/processor.go to do checks like if len(next.Traces) > 0 for a similar reason as well.

[rfratto]

Thanks!