Can I remove prometheus exemplar labels?

[myeongseok-rpls]

I'm not good at English, so please understand if it's awkward.

I ran the agent using java auto-instrument.
Everything seemed to be working fine, but I didn't see any exemplar in grafana.
I checked the error below through the prometheus log.

err="label length for exemplar exceeds maximum of 128 UTF-8 characters"

When I checked the metric, there were too many labels attached to the exemplar.

# client_address="127.0.0.6",network_peer_address="127.0.0.6",network_peer_port="55579",server_address="10.3.17.168",server_port="8081",span_id="049178b29912fdb4",trace_id="0669315b30dbe08683c19ed9bd24068b",url_path="/actuator/health",user_agent_original="kube-probe/1.29+"}

Is there a way to remove the Exemplar's Label?

[laurit]

cc @jack-berg

[dashpole]

There's no way for exporters to be aware of the limit on characters.

Worst case, the prom exporter could count the characters and compare to the limit, right?

[jack-berg]

How does the prom exporter know about the limit? Is it somehow configured by the user or included as a request parameter by the client?

[dashpole]

It is a global constant right now, at least in Go: https://github.com/prometheus/client_golang/blob/93c851f0b33308b6d31bb36d6072f44d0d839c2c/prometheus/value.go#L240

[jack-berg]

Interesting.. ok yeah limiting to a hardcoded value seems good for now. If it ever becomes configurable for prometheus, we can add corresponding configuration options in the exporter.

Ideally the spec's prometheus exporter doc would give this guidance so maintainers can be aware of this common pitfall.

[jack-berg]

Issue to track: https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/12417