[prometheusexporter] Expose native histograms

[krisztianfekete]

Component(s)

exporter/prometheus

What happened?

Description

Currently, prometheusreceiver supports native histograms, but prometheusexporter cannot expose them (because Prometheus cannot expose native histograms in text format yet).

Steps to Reproduce

1. Instrument your code with native histograms
2. Scrape the metrics with prometheusreceiver by following the instructions here: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/f5d54da37ae664e1a0b6934a61d38f1afab5501f/receiver/prometheusreceiver/README.md#prometheus-native-histograms

Expected Result

prometheusexporter should expose native histograms

Actual Result

Logs via debugexporter:

{"level":"error","ts":1718965637.50062,"caller":"[email protected]/accumulator.go:94","msg":"failed to translate metric","kind":"exporter","data_type":"metrics","name":"prometheus","data_type":"\u0004","metric_name":"gloo_mesh_translation_time_sec","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:94
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:71
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:88
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:85
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:158
--

Collector version

v0.103.0

Environment information

N/A

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      global:
        evaluation_interval: 1m
        scrape_interval: 15s
        scrape_timeout: 15s
        scrape_protocols: [ PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4 ]
        scrape_configs:
          <your targets exposing native histograms>
          ...
pipelines:
  metrics/test:
    receivers:
    - prometheus
    exporters:
    - debug
    - prometheus

Log output

{"level":"error","ts":1718965637.50062,"caller":"[email protected]/accumulator.go:94","msg":"failed to translate metric","kind":"exporter","data_type":"metrics","name":"prometheus","data_type":"\u0004","metric_name":"gloo_mesh_translation_time_sec","stacktrace":"github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:94
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:71
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:88
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
\tgithub.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:85
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:59
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:37
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
\tgo.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:158
--

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheus: @Aneurysm9

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[wildum]

Hi, I tried it with the following config, and the native histograms are exposed just like they are exposed by my app:

receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: 'otel-collector'
            scrape_interval: 5s
            static_configs:
              - targets: ['localhost:9001']

exporters:
  prometheus:
    endpoint: "localhost:9010"
    const_labels:
      test_name: v1

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheus]
  telemetry:
    logs:
      level: debug

Native histogram metrics exposed by my app:
golang_native_histogram_bucket{address="0.0.0.0",port="9001",le="+Inf"} 1523
golang_native_histogram_sum{address="0.0.0.0",port="9001"} 773369.1551833635
golang_native_histogram_count{address="0.0.0.0",port="9001"} 1523

Native histogram metrics exposed by the prometheus exporter:
golang_native_histogram_bucket{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1",le="+Inf"} 1284
golang_native_histogram_sum{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1"} 656156.1718255163
golang_native_histogram_count{address="0.0.0.0",instance="localhost:9001",job="otel-collector",port="9001",test_name="v1"} 1284

Could you share your full setup?

[krisztianfekete]

Are you sure these are native histograms? They look like regular ones to me. The required options are seemingly missing from your receiver config: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md#prometheus-native-histograms, so my guess is that you're falling back to the legacy ones here.

EDIT: I added them to the original post, so it's easier to reproduce.

[wildum]

Ah you are right, I was missing the scape_protocols, I can reproduce it now, thanks :)

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Blocked by prometheus/client_golang#1617.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

This is mostly unblocked after prometheus/client_golang#1654. We just need to wait for a release. Exemplars are not yet supported