Skip to content

host_metrics source is not forwarded to opentelemetry sink #25622

Description

@olibanez

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I have a setup containing several servers with vector 0.55 running that export the native source host_metrics as well as a single dedicated server that collects all the vector data and forwards it somewhere.

Unfortunately, the collected data on the didcated server is not forwarded to Opentelemetry for any reason. I added a second and third sink for verification (prometheus_exporter and file) that both work well.

Everything is working well except the otlp shipping.
What Im wondering most about is that if I run vector on collecting side with sudo -u vector VECTOR_LOG=trace vector --config=/etc/vector/vector.yaml I see logs for all the other sinks but there is no log for the otel sink custB except from "Starting sink" etc. but no logs for this sink when events come in while the same metrics are written into the other verification/debugging sinks.

If I run vector tap --inputs-of custB in parallel, I can see incoming events. But from there on I'm blind and I can't find out why nothing is actually sent out from that sink. tcpdump also doesn't show any traffic.
It seems like there are any errors in between that aren't traced/logged anywhere.

Configuration

This is the config on the exporting systems:

sources:
  host:
    type: host_metrics
transforms:
  set_host:
    type: remap
    inputs:
      - host
    source: |-
      .tags.host = "myhostname"
sinks:
  metrics_sink:
    type: prometheus_remote_write
    inputs:
    - host
    endpoint: http://myhostname:8428/api/v1/write
    healthcheck:
      enabled: false
    acknowledgements:
      enabled: false
  vector_sink:
    type: vector
    inputs:
    - set_host
    address: mycollectorhost:5500
    healthcheck:
      enabled: true
api:
  enabled: true
  address: 127.0.0.1:8686

This is the config of the collector:

sources:
  vector:
    type: vector
    address: "0.0.0.0:5500"
transforms:
  reroute_fb:
    type: route
    inputs:
      - vector
    route:
      route_custA: 'contains(string!(.tags.host), ".custA.")'
      route_custB: 'contains(string!(.tags.host), ".custB.")'
      route_custC: 'contains(string!(.tags.host), ".custC.")'
sinks:
  custA:
    type: prometheus_exporter
    inputs:
      - reroute_fb.route_custA
    address: 0.0.0.0:9598
    flush_period_secs: 360
  custB:
    type: opentelemetry
    inputs:
      - reroute_fb.route_custB
        #    batch:
        #      max_events: 1
    protocol:
      type: http
      uri: https://my-otel-receiver/v1/metrics
      method: post
      encoding:
        codec: otlp
      auth:
        strategy: bearer
        token: <mytoken>
  debug_otlp2:
    type: prometheus_exporter
    inputs:
      - reroute_fb.route_custB
    address: 0.0.0.0:9599
    flush_period_secs: 360
  debug_otlp:
    type: file
    inputs:
      - reroute_fb.route_custB
    encoding:
      codec: "json"
      json:
        pretty: true
    path: /tmp/custB-sysinfo.json
  custC:
    type: file
    inputs:
      - reroute_fb.route_custC
    encoding:
      codec: "json"
      json:
        pretty: true
    path: /tmp/custC-sysinfo.json
  unmatched:
    type: file
    inputs:
      - reroute_fb._unmatched
    encoding:
      codec: "json"
      json:
        pretty: true
    path: /tmp/other-sysinfo.json

api:
  enabled: true
  address: 127.0.0.1:8686

Version

0.55.0 on almalinux9

Debug Output


Example Data

Example metrics that is written to the file sink for custB:

{
  "name": "process_memory_usage",
  "namespace": "host",
  "tags": {
    "collector": "process",
    "command": "",
    "host": "mysourcehost",
    "name": "idle_inject/190",
    "pid": "1170"
  },
  "timestamp": "2026-06-15T08:56:42.174077193Z",
  "kind": "absolute",
  "gauge": {
    "value": 0.0
  }
}

Additional Context

We found out that if we use the following config with codec: json + framing + headers options, the sink is at least sending data and we get a "200 OK" http response:

  custB:
    type: opentelemetry
    inputs:
      - reroute_fb.route_custB
    buffer:
      max_events: 1
    protocol:
      batch:
        max_events: 1
      type: http
      uri: https://my-otel-receiver/v1/metrics
      method: post
      encoding:
        codec: json
      auth:
        strategy: bearer
        token: <mytoken>
      framing:
        method: newline_delimited
      request:
        headers:
          content-type: application/json

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions