Kafka sink: static (config-constant) headers and key, for parity with the HTTP sink's request.headers

[therajvira]

Summary

The Kafka sink can set the message key (key_field) and headers (headers_key) only from an event field. There is no way to attach a static, config-constant header or key from the sink configuration itself. This is a consistency gap: Vector already supports static, config-constant transport headers on another sink, and the Kafka sink already supports a static topic.

This is a consistency gap, not a new paradigm

• The HTTP sink already has request.headers: "Additional HTTP headers to add to every HTTP request," accepting constant values (for example "X-My-Custom-Header": "A-Value"), with no transform and no event field. So a sink attaching static, config-constant transport headers is already an accepted, shipped pattern in Vector.
• The Kafka sink's own topic is already a static or templated config constant that lands in transport metadata.
• The Kafka sink simply lacks the equivalent for headers and key. So today the only way to attach a constant identifier is a remap transform that injects it into the event body.

Headers and the message key are transport-envelope metadata, which the sink already constructs (topic, key_field, headers_key, encoding). Setting a constant in that envelope from config is within the sink's existing role, not a transform's job.

Use case

In fan-in topologies where many independent pipelines (source to transforms to sink) write to one shared Kafka topic, downstream consumers need to know which pipeline/source produced each message, for provenance, multi-tenancy, routing, lineage, and debugging. The producing pipeline is known at config time (each sink belongs to one pipeline), so the natural place to attach its identifier is the sink config.

Forcing a remap transform for this:

1. mutates the event payload just to carry transport-level metadata,
2. adds a transform to every pipeline (config sprawl, overhead),
3. is fragile, since a transform that rebuilds the event (for example . = .parsed) drops any earlier tag, so the constant must be re-injected right before the sink.

Proposal

Add optional static header(s) and/or a static key to the Kafka sink, sourced from configuration rather than an event field, mirroring the HTTP sink's request.headers:

[sinks.my_kafka]
type = "kafka"
# new: constant headers on every produced message (parity with http sink request.headers)
headers = { source_id = "pipeline-7", env = "prod" }
# optional: a constant message key
key = "pipeline-7"

These would merge with the existing headers_key / key_field (event-field) values, so static and dynamic can coexist.

Why generally useful

• Provenance and lineage without touching payloads.
• Multi-tenancy and routing: consumers can filter or route by a constant the producer set. For example the ClickHouse Kafka engine exposes _headers and _key virtual columns, directly readable downstream; today the only config-only signal is the topic name, which forces a topic-per-source design.
• Clean separation of payload vs transport metadata.
• The same gap exists for other framed sinks. Related requests: Allow headers in NATS Jetstream sink #23509 (NATS), Support for headers in Loki Sink Configuration #21332 (Loki).

Current workaround

Inject the constant via a remap transform and reference it with headers_key / key_field. It works, but it pollutes the event body and adds a transform per pipeline.