introduce an otelcol.exporter.kafka component #6752

tpaschalis · 2024-03-22T13:28:04Z

PR Description

This component adds the otelcol.exporter.kafka component so that Grafana Agent can work with Kakfa pipelines end-to-end.

Which issue(s) this PR fixes

No issue files

Notes to the Reviewer

PR Checklist

CHANGELOG.md updated
Documentation added
Tests updated
Config converters updated

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis · 2024-03-22T15:50:02Z

internal/component/otelcol/exporter/kafka/kafka_test.go

+				rr := broker.History()[0]
+				ch <- fmt.Sprint(rr.Request)


I don't particularly like this, but the sarama library's mock doesn't allow for better introspection of the messages as most moving pieces are unexported :/

rfratto · 2024-03-22T15:57:25Z

internal/component/otelcol/exporter/kafka/kafka.go

+func init() {
+	component.Register(component.Registration{
+		Name:      "otelcol.exporter.kafka",
+		Stability: featuregate.StabilityStable,


Is this the right stability? This would mean committing to the config schema for at least one year, and declaring knowledge about risk / production readiness. Is this ready for that?

clayton-cornell

Some minor doc suggestions

clayton-cornell · 2024-03-23T00:13:49Z

docs/sources/flow/reference/components/otelcol.exporter.kafka.md

+> **NOTE**: `otelcol.exporter.kafka` is a wrapper over the upstream
+> OpenTelemetry Collector `kafka` exporter. Bug reports or feature requests will
+> be redirected to the upstream repository, if necessary.


Suggested change

> **NOTE**: `otelcol.exporter.kafka` is a wrapper over the upstream

> OpenTelemetry Collector `kafka` exporter. Bug reports or feature requests will

> be redirected to the upstream repository, if necessary.

{{< admonition type="note" >}}

`otelcol.exporter.kafka` is a wrapper over the upstream OpenTelemetry Collector `kafka` exporter.

Bug reports or feature requests will be redirected to the upstream repository, if necessary.

{{< /admonition >}}

clayton-cornell · 2024-03-23T00:15:20Z

docs/sources/flow/reference/components/otelcol.exporter.kafka.md

+> be redirected to the upstream repository, if necessary.
+
+This component uses a synchronous producer that blocks and does not batch
+messages, therefore it should be used with batch and queued retry processors


Suggested change

messages, therefore it should be used with batch and queued retry processors

messages. Therefore, it should be used with batch and queued retry processors

clayton-cornell · 2024-03-23T00:15:27Z

docs/sources/flow/reference/components/otelcol.exporter.kafka.md

+
+This component uses a synchronous producer that blocks and does not batch
+messages, therefore it should be used with batch and queued retry processors
+prepended to it, for higher throughput and resiliency.


Suggested change

prepended to it, for higher throughput and resiliency.

prepended to it for higher throughput and resiliency.

wildum · 2024-03-25T09:08:13Z

internal/component/otelcol/exporter/kafka/kafka.go

+
+	Auth     kafka.AuthenticationArguments `river:"authentication,block,optional"`
+	Metadata kafka.MetadataArguments       `river:"metadata,block,optional"`
+	Producer Producer                      `river:"producer,block,optional"`


this block is missing in the documentation

wildum · 2024-03-25T09:13:22Z

docs/sources/flow/reference/components/otelcol.exporter.kafka.md

+
+Name | Type | Description | Default | Required
+---- | ---- | ----------- | ------- | --------
+`protocol_version` | `duration` | Kafka protocol version to use. | | yes


wildum · 2024-03-25T09:27:09Z

docs/sources/flow/reference/components/otelcol.exporter.kafka.md

+`protocol_version` | `duration` | Kafka protocol version to use. | | yes
+`brokers` | list(string) | Kafka brokers to connect to. | `["localhost:9092"]` | no
+`client_id` | string | The client ID to configure the Sarama Kafka client with for all produce requests. | `"sarama"` | no
+`topic` | string | The name of the Kafka topic to export to. | see below | no


To what is this "see below" referring?

wildum · 2024-03-25T09:28:51Z

docs/sources/flow/reference/components/otelcol.exporter.kafka.md

+`brokers` | list(string) | Kafka brokers to connect to. | `["localhost:9092"]` | no
+`client_id` | string | The client ID to configure the Sarama Kafka client with for all produce requests. | `"sarama"` | no
+`topic` | string | The name of the Kafka topic to export to. | see below | no
+`encoding` | string | The encoding of data sent to Kafka. | see below | no


I think that we should still say that otlp_proto is the default

github-actions · 2024-05-10T00:10:48Z

This PR has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If you do not have enough time to follow up on this PR or you think it's no longer relevant, consider closing it.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your PR will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

rfratto · 2024-06-14T16:30:42Z

I'm going to close this; the otelcol.exporter.kafka component in Alloy has taken precedence.

tpaschalis added 4 commits March 22, 2024 15:26

introduce an otelcol.exporter.kafka component

a75027d

Signed-off-by: Paschalis Tsilias <[email protected]>

Add converter

19e2619

Signed-off-by: Paschalis Tsilias <[email protected]>

Add aws_msk block to docs

235a34a

Signed-off-by: Paschalis Tsilias <[email protected]>

Regenerate compatible components

b9078ac

Signed-off-by: Paschalis Tsilias <[email protected]>

tpaschalis marked this pull request as ready for review March 22, 2024 15:22

tpaschalis requested a review from clayton-cornell as a code owner March 22, 2024 15:22

tpaschalis commented Mar 22, 2024

View reviewed changes

rfratto reviewed Mar 22, 2024

View reviewed changes

clayton-cornell reviewed Mar 23, 2024

View reviewed changes

clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Mar 23, 2024

wildum reviewed Mar 25, 2024

View reviewed changes

rfratto added the variant/flow Relatd to Grafana Agent Flow. label Apr 9, 2024

github-actions bot added the needs-attention An issue or PR has been sitting around and needs attention. label May 10, 2024

rfratto closed this Jun 14, 2024

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Jul 15, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jul 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce an otelcol.exporter.kafka component #6752

introduce an otelcol.exporter.kafka component #6752

tpaschalis commented Mar 22, 2024 •

edited

Loading

tpaschalis Mar 22, 2024

rfratto Mar 22, 2024 •

edited

Loading

clayton-cornell left a comment

clayton-cornell Mar 23, 2024

clayton-cornell Mar 23, 2024

clayton-cornell Mar 23, 2024

wildum Mar 25, 2024

wildum Mar 25, 2024

wildum Mar 25, 2024

wildum Mar 25, 2024

github-actions bot commented May 10, 2024

rfratto commented Jun 14, 2024

-> **NOTE**: `otelcol.exporter.kafka` is a wrapper over the upstream
-> OpenTelemetry Collector `kafka` exporter. Bug reports or feature requests will
-> be redirected to the upstream repository, if necessary.
+{{< admonition type="note" >}}
+`otelcol.exporter.kafka` is a wrapper over the upstream OpenTelemetry Collector `kafka` exporter.
+Bug reports or feature requests will be redirected to the upstream repository, if necessary.
+{{< /admonition >}}

	messages, therefore it should be used with batch and queued retry processors
	messages. Therefore, it should be used with batch and queued retry processors

	prepended to it, for higher throughput and resiliency.
	prepended to it for higher throughput and resiliency.

	`protocol_version` \| `duration` \| Kafka protocol version to use. \| \| yes
	`protocol_version` \| string \| Kafka protocol version to use. \| \| yes

introduce an otelcol.exporter.kafka component #6752

introduce an otelcol.exporter.kafka component #6752

Conversation

tpaschalis commented Mar 22, 2024 • edited Loading

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

Choose a reason for hiding this comment

rfratto Mar 22, 2024 • edited Loading

Choose a reason for hiding this comment

clayton-cornell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 10, 2024

rfratto commented Jun 14, 2024

tpaschalis commented Mar 22, 2024 •

edited

Loading

rfratto Mar 22, 2024 •

edited

Loading