Problem with Kafka 4.X.X Kraft and Scram

[nico34638]

Describe the bug
Impossible to connect to a Kafka 4.x.x cluster and the latest release of quickwit with a TLS kraft cluster and SCRAM authentication. There's a problem in rkafka.

You'll need to update the version of rust rdkafka that came out this week to fix it.

Steps to reproduce (if applicable)
Steps to reproduce the behavior:

1. Quickwit 0.8.2 and Kafka source

Configuration:

Latest quickwit release and otel traces default index

[fmassot]

Thanks @nico34638 for reporting this issue. Do you want to open a PR on this?

[otetard]

I can confirm that without using the latest version of rdkafka (>= 0.38), and some basic Rust code, I can’t connect to a Kafka 4 broker using SCRAM-SHA-256 (I didn’t try to use previous Kafka versions, though).

The rdkafka crate fix was introduced in fede1024/rust-rdkafka@3a5ad93, which updates librdkafka from 2.4.0 to 2.10.0. Using rdkafka crate with the dynamic-linking feature and librdkafka 2.8.0 (current version in Debian stable) also works fine.

[nico34638]

I created this PR I need a review and are there any tests on the kafka part?

[nico34638]

The issue can be closed, is it possible to launch a build of the tag edge so that I can test?

[rdettai]

it was just deployed to edge https://hub.docker.com/r/quickwit/quickwit/tags