Connection count spike

[showuon]

Hi,
We recently found the connection count has 2x ~ 3x spike during a short network issue (supposedly the connection dropping and then resumed immediately) using confluent-kafka-python 1.9.2 (librdkafka 1.9.2). Note that brokers are all healthy, but just the network connections between client/broker are lost simultaneously. Here's the connection count graph:

We're producing records to 1 topic with 1 partition. At first, everything goes well, bootstrap server connection + partition leader connection. And then, connection idle ms expired, server cut the connection, so we have 1 left. Suddenly, the network issue comes, the connection drop to 0 at first, then, from the log, we found, not only the existing connection keeps retrying the connection, there are also other connections happening at the same time.

1. broker 1 is selected due to "broker down"

%7|1670902345.594|CONNECT|rdkafka#producer-1| [thrd:sasl_ssl://broker-5.redacted] sasl_ssl://broker-1.redacted:443/1: Selected for cluster connection: broker down (broker has 0 connection attempt(s))

This makes sense because we tried to get metadata update from other available broker. But following this log, I can also see

2. broker 2 is selected due to "no cluster connection"

%7|1670902346.625|CONNECT|rdkafka#producer-1| [thrd:main]: sasl_ssl://broker-2.redacted:443/2: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s))

This makes me confused. Didn't we just try to create a connection to broker 1 above, and it just started to SSL handshaking with broker(without error) + 1 original connection is re-connecting, and now, another now connection created for "no cluster connection"?

Also, sometimes, I can also see multiple new connections created at the same time due to another reason: "refresh unavailable topics".

Question:

1. Why should we create multiple new connections at the same time and all doing the same thing: metadata update?
2. Is this behavior expected or a bug?
3. Are there any configuration could help alleviate this issue?

Env:
confluent-kafka-python 1.9.2 (i.e. librdkafka 1.9.2)
client configs: All default configs + sasl/ssl connection configs
Broker in kafka v3.2.3
server configs: Nothing special except connections.max.reauth.ms=0 (disabled)

Thank you.

[showuon]

@edenhill , any comments?

[edenhill]

It will try multiple brokers to get reconnected to the cluster.

[showuon]

@edenhill , thanks for the comments. So you mean this is working as expected, right? And currently, there's no configuration can help for this issue, right?
Thanks.

[edenhill]

The unused connections will be closed automatically as they reach their idle timeout, either the broker's or the client's, whichever comes first.

[AimerZH]

您好，您的已收到，我会尽快回复