MSK - KafkaConsumer failing to connect silently

[dirceumaranhaopn]

I think I found an issue connecting to MSK.

In the Kafka-python documentation, it states that if at least one host of the connection is up and able to respond to the Metadata API Request, the connection should be fine.

However, connection is failing silently; I only noticed because messages were not being consumed.

Does that relate to #2390 and #2281?

Kafka-python 2.0.2, consumer is running on a docker image hosted in an AWS EC2.
My configuration is straightforward:
consumer = KafkaConsumer( 'mytopic', group_id=group_id, bootstrap_servers=['host1:port1','host2:port2','host3:port3','host4:port4'], security_protocol='SSL' )
This works when all hosts are healthy but fails if there is a typo on the hostname or a host is unhealthy.

For this bug, I would expect that connection to succeed despite one host not being accessible as long at least of the list it is. And in the worst case scenario a failure to connect.

[stephenhutch]

Did you enable DEBUG logs?

[dirceumaranhaopn]

Hi @stephenhutch I honestly don't remember, I will check tomorrow at work, but even if the message is on the debug logs I am failing to see why that would not be a hard stop error. If you cannot connect it should let you know IMO.

[zuerst]

AWS officially released msk ism sasl singer and it has example using dpkp/kafka-python. Take a look at this

[dirceumaranhaopn]

Hi @zuerst and @stephenhutch, this issue still needs to be solved entirely, but we have used some workarounds. So, I will close it.

We are checking to see if all hosts on the list are healthy before supplying them.

Thank you for your time.

[dpkp]

You should always pass api_version as a configuration to production consumers. Otherwise you're relying on dynamic version checking which blocks and will fail if brokers are not immediately available.