Confluent Kafka Producer Error - Request Timed out and Timed out MetadataRequest in flight #4039
Replies: 2 comments 5 replies
-
|
This is most likely an issue with Azure's load balancers that silently close idle connections. Try configuring ConnectionsMaxIdleMs to a value below what the load balancer timeout is, about a year ago it was 4 minutes, but we can see here that the connection is closed almost exactly after 10 minutes, so maybe go with that?
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick reply @edenhill. Definitely will try it out and see the differences. Will update you on the result |
Beta Was this translation helpful? Give feedback.
-
|
@edenhill i referred the below link the timeout error has gone after adding the below producer configuration properties I have implemented Producer Log Handler and i see the below error. Exactly every 5 min am seeing this error Producer Log Handler : ERROR|30-10-2022T04:35:18.9967269Z|ERROR|rdkafka#producer-1|[[thrd:sasl_ssl://[EH-NAME-SPACE].servicebus.windows.net:9093/11]: 16/16 brokers are down] Producer Log Handler : ERROR|30-10-2022T04:40:18.9942533Z|ERROR|rdkafka#producer-1|[[thrd:sasl_ssl://[EH-NAME-SPACE].servicebus.windows.net:9093/6]: 16/16 brokers are down] Producer Log Handler : ERROR|30-10-2022T04:45:18.9971873Z|ERROR|rdkafka#producer-1|[[thrd:sasl_ssl://[EH-NAME-SPACE].servicebus.windows.net:9093/12]: 16/16 brokers are down] Producer Log Handler : ERROR|30-10-2022T04:50:18.9942151Z|ERROR|rdkafka#producer-1|[[thrd:sasl_ssl://[EH-NAME-SPACE].servicebus.windows.net:9093/9]: 16/16 brokers are down] Producer Log Handler : ERROR|30-10-2022T05:00:27.2012375Z|ERROR|rdkafka#producer-1|[[thrd:sasl_ssl://[EH-NAME-SPACE].servicebus.windows.net:9093/3]: 16/16 brokers are down] Could you direct me to the right path. Appreciate your valuable inputs |
Beta Was this translation helpful? Give feedback.
-
|
Configure |
Beta Was this translation helpful? Give feedback.
-
|
@edenhill Thanks for your input. ConnectionsMaxIdleMs set as 180000 (3 Min). still i was getting the same error. Appreciate your valuable inputs |
Beta Was this translation helpful? Give feedback.
-
|
When all broker connections are down librdkafka will log this ".. brokers are down" error, and this is likely to happen with aggressive load balancers if the producer is only connected to a single broker (because only producing to a single partition, for instance). I think your best option is to filter out these log messages in your log handler. |
Beta Was this translation helpful? Give feedback.
-
|
I got a coredump when I faced this #1 0x00007f9e808b5a7b in rd_kafka_log_buf (buf=0x7f9d8dfedee0 "[thrd:11.145.17.95:9092/bootstrap]: 11.145.17.95:9092/bootstrap: Timed out ApiVersionRequest in flight (after 10008ms, timeout #0)", fac=0x7f9e80a4fee5 "REQTMOUT", level=5, rk=0x247e880, conf=0x247e9b8) at rdkafka.c:247 #2 rd_kafka_log0 (conf=0x247e9b8, rk=0x247e880, extra=0x7f9d8dfee870 "11.145.17.95:9092/bootstrap", level=5, fac=0x7f9e80a4fee5 "REQTMOUT", fmt=) at rdkafka.c:287 #3 0x00007f9e808c128d in rd_kafka_broker_bufq_timeout_scan (rkb=0x2480460, is_waitresp_q=1, rkbq=0x24807a8, partial_cntp=0x0, ApiKey=, err=RD_KAFKA_RESP_ERR__TIMED_OUT, now=104009242698, description=0x7f9e80a4f6b1 "in flight", log_first_n=5) at rdkafka_broker.c:704 #4 0x00007f9e808c3644 in rd_kafka_broker_timeout_scan (rkb=0x2480460, now=104009242698) at rdkafka_broker.c:737 #5 0x00007f9e808cecec in rd_kafka_broker_ops_io_serve (rkb=0x2480460, abs_timeout=) at rdkafka_broker.c:3048 #6 0x00007f9e808cfb30 in rd_kafka_broker_producer_serve (abs_timeout=104009241856, rkb=0x2480460) at rdkafka_broker.c:3595 #7 rd_kafka_broker_serve (rkb=0x2480460, timeout_ms=) at rdkafka_broker.c:4388 #8 0x00007f9e808d012d in rd_kafka_broker_thread_main (arg=0x2480460) at rdkafka_broker.c:4523 #9 0x00007f9e8090f2bf in _thrd_wrapper_function (aArg=) at tinycthread.c:576 #10 0x00007f9e89a84ea5 in start_thread () from /lib64/libpthread.so.0 #11 0x00007f9e884e09fd in clone () from /lib64/libc.so.6 |
Beta Was this translation helpful? Give feedback.
-
For some reason kafka producer is throwing the below error while trying to produce a method
Am using the Confluent Kafka Package with version 1.8.2 to connect to Eventhub.
Below are the producer config for reference.
BootstrapServers = [EH-NAME-SPACE].servicebus.windows.net:9093,
SaslUsername = "$ConnectionString",
SaslPassword = <EH-SHARED_ACCESS-POLICY-PRIMARY-CONNETCION-STRING>,
SaslMechanism = SaslMechanism.Plain,
SecurityProtocol = SecurityProtocol.SaslSsl
MessageMaxBytes = 20971520,
MessageCopyMaxBytes = 20971520,
LogConnectionClose = false,
SocketKeepaliveEnable = false,
BrokerAddressFamily = BrokerAddressFamily.V4,
ConnectionsMaxIdleMs = 0,
Acks = Acks.Leader,
BatchSize = 200000,
LingerMs = 0,
CompressionType = CompressionType.None
SocketNagleDisable = true,
Below are the logs for the reference,
%5|1666936636.307|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out MetadataRequest in flight (after 60039ms, timeout #0)
%4|1666936636.307|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1666936636.307|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 355567ms in state UP)
←[40m←[32minfo←[39m←[22m←[49m: Logger[0]
Producer Error Handler: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 355567ms in state UP)
%5|1666936636.314|REQTMOUT|rdkafka#producer-2| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out MetadataRequest in flight (after 60029ms, timeout #0)
%4|1666936636.314|REQTMOUT|rdkafka#producer-2| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1666936636.314|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 354737ms in state UP)
←[40m←[32minfo←[39m←[22m←[49m: Logger[0]
Producer Error Handler: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 354737ms in state UP)
%5|1666937236.305|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out MetadataRequest in flight (after 60037ms, timeout #0)
%4|1666937236.305|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1666937236.305|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/bo]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 598972ms in state UP, 1 identical error(s) suppressed)
←[40m←[32minfo←[39m←[22m←[49m: Logger[0]
Producer Error Handler: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 598972ms in state UP, 1 identical error(s) suppressed)
%5|1666937236.318|REQTMOUT|rdkafka#producer-2| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10: Timed out MetadataRequest in flight (after 60033ms, timeout #0)
%4|1666937236.318|REQTMOUT|rdkafka#producer-2| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1666937236.318|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10]: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10: 1 request(s) timed out: disconnect (after 598996ms in state UP)
←[40m←[32minfo←[39m←[22m←[49m: Logger[0]
Producer Error Handler: sasl_ssl://<PRODUCER_EH_NAMESPACE>.servicebus.windows.net:9093/10: 1 request(s) timed out: disconnect (after 598996ms in state UP)
%5|1666937470.520|REQTMOUT|rdkafka#consumer-3| [thrd:sasl_ssl://<CONSUMER_EH_NAMESPACE>.servicebus.windows.net:9093/bootst]: sasl_ssl://<CONSUMER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out FetchRequest in flight (after 60882ms, timeout #0)
%4|1666937470.520|REQTMOUT|rdkafka#consumer-3| [thrd:sasl_ssl://<CONSUMER_EH_NAMESPACE>.servicebus.windows.net:9093/bootst]: sasl_ssl://<CONSUMER_EH_NAMESPACE>.servicebus.windows.net:9093/0: Timed out 1 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1666937470.520|FAIL|rdkafka#consumer-3| [thrd:sasl_ssl://<CONSUMER_EH_NAMESPACE>.servicebus.windows.net:9093/bootst]: sasl_ssl://<CONSUMER_EH_NAMESPACE>.servicebus.windows.net:9093/0: 1 request(s) timed out: disconnect (after 1193914ms in state UP)
The above error is happening while running the producer in AKS. The same is not happening while running the executable locally on my machine.
Any help much appreciated.
Beta Was this translation helpful? Give feedback.
All reactions