kafka-logger reported an error when sending a message, the error message is "closed".

[renshunhang]

1、This is my route:
{"id":"safe-check-route","name":"safe-check-route","desc":"服务群组转发路由","enable_websocket":true,"uri":"/kunlun/ingress/api/cluster2/87c2743baa81475c9a0b54ecb0b902cd/ai-2eeed9b61bb94bf2b0f32b038ebf80de/*","plugins":{"safe-check-forward":{"name":"safe-check-forward","safe_check":true,"safe_check_upstream":"safe_check_upstream","serving_upstream":"servering_upstream"},"kafka-logger":{"_meta":{"filter":[["http_req_safe_checked","==","false"]]},"brokers":[{"host":"172.31.191.10","port":9092,"sasl_config":{"user":"kaclient","password":"0x1jZTDWGtO10+gsxtNP7s1T1lZNaNmy72QYE9HVBEM="},"ssl":false,"ssl_verify":false}],"kafka_topic":"test-topic","log_format":{"completion_tokens":"$completion_tokens","issuer":"$issuer","prompt_tokens":"$prompt_tokens","request_url":"$request_url","appcode_id":"$appcode_id","src":"$src","service_name":"$service_name","http_code":"$status","tpot":"$tpot","ttft":"$ttft","resp_type":"$resp_type","request_time":"$request_time","request_length":"$request_length","upstream_response_length":"$upstream_response_length","start_time":"$start_time","end_time":"$end_time"}}}}
2、I have enabled SASL authentication for Kafka, and my Kafka configuration is correct with no network connectivity issues to Kafka. However, when configuring the route with kafka-logger to send messages, I'm getting an error:
2025/10/27 09:08:37 [error] 59#59: *13938297 [lua] kafka-logger.lua:209: func(): =====partition_id: 0, context: ngx.timer, client: 10.244.99.64, server: 0.0.0.0:9080
2025/10/27 09:08:37 [error] 59#59: *13938300 [lua] producer.lua:268: buffered messages send to kafka err: closed, retryable: true, topic: test-topic, partition_id: 0, length: 1, context: ngx.timer, client: 10.244.99.64, server: 0.0.0.0:9080

How can I troubleshoot this issue?

[dosubot[bot]]

The "closed" error from kafka-logger means the connection to the Kafka broker was closed before the message could be sent. This can happen due to authentication failures, network interruptions, or misconfiguration of SASL parameters. Even if network connectivity is fine, SASL authentication issues or timeouts can still cause this error.

A few things to check:

• Make sure your Kafka broker's SASL mechanism matches what APISIX expects. As of a recent update, kafka-logger now supports SASL/PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 mechanisms. If your broker uses SCRAM, ensure you specify the correct mechanism in your sasl_config (e.g., "mechanism": "SCRAM-SHA-256"). This support was added very recently (details).
• Double-check your SASL username and password.
• If you use SSL or SASL_SSL, ensure the ssl and ssl_verify options are set correctly and that your broker supports plaintext if you have SSL disabled.
• Review your producer settings like timeout and producer_type—short timeouts or aggressive batching can cause premature connection closure if the broker is slow to respond (docs).
• If you see this error only after some idle time, it could be a keepalive or idle timeout issue on the broker or network path.

If you recently upgraded APISIX, make sure your configuration uses the new mechanism field if your broker requires SCRAM. If you still see the error after these checks, try enabling debug logging for more details on the authentication handshake and connection lifecycle.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[renshunhang]

The "closed" error from kafka-logger means the connection to the Kafka broker was closed before the message could be sent. This can happen due to authentication failures, network interruptions, or misconfiguration of SASL parameters. Even if network connectivity is fine, SASL authentication issues or timeouts can still cause this error.

A few things to check:

• Make sure your Kafka broker's SASL mechanism matches what APISIX expects. As of a recent update, kafka-logger now supports SASL/PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 mechanisms. If your broker uses SCRAM, ensure you specify the correct mechanism in your sasl_config (e.g., "mechanism": "SCRAM-SHA-256"). This support was added very recently (details).
• Double-check your SASL username and password.
• If you use SSL or SASL_SSL, ensure the ssl and ssl_verify options are set correctly and that your broker supports plaintext if you have SSL disabled.
• Review your producer settings like timeout and producer_type—short timeouts or aggressive batching can cause premature connection closure if the broker is slow to respond (docs).
• If you see this error only after some idle time, it could be a keepalive or idle timeout issue on the broker or network path.

If you recently upgraded APISIX, make sure your configuration uses the new mechanism field if your broker requires SCRAM. If you still see the error after these checks, try enabling debug logging for more details on the authentication handshake and connection lifecycle.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other    

The Kafka I use supports both enabling and disabling SSL authentication. If I want to disable SSL authentication in the plugin, how do I configure it

[dosubot[bot]]

To disable SSL authentication in the APISIX kafka-logger plugin, set "ssl": false and "ssl_verify": false in your broker configuration, or simply omit these fields. Your configuration should look like this:

"brokers": [
  {
    "host": "your-kafka-host",
    "port": 9092,
    "sasl_config": {
      "user": "your-username",
      "password": "your-password",
      "mechanism": "PLAIN" // or SCRAM-SHA-256/SCRAM-SHA-512 if needed
    },
    "ssl": false,
    "ssl_verify": false
  }
]

This tells APISIX to connect to Kafka over plaintext (non-SSL). Make sure your Kafka broker is listening on a plaintext port and allows SASL authentication without SSL. You can find more details and examples in the official documentation (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[renshunhang]

Still reporting the above error

[dosubot[bot]]

Since the error persists after disabling SSL, the root cause is likely related to SASL authentication or a protocol mismatch. The kafka-logger plugin recently added support for SCRAM-SHA-256 and SCRAM-SHA-512 mechanisms—if your Kafka broker uses one of these, you must explicitly set the correct mechanism in your sasl_config (e.g., "mechanism": "SCRAM-SHA-256" or "SCRAM-SHA-512"). If you’re using the latest APISIX, double-check that your configuration matches your broker’s SASL mechanism and that your broker allows SASL/PLAIN or SCRAM over plaintext connections. Also, confirm that your username and password are correct and that the broker logs show successful authentication attempts (not just network connectivity) source.

If you’re still seeing “closed” errors, try enabling debug logging in APISIX to capture more details about the authentication handshake and connection lifecycle. Broker-side logs can also help pinpoint whether authentication is failing or the connection is being closed for another reason.

Let me know your APISIX version, the exact sasl_config you’re using, and what your broker logs show during a failed attempt—this will help narrow it down further.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[renshunhang]

Connecting to Kafka does not report any errors, and topics can be created successfully, but calling the send method results in an error. The error message is as follows:
batch-processor.lua:96: Batch Processor[kafka logger] failed to process entries: failed to send data to Kafka topic: closed.