🔥 Critical Performance Issue: partitionsConsumedConcurrently > 1 Causes CPU Overload Without Consumer Lag

[maksym-opanasenko-ft]

Environment Information

• Reproduced on Mac (silicone) and confirmed in AWS ECS:
• Node Version: v18.20.4
• NPM Version: 10.7.0
• Confluent Platform : 7.7.x, 7.5.x, 7.3.x; also confirmed with Apache Kafka 3.3.x
• confluent-kafka-javascript version: ^0.5.2

Steps to Reproduce

1. Launch Kafka in docker or in AWS MSK
2. Create a topic with many partitions
3. Instantiate a consumer using the kafkaJs constructor:

	const consumer = kafka.consumer({
		kafkaJS: {
			groupId: 'test-group',
		},
	});

4. Run the consumer with partitionsConsumedConcurrently > 1

	await consumer.run({
		partitionsConsumedConcurrently: 2, // This causes the CPU load
		eachMessage: async ({ topic, partition, message }) => {
			console.log({
				topic,
				partition,
				offset: message.offset,
				value: message.value.toString(),
			});
		},
	});

5. Don't write any messages to the topic or wait for the consumer to catch up with the lag
6. Wait for at least 30 seconds and observe the CPU usage

confluent-kafka-javascript Configuration Settings

Here is the full minimal snippet we use to reproduce:

const { Kafka } = require('@confluentinc/kafka-javascript').KafkaJS;

const topic = 'TestTopic';
const clientId = 'test-client-id';

const kafka = new Kafka({
	kafkaJS: {
		clientId: clientId,
		brokers: ['localhost:9092'],
		logLevel: 4, // optional, but gives a hint what overuses the CPU
	},
});

const consumerRun = async () => {
	const consumer = kafka.consumer({
		kafkaJS: {
			groupId: 'test-group',
		},
	});

	await consumer.connect();
	console.log('Consumer connected');

	await consumer.subscribe({ topic });
	console.log('Consumer subscribed to topic');

	await consumer.run({
		partitionsConsumedConcurrently: 2, // This causes the CPU load
		eachMessage: async ({ topic, partition, message }) => {
			console.log({
				topic,
				partition,
				offset: message.offset,
				value: message.value.toString(),
			});
		},
	});
};

consumerRun().catch((err) => {
	console.error('Error running producer or consumer:', err);
});

Additional context

When setting logLevel: 4 in the Kafka constructor and using partitionsConsumedConcurrently: 1, the following log message

{
  message: 'Attempting to fetch 1 messages to the message cache',
  name: 'test-client-id#consumer-1',
  fac: 'BINDING',
  timestamp: 1733298033731
}
```  is printed ~ once in a second, however, when setting `partitionsConsumedConcurrently: 2` the console is spammed with the message. 
Setting `logLevel: 3` makes it better as IO is used less, but the code that prints the message is still running causing high usage of CPU

[larsha]

We're experiencing the same issue, we have multiple consumers that don't perform CPU intensive work, but we have topics with around 100 partitions so we have been using partitionsConsumedConcurrently with pretty high numbers in KafkaJS with great success. But just migrating and using same high numbers (10, 25, 5 etc) we're seeing issues with CPU, GC durations skyrocketing and that's basically when the consumers are idle. When changing it back to 1 again, everything is super cool and steady as you can see on the graphs.

I have also been testing out different high values of UV_THREADPOOL_SIZE but can't say I noticed any difference, but will keep on testing it.

[ft-vladislav-hadzhiyski]

Log level 4 is DEBUG so this could be easily mitigated. Why would an app using Kafka care about debug messages anyways?

const logLevel = {
  NOTHING: 0,
  ERROR: 1,
  WARN: 2,
  INFO: 3,
  DEBUG: 4,
};

[maksym-opanasenko-ft]

Log level 4 is DEBUG so this could be easily mitigated. Why would an app using Kafka care about debug messages anyways?

const logLevel = {
  NOTHING: 0,
  ERROR: 1,
  WARN: 2,
  INFO: 3,
  DEBUG: 4,
};

That's true, however, logging isn't the root cause here. It only gives a hint at what code is being called too often.

[troch]

Hello, we also have observed CPU usage issues after migrating from kafkajs (CPU 100%), changing partitionsConsumedConcurrently to 1 fixes it but is not a fix we can apply to production.

We observe the same symptoms described by @maksym-opanasenko-ft: Attempting to fetch 1 messages to the message cache happening every second, and then going out of control (multiple times per ms).

[AntonPuko]

Hi, we have the same issue :(

[emasab]

Thanks @maksym-opanasenko-ft for reporting this issue. Fixing it in #300.