Understand relations among: kafka client, number of topics, consumer groups, manual commit and threads

[gianluca-sabena]

Dear All,
I need to develop an application with this use case:

• accept http subscription from a client with 3 parameters: kafka-topic, http-callback-url, client-id
• create a kafka client with a specific consumer group for any client-id
• the applicationreceives messages from kafka-topic for a specific client-id, sends this message to an http endpoint (http-callback) and manual commit offset when http client send the http response
• the application must subscribe to many topics with different consumer-groups and manage manual offset commit.

This is a simplified version (no grpc, no out-of-order commit) of what uber has developed with https://github.com/uber/uForwarder

Scalability is designed this way:

• any app instance subscribes to all topic
• any app instance subscribe to 1 or more partitions (consumer group are spread across different instances)
• es topic-A#partition-0 -> app-instance-1 topic-A#partition-1 -> app-instance-2
• the application scales based on topic partition and is independent from the number and subscription coming from the http client side

I need to better understand:

• best approach to implement this with spring-kafka
• scalability and performance impacts and limits (cpu/memory) for a single app instance when the number of subscription increase (50 topic, 20 client/consumer-group.... 100 topic + 50 client/consumer-group...)
• What is the relation between kafka client, topics, consumer group and number of threads?
• best approach to scale this application on different instances
• Is it possible to use a thread-pool to segregate kafka-consumer workload?
• Does virtual threads may help in this scenario?

Thank you for your help!

[gianluca-sabena]

The original question can be simplified: "How can I mange hundreds of kafka clients efficiently?"