Batch Kafka consumer

[z0isch]

Add ability to handle batches for our kafka consumers

[z0isch]

Is this the right shape for the handler? Should it be [a] -> m () instead?

Seems like it'd more flexible for the client to do "smart" things if we hand them the entire batch. I'm thinking things like doing concurrent actions or grouping stuff together.

[pbrisbin]

Good question. I defer to you.

Practically speaking, I'd put my money on consumers not doing smart things, but it's also trivial for them to do their own traverse_.

[z0isch]

I think [a] -> m () also implies that we can't store or commit offsets on each event, so I guess for the first attempt at this I can go with the a -> m () shape and see what happens.

[pbrisbin]

So this re-introduces some duplicate processing. If we're deploying, a second consumer can come up and would reprocess any messages the existing consumers have stored but not yet committed, which is up to batchSize messages.

Seems like an OK middle-ground with how it was before. However, I still feel like we could've arrived at the same rough behavior in this regard if we had simply reduced the auto-commit setting to do so more frequently, no?

[pbrisbin]

If we wanted to keep the reduction in duplicate processing, I think still doing a store+commit on every message, but now with batching, could also help things... since you're polling less frequently it would take pressure off the broker even though you're committing more frequently. WDYT?

[z0isch]

If we wanted to keep the reduction in duplicate processing, I think still doing a store+commit on every message, but now with batching, could also help things... since you're polling less frequently it would take pressure off the broker even though you're committing more frequently. WDYT?

Probably worth a try to see what happens; however a client could still get this behavior by having batchSize=1 right?

edit: Oh I guess we'd not get the poll reduction then

[z0isch]

Closing this as the performance is back to normal with doing auto-commits for the answers consumer.

It seems like there is just a trade off on size of the batch of offset commits vs. reprocessing events. For our current consumer re-processing is fine due to the idempotency of the consumer.

It also seems like there is just going to be a performance cost to pay when deploying new consumers that is un-related to reprocessing events. The performance hit seems likely due to the kafka rebalance when adding/removing consumers to a consumer group and not due to reprocessing a handful (100s of events).

[pbrisbin]

It also seems like there is just going to be a performance cost to pay when deploying new consumers that is un-related to reprocessing events

Did slowing the rotation from double-then-halve to plus-one-minus-one solve it?

[z0isch]

It also seems like there is just going to be a performance cost to pay when deploying new consumers that is un-related to reprocessing events

Did slowing the rotation from double-then-halve to plus-one-minus-one solve it?

Nope, it just made it have 3 blips instead of one. I think this makes sense because there was probably 3 re-balances

AFAICT it seems like the only way to avoid these performance issues is to not rebalance which I think implies doing some sort of static partition assignment for the consumers. This would require a lot of complexity in coordination between the consumers though, so I think we'd need to weigh that against how important it is to not have these spikes.