Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #307. Should be replaced in favor of #262 (but my PR might be used to fix the Kafka sink).
I was able to get stable performance during snapshots and huge single statement updates:
LinkedBlockingQueue
(which is also thread-safe) with limit of 200,000 records instead ofConcurrentLinkedQueue
. When a queue becomes full thread sleeps in a loop until space in the queue is freed. I didn't use @IlyaTsoi solution sinceConcurrentLinkedQueue::size
is an O(n) (i.e. quite slow for our purpose).What I don't like here:
I've just recently stumbled upon #262 which in theory should be a perfect solution for the problem and also fixes #342. Alas the PR only solves the problems for the lightweight sink. That is not a problem for me, but looks like the Kafka sink suffers from the same problems, the problematic parts of the code are the same.