check remaining bytes when marking batch as empty #1192
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes issue-1188 and drastically reduce the number of reader errors and dials. Across a couple hundred reader instances, we saw the number of reader errors drop from a couple hundred per minute to 0.
When hwm==offset and remaining bytes in the buffer is not zero, it means the buffer contains information that has not been replicated yet, as least according to the hwm. However from our testing, this is mostly entirely due to race condition (i.e. we asked for hwm too soon, before the message is replicated.), and hwm always catches up so there doesn't seem to be any harm reading those bytes in the buffer.
Alternatively, we could discard the remaining bytes in the buffer when we mark the batch empty based on
hwm==offset
, similar to this PR.