Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove unnecessary offset fetch during start of Consumer #2454

Open
sehz opened this issue Jul 1, 2022 · 1 comment
Open

Remove unnecessary offset fetch during start of Consumer #2454

sehz opened this issue Jul 1, 2022 · 1 comment

Comments

@sehz
Copy link
Contributor

sehz commented Jul 1, 2022

When consumer starts, it performs FetchOffsetsRequest in order to map logical offset (beginning, end, etc) to physical offset. This can be performed in the SPU itself instead of client reduces round trip as make it more reliable.

https://github.com/infinyon/fluvio/blob/master/crates/fluvio/src/consumer.rs#L290

@galibey
Copy link
Contributor

galibey commented Jul 5, 2022

It turned out, that the fix is much more complicated than it was supposed to be. The absolute offset received by FetchOffsetsRequest is used by (among other things) records filtering on the client-side. This filtering is needed because the server always sends back data by batch granularity (due to zero-copy responses). If the requested starting offset is in the middle of the batch, the whole batch will be sent to the client and the client code will filter out records before it sends to the end user's code. This only matters for the first batch in the stream but can't be ignored or workarounded somehow.

We could also add a response for StreamFetchRequest (currently, we do not expect a response for this request) and put resolved started offset there but it requires many changes in different places. The current flow of stream creation is generic and not specific to StreamFetchRequest. The gain from the fix seems less than efforts for refactoring, in my opinion.

@sehz sehz removed this from the 0.9.31 milestone Jul 5, 2022
@galibey galibey removed their assignment May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants