Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mitigating blocking IO in highly batched payloads #441

Open
trueleo opened this issue Jun 22, 2023 · 0 comments
Open

Mitigating blocking IO in highly batched payloads #441

trueleo opened this issue Jun 22, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@trueleo
Copy link
Contributor

trueleo commented Jun 22, 2023

Although not fully pinpointed there are few indications is that blocking writes to disk is causing lesser performance that could otherwise be achieved in similar test scenarios.

Primarily StreamWriter is blocking in nature. This should change anyways as IO ops should mostly be async.
There are few considerations for solving this issue.

  • The obvious one is working with arrow-rs community to bring this to arrow-ipc directly. This is most likely is the best thing to do.
  • Or we can mitigate this on our side by keeping a separate thread pool primarily for making blocking calls to disk.
@trueleo trueleo changed the title Mitigating blocking IO in high batching payloads Mitigating blocking IO in highly batched payloads Jun 22, 2023
@trueleo trueleo self-assigned this Jun 22, 2023
@nitisht nitisht added the enhancement New feature or request label Jun 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants