You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Although not fully pinpointed there are few indications is that blocking writes to disk is causing lesser performance that could otherwise be achieved in similar test scenarios.
Primarily StreamWriter is blocking in nature. This should change anyways as IO ops should mostly be async.
There are few considerations for solving this issue.
The obvious one is working with arrow-rs community to bring this to arrow-ipc directly. This is most likely is the best thing to do.
Or we can mitigate this on our side by keeping a separate thread pool primarily for making blocking calls to disk.
The text was updated successfully, but these errors were encountered:
trueleo
changed the title
Mitigating blocking IO in high batching payloads
Mitigating blocking IO in highly batched payloads
Jun 22, 2023
Although not fully pinpointed there are few indications is that blocking writes to disk is causing lesser performance that could otherwise be achieved in similar test scenarios.
Primarily StreamWriter is blocking in nature. This should change anyways as IO ops should mostly be async.
There are few considerations for solving this issue.
The text was updated successfully, but these errors were encountered: