Open
Description
Although not fully pinpointed there are few indications is that blocking writes to disk is causing lesser performance that could otherwise be achieved in similar test scenarios.
Primarily StreamWriter is blocking in nature. This should change anyways as IO ops should mostly be async.
There are few considerations for solving this issue.
- The obvious one is working with arrow-rs community to bring this to arrow-ipc directly. This is most likely is the best thing to do.
- Or we can mitigate this on our side by keeping a separate thread pool primarily for making blocking calls to disk.