Skip to content

Mitigating blocking IO in highly batched payloads #441

Open
@trueleo

Description

@trueleo

Although not fully pinpointed there are few indications is that blocking writes to disk is causing lesser performance that could otherwise be achieved in similar test scenarios.

Primarily StreamWriter is blocking in nature. This should change anyways as IO ops should mostly be async.
There are few considerations for solving this issue.

  • The obvious one is working with arrow-rs community to bring this to arrow-ipc directly. This is most likely is the best thing to do.
  • Or we can mitigate this on our side by keeping a separate thread pool primarily for making blocking calls to disk.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions