idea: Add parquet arrow async write/read support

[Xuanwo]

We can implement native parquet arrow support to make our users happy:

• Read: https://docs.rs/parquet/latest/parquet/arrow/async_reader/trait.AsyncFileReader.html
• Write: https://docs.rs/parquet/latest/parquet/arrow/async_writer/trait.AsyncFileWriter.html

By implementing this integration, users can avoid using the low-performance AsyncRead/AsyncWrite trait or creating their own shims.

Steps:

• feat: introduce opendal AsyncWriter for parquet integrations  #4958
• feat: introduce opendal AsyncReader for parquet integrations #4972
• chore(integrations/parquet): add README #4980

[WenyXu]

Interesting, I'm going to have a try🤩

[Xuanwo]

Interesting, I'm going to have a try🤩

Perfect!

[Xuanwo]

Perhaps we could have an integrations/parquet.

[WenyXu]

I'm considering introducing multiple versions of parquet via

[dependencies]
parquet_51 = { package = "parquet", version = "51.0", optional: true }
parquet_52 = { package = "parquet", version = "52.0", optional: true }

Due to we are still using parquet 51.0.0 🥲. Would you like to have any ideas? cc @waynexia

[Xuanwo]

Since arrow's release going to be stable, maybe we can keep track with upstream instead?

[WenyXu]

Since arrow's release going to be stable, maybe we can keep track with upstream instead?

Make sense🥹

[WenyXu]

For AsyncFileReader, I am considering introducing a feature to merge small ranges into a large chunk(e.g., https://github.com/datafuselabs/databend/blob/a98335d33e7abfd34189e7f32c06ab34d53c64d0/src/query/storages/fuse/src/io/read/block/block_reader_merge_io_async.rs#L46-L68). Would you like to have any ideas?

[waynexia]

Depends on multiple versions of parquet may bring lots of burdens on maintenance, especially when the versions increase. This is usually achieved via maintaining multiple branches and are released separately to migrate the API change of parquet, which doesn't seems to match our situation at present neither. So I prefer not to have this at least at beginning. We can reconsider this when there's a real requirement like a specific version of parquet is LTS and used widely.

[Xuanwo]

For AsyncFileReader, I am considering introducing a feature to merge small ranges into a large chunk(e.g., datafuselabs/databend@a98335d/src/query/storages/fuse/src/io/read/block/block_reader_merge_io_async.rs#L46-L68). Would you like to have any ideas?

Seems we can use Reader::fetch internally.

[WenyXu]

Seems we can use Reader::fetch internally.

Cool!