Add ipc_compression feature to arrow dependency

[binary-signal]

by adding the ipc_compression fluss-rust client can read zstd compressed data from log tables

[luoyuxia]

two question:
1: will it automatically decompress data when read to arrow record batch? Do we need to introduce decompress code logic?
2: will it handle all compress type that fluss supports, like ltz?

[binary-signal]

@luoyuxia
1 → Yes, Arrow automatically decompresses data when reading into a RecordBatch.

From the Arrow IPC docs: https://arrow.apache.org/rust/arrow_ipc/index.html

The Arrow IPC format defines how to read/write RecordBatches to/from files or byte streams. It handles serialization and deserialization.

2 → Arrow IPC supports LZ4 and ZSTD.
In the Fluss docs, I've also seen support for LZ4 and ZSTD:

table.log.arrow.compression.type can be NONE, LZ4_FRAME, or ZSTD (default: ZSTD).

Decompression happens at a lower layer than the Fluss logical types which is transparent for the most part, meaning it can handle all types the arrow supports. For the non-standard types like ltz (I assume you mean TIMESTAMP_LTZ), this must be handled in the Rust binding code since you need to parse the metadata stored in the arrow timestamp which has information about the precision and the timezone. I already have another PR (not submitted) adding experimental support for ltz + timestamps.

But also found PR #53 , which implements all types including ltz timestamp. Since this is more holistic PR than what I was planning to submit, it’s worth reviewing and merging this PR instead for adding support for ltz.

[luoyuxia]

@binary-signal Thanks for explantion. Make sense to me

[luoyuxia]

+1