no encodings/compression when converting parquet file to lance

[shenlei149]

We have a lot of data for training, and now we use parquet format to store them.
Lance says it can support very wide schemas to solve the issue we have to read all metadata from parquet footer even if only few columns projects.

I write the following code to convert parquet file to lance file, but I find the lance file size is very very large. It is almost the same size as the original data and may not use any encoding or compression. Is there something wrong?

import lance
import pyarrow.parquet as pq

src = "A0.parquet"
dst = "A0.lance"

table = pq.read_table(src)
schema = table.schema

lance.write_dataset(table, dst, schema = schema, use_legacy_format=False)

[BubbleCal]

hi @shenlei149
could you share your schema? that would be helpful for investigating!

[shenlei149]

could you share your schema? that would be helpful for investigating!

you can use this example file to reproduce issue.
https://drive.google.com/file/d/1VO2bEW_Q8lJamFQ4Id3WhLsxbua8x2oq/view?usp=drive_link