-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there any actual conversion implementation for arrow and parquet? #2928
Comments
If possible to use C++, I think parquet-cpp in the Apache Arrow is the best solution to your case: https://arrow.apache.org/docs/cpp/parquet.html |
Yes, We know there is a c++ implementation here, but I was wondering if there is a corresponding implementation for java, since all our code is java . |
parquet-arrow The library looks like it's meant to do this, But I don't know why it's always just the schema part. |
I think conversion between parquet and arrow is a valid use case. The parquet-java provides built-in row-level interfaces like avro/thrift/protobuf. Other parquet (Java) implementations (Presto/Trino/Spark) simply leverage the page & metadata reader/writer from this library to build extensions. Extending native arrow support would be a welcome extension to this library, IMO. |
So, the library parquet-arrow hasn't been used yet? because it only has schema mappings. And We've looked at transformations in Spark, which are missing some types due to limitations in Spark, such as uint.So it's not really a universal conversion. |
We found that there are only schema conversions under the parqeut-arrow, so we wanted to ask if there is any code that includes the actual data conversions between parquet and arrow.
The text was updated successfully, but these errors were encountered: