You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm unable to create a bloomfilter for a field, when I perform writes with repeating values. The bloomfilter returned is null when I try to read such a parquet file. If there are no repeating values, the bloomfilter is created without any issue.
The working and non-working case in captured in the below repo
Gang Wu / @wgtmac:
By default, bloom filter will be disabled if dictionary encoding is applied. Please set withDictionaryEncoding(false) while creating the parquet writer and check if this issue still happens.
Mahesh Pai:
Thanks @wgtmac!
When I disable dictionary encoding the bloomfilter is getting created. But I'm curious as to why this works when distinct values are written. With distinct values and dictionary encoding enabled, the writer is generating the bloomfilter
I'm unable to create a bloomfilter for a field, when I perform writes with repeating values. The bloomfilter returned is null when I try to read such a parquet file. If there are no repeating values, the bloomfilter is created without any issue.
The working and non-working case in captured in the below repo
https://github.com/MaheshGPai/parquet-mr-test
https://github.com/MaheshGPai/parquet-mr-test/blob/main/src/test/java/com/mahesh/test/AppTest.java#L73
Reporter: Mahesh Pai
Note: This issue was originally created as PARQUET-2484. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: