You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Motivation: The current behavior of ParquetFilterReader#readNextRowGroup is to eagerly enumerate all chunks in the row group, then read all pages in the chunk. For distributed data workloads, this can cause significant memory pressure, particularly for use cases that require the colocation of multiple Parquet files on a single worker.
Proposal: A Parquet Configuration option that enables lazy row group reading, i.e., only a page at a time (plus whatever header is necessary to read that header). The Configuration option could be either a flag, or an int value for how many pages/page bytes to buffer at a time.
I think this could be accomplished by modifying ParquetFileReader#readAllPages to re-implement pagesInChunk as an Iterator, rather than a List. Then, ColumnChunkPageReader could parse the Configuration option above and decide whether to fully materialize the iterator or not.
I'm happy to try to create a draft/branch for this to get some early feedback on the idea!
However... it looks like ColumnReaderBase#checkRead works by continually invoking readPageuntil the row group is fully consumed, so the row group effectively gets materialized there, even if my ColumnChunkPageReader is now backed by lazy Iterator. Any pointers on how I should modify that code block?
Claire McGinty / @clairemcginty:
Thanks Gang! I'll check out the Iceberg pattern. My implementation was as a java Iterator, which is a bit tricky because a chunk is a mixture of dictionary+data pages, so there are some awkward workarounds.
Motivation: The current behavior of ParquetFilterReader#readNextRowGroup is to eagerly enumerate all chunks in the row group, then read all pages in the chunk. For distributed data workloads, this can cause significant memory pressure, particularly for use cases that require the colocation of multiple Parquet files on a single worker.
Proposal: A Parquet Configuration option that enables lazy row group reading, i.e., only a page at a time (plus whatever header is necessary to read that header). The Configuration option could be either a flag, or an int value for how many pages/page bytes to buffer at a time.
I think this could be accomplished by modifying ParquetFileReader#readAllPages to re-implement pagesInChunk as an Iterator, rather than a List. Then, ColumnChunkPageReader could parse the Configuration option above and decide whether to fully materialize the iterator or not.
I'm happy to try to create a draft/branch for this to get some early feedback on the idea!
Reporter: Claire McGinty / @clairemcginty
PRs and other links:
Note: This issue was originally created as PARQUET-2443. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: