Is Spark limited to split the Parquet read granularity by Row Group level only?

[felipepessoto]

According to some articles I found:

• https://cloudsqale.com/2021/03/19/spark-reading-parquet-why-the-number-of-tasks-can-be-much-larger-than-the-number-of-row-groups/
• https://www.gresearch.com/news/parquet-files-know-your-scaling-limits/

Seems Spark can only parallelize the reads across row groups. Is this a known limitation?

Is there any way to split it by row or page level?

If a file has a single row group, it means all the tasks, except 1, would be idle? And that one task would read the entire file?

Thanks.

[pan3793]

yes, it's how it works for now. and I can contribute an extremely bad case apache/parquet-java#3235

[felipepessoto]

@pan3793 is the row count limit available in Spark 4.1?

Is there any plan in Spark to support more granular splits? cc @cloud-fan

[pan3793]

is the row count limit available in Spark 4.1?

yes, spark 4.1 uses parquet 1.16.0, which includes the write-side row count limit for row group. you can try it with

spark-sql --conf spark.hadoop.parquet.block.row.count.limit=1000000

[felipepessoto]

I created an item in JIRA: https://issues.apache.org/jira/browse/SPARK-56871