Repository navigation
feat: DH-24165: Added sparse/dense parquet benchmarks - #444
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the sparse and dense Parquet read benchmark changes.
No open issues 🎉
What changed in this PR
Adds sparse and dense Parquet read benchmarks to detect regressions in selective row decoding, using the existing ParquetMultiColTest benchmark suite.
Changes:
- Adds a filtered-read helper that filters on
long10K, selects requested columns, and reports throughput in file rows per second. - Adds 1% sparse and 50% dense reads of the existing Snappy Defaults file, excluding array columns.
- Makes scale-factor setup explicit per test and reduces the Gzip tests’ scale factor.
| File | Description |
|---|---|
| src/it/java/io/deephaven/benchmark/tests/standard/file/ParquetMultiColTest.java | Adds sparse/dense benchmark cases and per-test scale-factor setup. |
| src/it/java/io/deephaven/benchmark/tests/standard/file/FileTestRunner.java | Adds the reusable filtered Parquet read benchmark helper. |
Review effort: Balanced
cpwright
reviewed
Nov 3, 2026
cpwright
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DH-23641 (deephaven-core #8878) decodes only the requested rows when a Parquet read touches a small fraction of a page. No nightly or release benchmark exercises that path: standard tests
select()their source before timing, and the ParquetRead tests time full reads. This adds one sparse benchmark and a dense control toParquetMultiColTest, so a later change can't regress the path unnoticed.FileTestRunner.runParquetFilteredReadTest: reads the file, filters onlong10K, then selectsstr10K,int10K,short10KandbigDec10Kfor the surviving rows. The rate is file rows per second, like the other read tests.ParquetRead- Sparse Multi Col -Statickeeps 1 in 100 rows;ParquetRead- Dense Multi Col -Statickeeps 1 in 2. The PR reads a page sparsely when under 12.5% of it is requested. The file's pages hold at least 5,184 rows for these columns, so the two tests fall clearly on either side of that cutoff.ParquetMultiColTest.Test Completed
b7ed9cb7c8vs #8878243a7f5b8e, 90M rows, 5 runs each, alternating build order.