Skip to content

feat: DH-24165: Added sparse/dense parquet benchmarks - #444

Merged
stanbrub merged 5 commits into
deephaven:mainfrom
stanbrub:sparse-parquet-benchmark
Nov 4, 2026
Merged

stanbrub merged 5 commits into
deephaven:mainfrom
stanbrub:sparse-parquet-benchmark

Conversation

@stanbrub

Copy link
Copy Markdown
Collaborator

DH-23641 (deephaven-core #8878) decodes only the requested rows when a Parquet read touches a small fraction of a page. No nightly or release benchmark exercises that path: standard tests select() their source before timing, and the ParquetRead tests time full reads. This adds one sparse benchmark and a dense control to ParquetMultiColTest, so a later change can't regress the path unnoticed.

  • FileTestRunner.runParquetFilteredReadTest: reads the file, filters on long10K, then selects str10K, int10K, short10K and bigDec10K for the surviving rows. The rate is file rows per second, like the other read tests.
  • ParquetRead- Sparse Multi Col -Static keeps 1 in 100 rows; ParquetRead- Dense Multi Col -Static keeps 1 in 2. The PR reads a page sparsely when under 12.5% of it is requested. The file's pages hold at least 5,184 rows for these columns, so the two tests fall clearly on either side of that cutoff.
  • The array columns are left out because the PR doesn't read repeated columns sparsely; they would only dilute the result.
  • Both tests read the Snappy Defaults file written earlier in the class, and they run in the nightly with the rest of ParquetMultiColTest.

Test Completed

  • B/A on the nightly server: main b7ed9cb7c8 vs #8878 243a7f5b8e, 90M rows, 5 runs each, alternating build order.
  • Every #8878 run beat every main run on the sparse test (6.91 s → 3.16 s).
  • On main, keeping 1% of rows costs as much as keeping 50%, because pages are decoded whole either way.
  • The existing reads and writes in the class showed no consistent change.

@stanbrub stanbrub self-assigned this Oct 29, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the sparse and dense Parquet read benchmark changes.

No open issues 🎉

What changed in this PR

Adds sparse and dense Parquet read benchmarks to detect regressions in selective row decoding, using the existing ParquetMultiColTest benchmark suite.

Changes:

  • Adds a filtered-read helper that filters on long10K, selects requested columns, and reports throughput in file rows per second.
  • Adds 1% sparse and 50% dense reads of the existing Snappy Defaults file, excluding array columns.
  • Makes scale-factor setup explicit per test and reduces the Gzip tests’ scale factor.
File Description
src/​it/​java/​io/​deephaven/​benchmark/​tests/​standard/​file/​ParquetMultiColTest.java Adds sparse/dense benchmark cases and per-test scale-factor setup.
src/​it/​java/​io/​deephaven/​benchmark/​tests/​standard/​file/​FileTestRunner.java Adds the reusable filtered Parquet read benchmark helper.

Review effort: Balanced

@stanbrub
stanbrub requested a review from cpwright November 3, 2026 19:03
@stanbrub
stanbrub marked this pull request as ready for review November 3, 2026 19:04
Copilot AI balanced review requested due to automatic review settings November 3, 2026 19:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the filtered Parquet benchmark changes.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 4, 2026 06:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the Parquet read benchmark changes.

No open issues 🎉

Review effort: Balanced

@stanbrub
stanbrub requested a review from cpwright November 4, 2026 17:41
@stanbrub
stanbrub merged commit ad00e11 into deephaven:main Nov 4, 2026
2 checks passed
@stanbrub
stanbrub deleted the sparse-parquet-benchmark branch November 4, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants