September 08, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The past 7 days saw 38 merged PRs and 29 open PRs across Velox, ClickHouse, Flink and build/infra areas. Velox backend dominated with daily version bumps, shuffle-read optimizations, and new function enablements. Flink activity surged (7 PRs) around Nexmark benchmark support, while Iceberg/Delta lake features and GPU/cuDF connectors are gaining momentum. Community is preparing for Gluten 1.5.0 release with documentation clean-ups and CI improvements.

Key Ongoing Projects

• Shuffle-read performance overhaul – @marin-ma merged [GLUTEN-10214][VL] Merge inputstream for shuffle reader #10499 to coalesce small batches and eliminate VeloxResizeBatches, cutting deserialize overhead in sort-based shuffle.
• Iceberg write & partition support – @jinchengchenghh landed [GLUTEN-9335][VL] Support iceberg partition write #10497 for partition write; [GLUTEN-9366][VL] Support Iceberg functions #10285 adds Iceberg functions (still fallback).
• Delta Lake read/write PoC – @zhztheplayer opened [GLUTEN-10215][VL] PoC: Delta Lake write for Spark 3.5.2 / Delta 3.3.1 #10216 for Delta 3.3.1 write and [VL] Support generating DV-enabled TPC-DS Delta table in gluten-it #10639 for DV-enabled TPC-DS table generation.
• Flink Nexmark completeness – @shuai-xu & @KevinyhZou enabled q11-q21, processing-time windows, and Java-17 compatibility ([GLUTEN-10546][FLINK] Support all flink operators for nexmark #10548, [GLUTEN-10630][FLINK] Support processing time for q12 #10631, [GLUTEN-10570][FLINK] Add --add-opens options to MAVEN_OPTS for Java 17 compatibility #10572).
• cuDF/GPU connector – @jinchengchenghh introduced [GLUTEN-10621][VL] feat: Support cudf parquet connector #10622 for cuDF parquet reader and Support GPU parquet connector #10621 for GPU connector tracking.
• Spark 4.0 CI readiness – @zhouyuan enabled TPC-DS tests on Spark-400 ([GLUTEN-10210][VL] Enable tpcds tests for Spark-400 in CI #10633) and moved Spark-3.2 tests to nightly ([GLUTEN-8960][VL] Move spark32 tests to nightly run #8961).

Priority Items

• Release blockers for 1.5.0 (Gluten 1.5.0 Release Preparation: Patch Porting, Documentation Updates, and Final Tasks #10574) – @PHILO-HE tracking open ports: [GLUTEN-10450][VL] Reclassify internal/public configs and remove internal configs from doc #10603 (config doc tidy-up by @zjuwangg) and [VL] Fix arrow url typo #10641 (arrow url typo by @liujiayi771).
• Critical bug fixes – NumberFormatException in WindowFunctionNode.setBound() #10644 (NumberFormatException in window bounds) by @mingyi6666; [VL] file INSTALL cannot set permissions #10635 (file INSTALL permission) by @beliefer; Delta column mapping mode caused wrong result with partition column filters #10511 (Delta column-mapping wrong results) by @sezruby.
• Memory OOM & stability – [VL] Global memory OOM #7249 global-memory OOM during spill needs Velox-side fix; [VL] Deadlock when memory arbitration in TableScan preload #9846 deadlock in TableScan preload; [VL] [ARM] Core Dump Issue for Benchmark with the TPCDS queries #9845 ARM core-dump under investigation.

Notable Discussions

• [Flink] About SARG Support in Flink and BaseVector-related Issues #10406: @ryyyyyy1 asks about SARG push-down in Flink and early Velox4j initialization—community feedback wanted.
• High deserialize time when doing shuffle read #10214: long thread on high shuffle deserialize time; [GLUTEN-10214][VL] Merge inputstream for shuffle reader #10499 already merged but further tuning invited.
• [CORE] Implement stage-level resourceProfile auto-adjust framework to avoid oom #8018: stage-level ResourceProfile auto-adjust design accepted; POC code to be contributed by @zjuwangg.

Emerging Trends

1. Lake-house acceleration – Iceberg & Delta PRs appearing daily; deletion-vector and column-mapping fixes show production readiness push.
2. Flink-first development – Nexmark benchmark now drives Flink function parity (UDFs, decimal, time attributes).
3. GPU/cuDF integration – Velox cuDF parquet connector merged; GPU memory config and connector stubs checked in.
4. Micro-performance focus – repeated identifyBatchType (Avoid repeated calls to identifiyBatchType #10649), StrictRule simplification ([GLUTEN-10559] Simplify StrictRule and remove unnecessary DummyLeafExec #10553), and hash-table build configs ([GLUTEN-10660][VL] Adding configuration for hash table build #10634) all target driver-side CPU.
5. Documentation & CI hygiene – daily Velox bumps automated, Spark-3.2 demoted to nightly, Maven enforcer rules relaxed ([VL] Update class duplication list in Maven enforcer #10536).

Good First Issues

• [CH] support expression MakeYMInterval #6814: add ClickHouse expression MakeYMInterval – pure CH backend, no native changes.
• [CH] support function date_from_unix_date  #4730: implement date_from_unix_date for ClickHouse – follow existing date function pattern.
• [CH] support split_part function #6807: support split_part string function in ClickHouse – straightforward string splitting logic.
• [CH] support function SparkPartitionID  #6812: expose SparkPartitionID in ClickHouse backend – reuse Spark’s partition ID.
• [CH] support expression MapZipWith #6815: implement MapZipWith for ClickHouse – entry-level map function, good for learning CH UDF framework.

All CH good-first issues need basic C++ and ClickHouse function registration knowledge; unit tests and documentation updates are expected.