Repository navigation
Conversation
Issue Number: close apache#69884 Problem Summary: Delta scans need native column mapping and deletion-vector semantics. Add a FileScannerV2-only DeltaReader that reuses Parquet decoding and TableColumnMapper, accepts optional schema/DV descriptors, handles disk and inline portable deletion vectors, and validates their cardinality on cache hits and misses. Existing Iceberg/Paimon decoding remains shared. This commit provides the BE foundation; FE planning and end-to-end catalog support follow in the same M1 work. Add the backend reader foundation for native Delta Lake scans, including column mapping, deletion vectors, timestamp semantics, and supported type widening. - Test: Unit Test and Manual test - ./build.sh --be -j8 (ASAN, macOS arm64): passed - ./run-be-ut.sh --run --filter='<Delta reader/routing and Iceberg/Paimon reader suites>' -j4: 165 tests passed - clang-format 16, check-format.sh, check-build-hygiene.sh, and run-clang-tidy.sh with the UT compilation database: passed - Thrift optional field IDs checked against master and branch-3.1/4.0/4.1/4.2 - Full FE/BE catalog regression testing remains pending - Behavior changed: Yes (Delta query scans select the native V2 reader even when enable_file_scanner_v2 is false) - Does this need documentation: Yes (complete catalog documentation follows with FE integration)
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Add the FE-01 through FE-09 and FE-12 foundation for native Delta Lake reads. Package a Delta Kernel 4.4.1 connector with filesystem and HMS table discovery, type mapping, reader-feature checks, and statement/MVCC snapshot identity. Own Hadoop clients per catalog, propagate plugin-local storage and HMS identities, and close resources independently of catalog DDL locks. Scan planning and end-to-end query support follow in later commits. Configured S3 endpoints use Hadoop S3A outbound HTTP. Catalog CREATE and ALTER privileges control these endpoints; querying requires SELECT privilege. This connector does not fetch or load remote executable artifacts. ### Release note Add the Delta Lake connector metadata and packaging foundation. Native query planning is not enabled by this incremental commit. ### Check List (For Author) - Test: Unit Test: 34 connector and 2 metastore tests passed; module Maven install, Checkstyle, architecture checks, and plugin zip packaging passed. Full FE startup and end-to-end regression remain pending. - Behavior changed: Yes, register the new deltalake metadata connector. - Does this need documentation: Yes, included with the completed M1 feature.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Spark 4 writes a Delta table's true path to HMS SerDe properties while its StorageDescriptor location can end in -__PLACEHOLDER__. Using that placeholder makes Doris report a missing Delta table during DESC. Prefer the SerDe path when present, retaining the storage location for tables that do not supply a data-source path. This completes the FE-05 location path. ### Release note Recognize the real location of Spark-registered Delta Lake tables in HMS. ### Check List (For Author) - Test: Unit Test: 35 connector tests passed, including the Spark placeholder case. Full build.sh --fe -j4 passed. Manual test: Spark 4.0 + Delta 4.4.1, Hive 3.1.2 and MinIO; both filesystem and HMS catalogs pass SHOW/DESC and SHOW CREATE TABLE after REFRESH. Data scan implementation remains pending. - Behavior changed: Yes, prefer Spark's SerDe path for Delta tables. - Does this need documentation: No, included in the M1 connector documentation.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Delta catalog metadata cannot yet be queried by the native BE reader. Add snapshot-pinned file and partition enumeration, typed partition values with explicit null flags, and Parquet byte ranges carrying the Delta descriptor (FE-10, FE-11). Cover HMS/filesystem reads, empty and nested data, split boundaries, V2 enforcement and session time zones with generated regression results. ### Release note Add basic native reads of Delta Lake tables through HMS and filesystem catalogs. ### Check List (For Author) - Test: Unit Test: 43 connector and 2 metastore tests; full FE build.sh --fe; Regression test: test_deltalake_basic with generated and verified results on local Spark 4.0 / Delta 4.4.1 / MinIO / HMS - Behavior changed: Yes, Delta catalog tables can be queried using the native reader - Does this need documentation: Yes, M1 user documentation is tracked in apache#69884
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Delta scans read every candidate data file and COUNT(*) still opens Parquet files. Convert compatible SPI predicates to Kernel filters while retaining BE residuals, preserve null/NOT semantics and reject lossy conversions. When every active file has row statistics and no filter remains, carry the exact count minus deletion cardinality on one range; otherwise scan normally. Add snapshot-version EXPLAIN output and regression coverage for OPT-01/02 and the non-DV OPT-04 path. ### Release note Enable Delta Lake file skipping and metadata-only COUNT(*) for unfiltered tables with complete statistics. ### Check List (For Author) - Test: Unit Test: 50 connector and 2 metastore tests; full FE build.sh --fe; Regression test: test_deltalake_basic and test_deltalake_skipping; Manual test: COUNT(*) profile reports zero data-file read bytes/calls - Behavior changed: Yes, compatible predicates prune files and exact COUNT(*) avoids data-file reads - Does this need documentation: Yes, M1 documentation is tracked in apache#69884
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Enable FEAT-01 through FEAT-06 by sending logical and physical field-ID schemas to the native Delta reader, allowing verified timestamp_ntz and type-widening features, and carrying deletion vectors through scan planning. Preserve optional collection IDs and reject unverified widening with its field path. Keep widened DATE predicates as BE residuals because Kernel 4.4.1 cannot parse the old DATE min/max statistics as timestamp_ntz. Add Spark evolution/DV fixtures and p0 coverage for both metadata sources, golden tables, three time zones, count pushdown and small byte ranges. ### Release note Native Delta reads support column mapping, timestamp_ntz, verified type widening and deletion vectors. Metadata COUNT(*) subtracts deletion-vector cardinality without reading data files. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 58 Delta connector tests and 2 Delta metastore tests; full build.sh --fe -j4 with checkstyle. - test_deltalake_basic, test_deltalake_skipping and test_deltalake_features via run-regression-test.sh. - 30 HMS/filesystem comparisons against Spark for 15 tables; DV count profile shows zero file read bytes/calls. - Behavior changed: Yes, enable verified Delta reader features. - Does this need documentation: Yes, catalog documentation is tracked by apache#69884.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Repeated Delta queries reload snapshot metadata and large scans materialize every byte range before execution. Cache latest references with a fixed TTL after load completion and user-isolated, size-governed snapshots; invalidate through the shared refresh registry. Plan large or unknown-size scans lazily from the pinned snapshot, preserving schema mapping, deletion vectors, and exact metadata count. Accept and verify classic, multipart and V2 checkpoints, sidecars and log compaction. ### Release note Delta catalogs cache the latest version for 30 seconds by default, with configurable TTL and explicit REFRESH invalidation. Large scans can generate ranges lazily. V2 checkpoints and vacuumProtocolCheck tables are supported. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 70 Delta connector tests and 2 metastore tests; full build.sh --fe with checkstyle. - Four local Spark/HMS/MinIO p0 suites; 10 metadata-golden comparisons against Spark. - Fixed TTL, refresh races, iterator cancellation/close, synchronous/streaming parity and schema refresh. - Million-file planning acceptance remains in the tracking tasklist. - Behavior changed: Yes (fixed-TTL latest-version cache and lazy split planning). - Does this need documentation: Yes (Delta catalog documentation is tracked in apache#69884).
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Hive catalogs treat Spark-registered Delta tables as ordinary Hive storage or reject their placeholder input format. Detect the Delta provider and route the original table handle to a lazy deltalake sibling plugin, sharing the gateway's HMS and storage settings. Reuse existing foreign-handle schema, MVCC and scan delegation, and forward invalidation and lifecycle cleanup to the sibling. ### Release note Hive catalogs can read registered Delta tables through the native Delta reader, including column mapping, deletion vectors and supported checkpoint formats. DESC and SHOW CREATE use the Delta schema; REFRESH updates the sibling metadata cache. ### Check List (For Author) - Test: Unit Test / Regression test - 274 Hive-related unit tests; full build.sh --fe with checkstyle. - Six local Spark/HMS/MinIO p0 suites, including mixed Hive/Delta reads, schema refresh and negative feature/existence cases. - Behavior changed: Yes (Hive catalogs route Delta tables to the deltalake plugin). - Does this need documentation: Yes (Delta catalog documentation is tracked in apache#69884).
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Small fixtures cannot verify the memory behavior or first-range latency of lazy Delta split planning. Add a reproducible planning-only checkpoint with one million AddFile actions, an opt-in connector test that enumerates all ranges three times in a 512 MiB JVM, and a p2 metadata COUNT check. The fixture deliberately omits data objects and does not represent physical read throughput. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - DeltaLakePlanningScaleTest passed with -Xmx512m: three passes of one million ranges, first range 22-60 ms, enumeration 1.46-1.86 s, highest sampled heap about 200 MiB. - Official generator executed; test_deltalake_planning output generated by the regression script and verified. - Local MinIO metadata COUNT returned 1,000,000 with zero data FileReadBytes/FileReadCalls. - Behavior changed: No (test-only opt-in fixture). - Does this need documentation: No (reproduction commands and measurement scope included in the p2 README).
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Each Delta split entered the authentication context again when the caller consumed an already-buffered range, and complete-file splits copied an immutable range unnecessarily. Keep interruption checks before the buffered fast path and reuse only normal ranges whose byte interval is unchanged. Metadata-count ranges still clear their count when converted to byte ranges. Add an opt-in phase probe and document why M1 retains DefaultEngine without a content cache. On the same million-action fixture and 512 MiB JVM, three enumeration samples changed from 1.86/1.65/1.46 s to 1.53/1.47/1.42 s; these are local samples, not a statistical performance guarantee. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 72 Delta connector tests plus 2 metastore tests, including scale/phase probes with -Xmx512m. - Full build.sh --fe; six p0 suites and one p2 planning suite passed. - Independent before/after range-enumeration measurements documented in the p2 README. - Behavior changed: No (same ranges, metadata-count semantics, interruption and cleanup). - Does this need documentation: No (measurement methodology and cache decision included).
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Cover OPT-05 with a Spark fixture whose single Delta data file contains multiple row groups and pages. Verify native Parquet min/max and page index pruning for ordinary, name-mapped and id-mapped renamed columns through both HMS and filesystem catalogs, while checking the selected rows. ### Release note None ### Check List (For Author) - Test: Regression test - test_deltalake_parquet generated and verified; all six catalog/mapping combinations pass - Behavior changed: No - Does this need documentation: No
Issue Number: close apache#69884 Problem Summary: Complete OPT-03 with snapshot, total/selected file, partition, and split counts. Reuse fixed-version statement observations and public Kernel reports without enumerating files for diagnostics. Streaming EXPLAIN reports pending counts; the execution profile distinguishes complete and partial scans. COUNT diagnostics describe all files rather than its representative range. Collect streaming connector profiles after closing their sources and before finishing scheduling, under the plugin classloader. Preserve primary failures, report close/profile and lazy class-loading failures, and handle rejected scheduling without leaving split consumers waiting. Delta scans expose file and partition selection in EXPLAIN and query profiles. Streaming connector source-close failures now fail scheduling instead of being logged after scheduling has already reported success. - Test: Unit tests - 75 Delta connector, 2 metastore, 25 FE core tests pass; includes million-file planning in a 512 MiB JVM - Test: Full build.sh --fe and 8 Delta p0 plus 1 p2 regression suites pass; regression outputs generated and verified using run-regression-test.sh - Behavior changed: Yes - add scan diagnostics and propagate streaming cleanup failures before scheduling completes - Does this need documentation: Yes - Delta catalog documentation remains tracked in apache#69884
Issue Number: close apache#69884 Problem Summary: TEST-01 through TEST-04 need a reproducible Spark 4.0, Delta 4.4.1, Hive 3 HMS and MinIO environment. Add an opt-in profile with isolated Spark configuration, checksum-pinned dependencies and 31 upstream golden tables. Run repeatable fixtures before marking Spark ready. Preserve the default Iceberg and Paimon service configurations. None - Test: Regression test / Manual test: all eight Delta p0 suites passed twice, including after Spark recreation; default Compose services/networks match; shell/Python/XML validation and dependency checksum/download checks passed. Existing Iceberg/Paimon suites were not rerun: baseline archive download failed TLS locally. - Behavior changed: No production behavior change; Delta test services opt in. - Does this need documentation: Yes, Docker setup is documented in the README.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: FE-03 and END-01 need end-to-end storage validation beyond MinIO. Add a configurable p2 suite that uses the same golden tables and expected results for each storage backend. Cover mapping, disk DVs, nested widening, timestamps and V2 checkpoint sidecars in both planning modes. ### Release note None ### Check List (For Author) - Test: Regression test: generated results on MinIO, then verified independently on MinIO and HDFS 3.2.1; 16 query tags per backend. Other cloud providers and Kerberos HDFS remain unverified pending environments. git diff --check passed. - Behavior changed: No production behavior change; p2 suite defaults off. - Does this need documentation: Yes, reproduction steps are included in p2 README.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Restarting the Delta test Spark service regenerated its tables, which made it unsuitable for retaining debugging and benchmark datasets. Persist fixture readiness and reuse the data by default. An explicit regeneration flag is required to replace initialized fixtures, and Thrift readiness remains checked on every startup. ### Release note None ### Check List (For Author) - Test: Manual test / Regression test: recreated Spark, verified health and reuse log, compared all 1,227 MinIO object SHA-256 values, and passed basic Delta p0. Shell syntax and git diff --check passed. - Behavior changed: Yes, test Spark restarts reuse fixtures by default. - Does this need documentation: Yes, retention and regeneration are documented in README.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Delta M1 reader acceptance needs repeatable result and timing comparisons with Hive over the same Parquet objects. Add SF1 data conversion, shared-file Hive/Delta registration and an opt-in regression using the existing 22 TPC-H and 103 TPC-DS queries. Preserve generated data, reject nonempty output, and record per-reader SQL failures while continuing coverage and failing the suite at the end. Document configuration, checksums and measurement limits. ### Release note None ### Check List (For Author) - Test: Unit/manual checks: Python syntax, complete 32-table converter replay, schema/row checks, nonempty output rejection, Spark row counts and Hive providers. Regression test: official p2 execution has completed 19 paired TPC-H checks; Hive Q7 reaches the memory limit and is recorded as failure. Full benchmark acceptance remains pending, not claimed passed. - Behavior changed: No production behavior changes; adds opt-in test tooling. - Does this need documentation: No separate documentation PR for test tooling; setup and retention instructions are included in the p2 README.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: Restarting the Delta Compose environment reruns the MinIO initializer, which previously overwrote golden objects even when Spark retained its fixtures. Persist a successful golden-upload marker in the shared fixture state and skip repeat uploads by default. Propagate the explicit regeneration flag to the initializer and remove the marker before a regeneration attempt. ### Release note None ### Check List (For Author) - Test: Shell syntax and Compose config checks; six script lifecycle cases including failed regeneration/retry; actual initializer recreation skipped upload; all 1,227 MinIO object paths, sizes and SHA-256 matched the retained export. - Behavior changed: Yes, optional Delta test setup retains golden objects on restart. - Does this need documentation: No separate documentation PR; README updated.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: The shared-file benchmark exposed three harness compatibility issues: Hive's default Q7 join order exceeded memory, Q39 floating aggregates differed by one ULP, and the older Q30 SQL referenced a column absent from the TPC-DS 3.2 generator schema. Apply the existing Hive regression's fixed Q7 order to both readers, use the framework's typed comparator, and adapt Q30 to the generator's review-date column. Record the Q7 control and document every change; retain the original failure evidence separately from controlled timings. ### Release note None ### Check List (For Author) - Test: Regression test: official focused suite passed Q7, Q39 and Q39_1, including three paired repetitions. The resumed official suite passed Q30 with 92 matching rows and continues the remaining query inventory. Original Q39 difference checked: one of 1,760 cells differed by 2.22e-16. - Behavior changed: No production changes; benchmark comparison/control fixes. - Does this need documentation: No separate documentation PR; p2 README updated.
### What problem does this PR solve? Issue Number: close apache#69884 Problem Summary: The default Hive Q85 plan joins two customer_demographics aliases on marital and education status before web_returns, exceeding the benchmark's 8 GiB limit. Apply the same fixed join order to both readers and label that control explicitly, while preserving default-plan failure evidence. All other query controls remain unchanged. ### Release note None ### Check List (For Author) - Test: Regression test: focused official Q85 suite passed, with five matching rows and three paired repetitions; EXPLAIN confirms the common join order. Together with the retained completed/focused runs, all 125 queries now have complete result and timing comparisons. This is not a claim that the original unmodified full-suite invocation passed. - Behavior changed: No production changes; benchmark baseline control only. - Does this need documentation: No separate documentation PR; p2 README updated.
Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Ordinary external-table queries only consulted the table-keyed row-count cache, which could keep a cold estimate after versioned connector statistics completed. Prefer statistics at the statement's pinned snapshot for ordinary and versioned queries, reuse the existing size/row-width fallback, and forward the pinned snapshot through the Hive sibling gateway. This covers STAT-02. External-table optimizer estimates can use completed statistics for the query's pinned snapshot without waiting for the table-only row-count cache to refresh. - Test: Full FE build and Checkstyle; 114 FE core and 267 Hive tests; focused compatibility run including 22 Paimon and 7 Iceberg tests; seven Delta p0 suites on the combined change, including new filesystem/HMS/Hive cardinality coverage. - Behavior changed: Yes, ordinary pinned queries prefer snapshot statistics. - Does this need documentation: Yes, Delta connector README and prepared EN/ZH website patch are updated in the accompanying statistics change; website PR pending.
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The native Delta connector did not supply basic table statistics to the optimizer. Cache logical row counts by user, table identity and version, summing active AddFile numRecords minus deletion-vector cardinality. Collect in a bounded background pool and return same-version CRC size while cold; missing row statistics stay unknown. Shared cache invalidation cancels pending work, and exact COUNT remains independent of optimizer estimates. Add shared-snapshot concurrency, real checkpoint/DV, lifecycle, authentication and million-file tests (STAT-01/03), plus filesystem/HMS/Hive regression coverage and connector documentation. ### Release note Native Delta Lake catalogs and Hive gateway queries gain basic versioned row/size statistics with asynchronous collection and a CRC size fallback. ### Check List (For Author) - Test: Full FE build and Checkstyle; 87 Delta and 2 metastore tests, including the opt-in million-file fixture in a 512 MiB JVM; focused compatibility/authentication rerun; seven Delta p0 suites passed on the final build. New .out generated by the official regression script and then verified without generation. Preserved all 1,227 MinIO objects, checked by path, size and SHA-256. - Behavior changed: Yes, Delta exposes basic optimizer statistics. Exact COUNT semantics and Delta data are unchanged; column NDV/histograms are not supplied. - Does this need documentation: Yes, connector README included; local design, tasklist and EN/ZH website patch updated. Website PR/full site build pending.
…stics ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Delta table row counts load asynchronously, so a post-statistics benchmark needs to distinguish ready statistics from a cold metadata lookup. Add an optional retained-manifest check that waits for Delta EXPLAIN cardinalities before timing and records Hive estimates. Keep the existing query texts, common controls, result comparator and three alternating measured pairs unchanged. Document the complete local TPC-DS rerun: the summed median gap narrows from 26.1% to 16.5%, with remaining regressions retained rather than replaced by focused samples. No production code or fixture data changes. ### Release note None ### Check List (For Author) - Test: Official regression script passed all 103 TPC-DS queries (824 executions) with matching results; all 24 Delta cardinalities match the independent manifest. A separate Q4 rerun passed, and eight complete diagnostic profiles were retained. All 1,227 MinIO objects match their original path/size/SHA-256 before and after. Source whitespace checks and code-review checkpoints passed. - Behavior changed: No production change. Optional benchmark warmup is disabled unless deltalakeBenchmarkStatisticsManifest is provided. - Does this need documentation: Yes, benchmark README includes warmup usage, before/after measurements, controls and remaining performance limitations.
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: File scans always use FileScannerV2. The basic and features Delta suites still exercised the same reader twice for both filesystem and HMS catalogs. Run them with enable_file_scanner_v2=true and preserve the retained query tags. The dedicated Hive gateway and native reader dispatch checks still exercise the deprecated false setting. ### Release note None ### Check List (For Author) - Test: Regenerated expected outputs with run-regression-test.sh and reran test_deltalake_basic and test_deltalake_features in comparison mode. Both suites passed both runs. All 146 retained query output blocks were byte-identical. Removed 32 redundant blocks (526 lines). - Behavior changed: No - Does this need documentation: No
Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Commit 598756e made ordinary plugin table reads resolve their pinned snapshot statistics on the planning thread. This changed the row-count path for Iceberg, Paimon and Hive as part of a Delta feature. Revert that commit's FE table and Hive statistics changes. Delta still collects logical rows asynchronously per version and supplies same-version CRC bytes when its count is unavailable. Exercise cold estimates in p0 and wait for the standard FE cache refresh before timing the p2 benchmark. Ordinary catalog queries use the standard FE external-table row-count cache. Delta estimates in EXPLAIN may retain their initial value until an access triggers a refresh (10 minutes by default). Exact COUNT is independent. - Test: Full FE build and Checkstyle. All 102 targeted FE tests passed. test_deltalake_statistics passed for filesystem, HMS and Hive. With the default refresh interval, all 24 TPC-DS manifest counts matched and Q1 passed for both readers in test_deltalake_benchmark. - Behavior changed: Yes. Restore master's row-count policy for ordinary plugin table reads and its original Hive statistics delegation. - Does this need documentation: Yes (connector README and benchmark guide updated in this PR; website documentation is tracked in apache#69884).
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Merging master 7acfd9b left two Delta gaps. First, apache#69717 replaced table_reader.h's full ShardedKVCache definition with a forward declaration, but deltalake_reader_test.cpp constructs a ShardedKVCache and relied on the transitive include. Include format/format_common.h directly, as hudi_reader_test.cpp does. Second, apache#69300 made Parquet readers in one scan share their profile counters through TableReader's parquet profile cache. Hudi, Iceberg and the base reader pass that cache, but DeltaReader still built its ParquetReader without it, so every Delta file created its own set of counters. Pass the cache, and keep binary-UUID preservation disabled as before. ### Release note None ### Check List (For Author) - Test: No local build. The merge was conflict-free. Both changes follow the matching master code. COMPILE and BE UT are left to CI. - Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do. - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Extracting decode_portable_deletion_vector_payload for inline Delta DVs left the framed Iceberg/Delta decoder checking the portable magic twice. Its first check runs before the CRC, and the payload function checked it again. The magic constant was also defined twice. Move the constant, the magic check and the Roaring decode into file-local helpers. The framed decoder again validates magic, then CRC, then decodes the bitmap once, with the same offsets and length it used before this PR. The inline payload decoder keeps its own size check and uses the same helpers. ### Release note None ### Check List (For Author) - Test: No local build. Existing Iceberg DV unit tests cover the magic and CRC rejection order. The Delta reader unit tests compare inline and on-disk decoding. Both are left to CI. - Behavior changed: No. Iceberg DV decoding is again identical to master's. - Does this need documentation: No
…n validation Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: validate_file_scan_support is the shared range gate for table markers that are no longer supported. The Delta integration also added a rule there that rejects non-Parquet Delta ranges. That state cannot occur. The Delta protocol stores data only as Parquet, and DeltaLakeScanRange always sends FORMAT_PARQUET. DeltaReader also asserts Parquet in create_file_reader. That runs for every data file, after prepare_split resets the reader format from the current range. Remove the shared rule and its unit test, so validate_file_scan_support is identical to master again. Keep the Delta entry in can_ignore_not_found_file. A missing file in a committed Delta snapshot must still fail the scan, as for Iceberg. None - Test: No local build. The remaining Delta dispatch and missing-file unit tests are left to CI. - Behavior changed: No for FE-generated plans. A malformed non-Parquet Delta range now fails the DeltaReader assertion instead of returning NotSupported. - Does this need documentation: No
… the Delta PR ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The Delta integration widened two fe-core catch blocks to Error. One is in executeWithProfileFinalization, on the batch planning path. The other is in the streaming profile finalizer. A lazily loaded plugin dependency then reported NoClassDefFoundError like any other failure. These are general hardening changes for every plugin connector, and Delta does not depend on them. Master's SplitProducer already catches Throwable while it iterates, closes the source and runs completion. The scan still fails without hanging and still calls finishSchedule. Of the four streaming tests added with the change, three cover existing master behavior. Only the profile-phase Error case needs the widened catch. Restore PluginDrivenScanNode and its scan profile test to master so the Delta PR leaves fe-core unchanged. The hardening can be proposed on its own. ### Release note None ### Check List (For Author) - Test: No local build. Both files are identical to master; left to CI. - Behavior changed: No relative to master. A Delta-only branch would have reported a profile-phase Error as a UserException. - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: apache#69723 removed the legacy file scanner. Every file scan now uses FileScannerV2, and enable_file_scanner_v2 is kept only as a deprecated session variable that nothing reads. Four Delta suites still set it: basic, features and the benchmark to true, and the Hive gateway suite to false. None of these settings has any effect, so the Hive suite's false setting does not check anything. Remove all four. SET statements produce no query output, so the expected .out files are unchanged. The Hive suite still sets file_split_size=128 so that one file becomes many ranges sharing the same deletion vector. ### Release note None ### Check List (For Author) - Test: No local run. The diff deletes only SET statements; regression runs are left to CI. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The optional deltalake Compose profile published its Hive Metastore on host port 9383 by default. Hive 3's stack publishes its metastore on the same port (hive-3x_settings.env, hive3HmsPort). With both stacks running, delta-hms cannot bind the port, so `up --wait` fails for the whole Iceberg stack. The external pipeline already runs Hive 3 and Iceberg together, so enabling Delta there would hit this every time. Change the default to 19383, which no other stack uses and which follows Hudi's 19083. Update the Compose template, the README port table and deltalakeHmsPort in regression-conf.groovy. The suites read the port from that setting, and the container still listens on 9083. ### Release note None ### Check List (For Author) - Test: No local run. Searched the repository for other uses of both ports; the stacks were not started. Left to the regression environment. - Behavior changed: No production change. The Delta test profile's default HMS host port moves from 9383 to 19383. A custom config that sets deltalakeHmsPort=9383 must also set DELTALAKE_HMS_PORT or be updated. - Does this need documentation: No (the README table is updated here)
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: HMS gives a Delta table only its format, name and root path, but the regression suites mostly registered tables with an explicit LOCATION whose directory matched the table name, so a path derived from the name or a stale name after a change could go unnoticed. Add test_deltalake_hms_registration, which runs every scenario through both the dedicated Delta catalog and the HMS gateway catalog: partitioned managed tables with a deletion vector, CTAS, external paths unrelated to the name, tables registered without columns, two names for one path, paths with a space or a trailing slash, mixed-case names under all three name-case modes, filtering of Parquet tables and views, rejection of Doris writes, and renames, drops, recreation and database drops made by Spark after both catalogs cached the tables. ### Release note None ### Check List (For Author) - Test: No test run. Only a Groovy syntax check was done; the suite has not run, and its .out file is not generated yet, so the suite fails with a missing output file wherever enableDeltaLakeTest is on until it is generated. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The Delta deletion vector loader calls ShardedKVCache::get. table_reader.h now forward-declares the cache. Compiling table_reader.cpp fails with member access into an incomplete type when no earlier include supplies its definition. Include format_common.h at the call site. ### Release note None ### Check List (For Author) - Test: Reproduced with the ASAN build script and rebuilt the failing Format compilation unit. Header hygiene, clang-format 16 and changed-line clang-tidy passed. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The new HMS registration suite needs expected results to check the dedicated Delta catalog and the HMS gateway on subsequent runs. Record the regression runner's output for managed and external tables, case handling, read-only operations and Spark metadata changes. ### Release note None ### Check List (For Author) - Test: Run test_deltalake_basic, generate test_deltalake_hms_registration with -genOut and rerun it without -genOut. All three runs passed. Independently checked 85 expected values and both catalogs' results. The repeat matched all 127 output blocks and the 27 expected SQL errors. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The HMS registration suite accepted any INSERT error containing "does not". An unrelated failure could satisfy its read-only check. Both the dedicated Delta catalog and the HMS gateway report the same write-plan refusal for reg_external. Match the complete refusal reason and table name. ### Release note None ### Check List (For Author) - Test: Ran test_deltalake_basic, regenerated test_deltalake_hms_registration with -genOut and reran it without -genOut. Both catalog paths returned the expected INSERT error in both runs. All 127 result blocks matched the existing golden file byte for byte. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: A deletion vector addresses rows by their position in the whole file, while row group and page pruning skip rows before the delete predicate runs. test_deltalake_parquet disables deletion vectors, and the tables that have them never assert pruning, so a shifted row position after skipped pages would not be caught for Delta. Add a one-file multi-row-group table whose ids equal their positions, with rows deleted inside the queried ranges and at both ends. The new suite asserts that pruning and the deletion vector were active in the same scan through both filesystem and HMS catalogs, compares against predicates that statistics cannot evaluate, and checks that COUNT and MIN/MAX do not use deleted rows. Also list this suite and test_deltalake_hms_registration in the Delta regression README. ### Release note None ### Check List (For Author) - Test: No test run. Only Groovy and Python syntax checks were done. The fixture must be regenerated with DELTALAKE_REGENERATE_FIXTURES=1 and the .out file is not generated yet, so the suite fails wherever enableDeltaLakeTest is on until both are done. - Behavior changed: No - Does this need documentation: No
### Background Delta deletion vectors identify rows by their positions in a Parquet file. The Delta regression queries early, page-boundary and late ranges while the reader skips pages and row groups. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (adds the Delta reader and this regression suite) PR Type: Test Problem Summary: The pruning and deletion-vector suite lacked its golden file. Record the regression runner's expected live rows, range counts and whole-table extrema for filesystem and HMS catalogs. ### Details - **Scenarios:** Compare indexed predicates and their abs(id) controls across three ranges, then check COUNT, MIN and MAX with deleted file extrema. - **What existing tests miss:** The pruning-only baseline cannot detect applying a deletion vector at a row position shifted by skipped pages or row groups. - **Fails when broken:** An extra deleted ID, a missing live ID, a wrong payload length, or a deleted footer extreme differs from these expected results. The suite also checks that pruning and the deletion vector were active in each tagged scan. ### Testing Regenerated the fixtures with Spark on MinIO and HMS, then ran the basic and Parquet baselines on the local ASAN FE/BE. Generated the new suite's output with -genOut and reran without it. All four runs passed. The 22 blocks matched an independent row-by-row oracle and both catalogs agreed. Spark confirmed 9,990 live rows and extrema 1 and 9,998. All four required counters were positive in each of the six tagged scan profiles. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: The Doris regression framework generated the golden file. Codex executed the suites, checked their output and wrote this commit message. - Human review: Pending review of the local patch against the supplied expectations.
### Background A Delta deletion vector uses positions in the entire Parquet file. Page pruning can skip the first candidate page, and a scan split can start after earlier row groups. The id and its projected payload must stay aligned across both boundaries. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta Lake reader) PR Type: Test Problem Summary: The existing range queries retained a contiguous set of pages in one split. An incorrect payload could also have the expected length because every row in the fixture had the same value. ### Details - **Scenarios:** Store a distinct 64-character payload derived from each id. Check rows and profiles for disjoint selected pages, predicates matching only deleted rows, and file-global deletion positions during a 32 KiB split scan. Check COUNT, MIN and MAX on the empty selection. - **What existing tests miss:** Scanning one retained range does not require resuming after a fully rejected page. The previous projection could not distinguish payloads from different rows. - **Fails when broken:** Reviving a deleted row, omitting a live row or pairing an id with its neighbour's payload changes the golden result. The profile and plan checks require the intended pruning and splits. ### Testing Regenerated the Spark Delta fixtures on local MinIO and HMS. Ran the existing basic and Parquet baselines, generated this suite's golden file with the Doris regression runner, and reran it without -genOut. Both catalogs passed all 16 expected blocks each. An independent comparison verified all 9,990 surviving (id, payload) pairs per catalog. Both scan plans used five splits and the cross-split profiles read two row groups. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Codex updated the fixture and regression suite and ran the independent checks. The Doris regression runner generated the golden. - Human review: Pending review of the changes in PR apache#69904.
### Background Delta Kernel uses file-level statistics to decide which Parquet files to schedule. Doris evaluates the original SQL predicates on the rows of the files that reach BE. A Hive catalog can dispatch Delta tables to the same Delta scan provider as a dedicated catalog. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (introduces the Delta reader) PR Type: Bugfix Problem Summary: A two-file Delta table containing 123.45 returned no rows for CAST(decimal_column AS BIGINT) = 123. Kernel skipped both files before Doris could evaluate the cast. Let the selected table's scan provider veto cast predicates while preserving independent conjuncts for file pruning. ### Details - **Frequency:** Reproducible with filesystem, dedicated HMS and Hive gateway catalogs in synchronous and streaming scans. - **Root cause:** FE unwraps CastExpr when converting a residual to a connector expression. Kernel compared the original decimal with 123. The catalog's default accepted the conversion even when its gateway routed a table to Delta. - **Why existing tests missed it:** Widening tests projected casts. Skipping tests compared columns directly. Neither used a decimal cast in WHERE that changes which files can contain matching rows. - **Fix:** Consult the selected provider's existing ConnectorPushdownOps capability while the original expression still contains its cast. Delta declines these predicates for pruning. BE retains the SQL predicate and an independent decimal conjunct can still prune. - **Other paths and branches:** The catalog's existing veto still takes precedence. Native Hive, Iceberg and Paimon readers use their existing policies. The connector API signatures and version stay the same. - **Behavior change:** Queries that previously skipped matching rows return those rows. ### Testing Built FE locally with build.sh. All 106 targeted FE tests passed, including Hive gateway routing, Iceberg, Paimon and JDBC controls. Reproduced the bug on the old FE and checked 36 queries against the same Parquet files on the fixed FE. Generated and reran the new regression on all three catalogs in both planning modes. All 24 golden blocks matched the fixture. All six mixed-predicate profiles selected one of two files. The existing basic, skipping and Parquet deletion vector regressions passed. ### Release note Delta readers return matching rows when CAST or TRY_CAST changes the value used in a filter, including when accessed through a Hive catalog. ### Check List (For Author) - Test: FE unit tests and Delta regression tests, detailed above - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Codex investigated the failure, prepared this change and its tests, and checked the regression-generated output. - Human review: Pending.
…catalog ### Background An HMS catalog can delegate different tables in one statement to the Delta and Iceberg connectors. Its scan nodes share catalog metadata, while Delta's scan provider rejects CAST-bearing file-pruning predicates and Iceberg retains its existing policy. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: None PR Type: Test Problem Summary: The existing Delta CAST regression tested each catalog entrance separately and did not mix Delta and Iceberg scans in one statement. Add mixed HMS queries and shared-metadata unit coverage so a catalog-wide CAST opt-out or a policy leaking between table scans cannot pass unnoticed. ### Details - **Scenarios:** UNION ALL and JOIN across Delta and Iceberg tables in one HMS catalog, both table orders, synchronous and streaming scans, and an independent ordinary Delta predicate. The FE test alternates two scan nodes sharing the same metadata instance in both orders. - **What existing tests miss:** A blanket HMS CAST opt-out returns correct rows for these fixtures but changes Iceberg's file-pruning behavior. Separate-catalog queries do not exercise the shared metadata boundary. - **Fails when broken:** Pushing the Delta CAST loses the matching 123.45 row. Disabling Iceberg's existing CAST pushdown removes its EXPLAIN predicate and makes the synchronous scan select both files instead of one. The ordinary Delta predicate must still prune one of its two files. ### Testing - Local run-regression-test.sh: test_deltalake_hms_mixed_cast with -genOut and an ordinary rerun both passed. Independently checked all 14 generated result blocks; the golden SHA-256 stayed unchanged. - Local run-regression-test.sh: test_deltalake_cast_pruning passed all 24 result blocks and test_deltalake_basic passed all 36 result blocks. - Local run-fe-ut.sh: 108 tests across PluginDrivenScanNodeCastPredicateTest, Hive routing, Delta, Iceberg, Paimon, JDBC and SPI passed, with no failures, errors or skipped tests. - Local FE Checkstyle reported zero violations; git diff --check passed. ### Notes for Reviewers Iceberg fixtures use integral decimal values to test its existing policy without relying on the separately known lossy CAST bug. Iceberg publishes SDK file counters only for synchronous planning; both modes check its EXPLAIN predicate and actual query results. Production code is unchanged in this commit. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Tests, generated regression output, and this description - Human review: Pending
### Background Delta's scan provider rejects CAST-bearing predicates before the shared converter strips their casts. The mixed HMS regression reads Delta and Iceberg tables in one statement, and originally asserted Iceberg's existing CAST pushdown behavior as well as query results. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: None PR Type: Test Problem Summary: The Delta connector unit tests did not directly guard its scan provider's CAST rejection. Meanwhile, the mixed regression would reject a future Iceberg correctness fix that disables unsafe CAST pushdown. Add a real-provider unit assertion and keep the mixed regression focused on query results and Delta pruning. ### Details - **Scenarios:** Obtain the real scan provider from the Delta connector and assert that it implements ConnectorPushdownOps and returns false for CAST pushdown. Keep the mixed HMS JOIN, UNION ALL, table-order and synchronous/streaming result checks. - **What existing tests miss:** Shared FE tests use provider stubs, so removing the real Delta override could pass them. Iceberg's current CAST pushdown policy is not a permanent correctness contract for a Delta regression. - **Fails when broken:** Removing Delta's interface opt-in or its false-returning override fails the connector unit test. Losing matching rows still fails the mixed golden results. Iceberg can change its own CAST policy without failing assertions about its plan text or selected file count. ### Testing - run-fe-ut.sh: DeltaLakeScanPlanProviderTest and PluginDrivenScanNodeCastPredicateTest passed all 32 tests, with no failures, errors or skips. - run-regression-test.sh: test_deltalake_hms_mixed_cast passed an ordinary rerun with all 14 result blocks. Its generated golden file is unchanged. - Delta connector Checkstyle reported zero violations; git diff --check passed. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Unit test, regression revisions, and this description - Human review: Pending
Resolve three conflicts with current master: - be/src/format_v2/table_reader.cpp: keep master's comment on the existing format_common.h include. - build.sh: keep master's MongoDB packaging step and the connector list that includes deltalake. - docker/thirdparties/docker-compose/iceberg/README.md: keep master's encrypted fixture text, followed by the Delta Lake regression section. Test: No local build. Build hygiene, git diff --check and bash -n build.sh passed. Compilation and tests are left to CI.
### Background Delta chooses synchronous or streaming split planning from the current snapshot's file count. Writers without a current checksum leave that count unknown, so even a two-file table streams and can start far more scanners than the selected file needs. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 reader) PR Type: Performance Problem Summary: Reuse complete file counts already available in the statement or in Delta's user/table/version-scoped statistics cache when no current checksum is available. Existing statistics collection counts files, and exhaustive normal scans publish their already-observed total without submitting a statistics task. This lets subsequent small-table queries use synchronous planning while keeping large and unknown tables streaming. Changes are confined to the Delta plugin, its tests, and its README. ### Details - **Why it was slow:** The streaming scanner path does not cap scanner creation by the actual split count. On the retained two-file fixture, the automatic path started 144 scanners to read one selected file and one row; forced synchronous planning started one. - **Optimization:** Check the current CRC, then the statement's complete count, then the same user/table identity/version cache entry. The gate never enumerates files, starts statistics work, or waits for storage. A fully consumed scan without partition-pruned replay publishes its total into the entry captured before scanning, so late reports cannot repopulate invalidated entries. Background row-count sessions can have a different identity from SQL sessions, making normal-scan publication necessary without relaxing user isolation. - **Results:** On local macOS ARM64 with the same ASAN BE and retained data, 16 alternating warm pairs per entrance reduced automatic median latency from 61.614 to 41.420 ms (filesystem), 66.370 to 38.509 ms (HMS), and 57.911 to 32.874 ms (HMS gateway). Scanner counts became one; all returned 123.45. Forced-sync controls were 35.978/41.056, 44.302/37.627, and 32.914/32.370 ms before/after. Million-file metadata enumeration medians were 1493.563/1451.023 ms with a 512 MiB heap; both scale tests passed. - **Applicability and cost:** Warm snapshots with an already-known complete small file count benefit. Cold unknown counts still stream. Reuse existing cache capacity, TTL, invalidation and executor ownership; add one count, not a file list. These are small local ASAN measurements, not a TPC-DS claim. Sampled scan heap peaks increased from 133327520 to 238172480 bytes while remaining within 512 MiB; actual million-file data IO was not measured. - **Behavior change:** Small Delta tables may select synchronous split planning after a complete count is available. Row results, public row-count cache behavior, BE scheduling, other connectors, and wire/storage formats are unchanged. ### Testing - Local `./build.sh --fe -j4`: passed, including Checkstyle. Existing ASAN BE binary and local source configurations were unchanged by SHA-256. - Official FE UT runner for `DeltaLake*Test,DeltaKernel*Test`: 92 actual tests passed, zero failures/errors; two optional classes skipped in that invocation. Separately enabled `DeltaLakePlanningScaleTest` with the retained million-file checkpoint and `-Xmx512m`: two tests passed. The diagnostic PlanningMetrics class remained disabled. - Tests cover cached/unknown/current-CRC counts, threshold equality, missing row statistics, user/version/UUID/path isolation, no work submission or IO wait, late invalidated reports, partial and partition-pruned scans, large-file splitting, and large-table streaming after count reuse. - Official regression runner: basic, statistics, cast_pruning, parquet_deletion_vector, hms_mixed_cast, and planning passed: six suites, 129 result blocks. Existing golden files were unchanged. The planning suite initially targeted the wrong local MinIO; using the retained fixture's endpoint resolved the missing-database error without changing the test or data. - `git diff --check`: passed. ### Release note Delta Lake queries can avoid excessive scanner startup for small tables without a checksum once a complete file count is available. Large tables and snapshots with unknown file counts continue to use streaming planning. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the Delta connector README documents selection, cache reuse and cold-query limitations. No new user configuration is introduced. - Pre-submission status: Partially verified: local tests and targeted performance checks passed; full TPC-DS and large-table data IO performance were not rerun. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, README, local evidence report and this description. - Human review: Pending.
### Background Delta fixes a snapshot for each statement, then enumerates its active files to plan native Parquet ranges. Repeated scans currently reopen the same checkpoint and reconstruct the same ranges. Other connectors already use the existing statement planning scope and external scan task reuse session flag. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 implementation) PR Type: Performance Problem Summary: Repeated identical synchronous Delta scans redo checkpoint IO and planning. Reuse completed immutable ranges and metrics within the existing planning scope, with typed request keys, bounded retention and cancellable single-flight loading. The change applies to dedicated Delta and HMS gateway reads. Production changes are confined to the Delta plugin; most added lines are tests and generated output. ### Details - **Change map:** Four production Java files (269 added / 50 removed lines), five test files (549 added / 4 removed), 244 generated expected-output lines and 33 README lines. The new memo owns keys, admission and concurrency; the existing scan body is extracted without changing its pruning semantics. Provider/scale tests and a three-route SQL suite cover reuse and bypass. - **Why it was slow:** Each scan independently enumerated the pinned snapshot, including repeated checkpoint range GETs, decode and range construction. - **Optimization:** Use the existing enable_external_scan_task_reuse flag and planning-only statement scope. Key by catalog/query/user, table coordinates/UUID/version, typed filter, partition selection, COUNT, split size and time zone. Coalesce identical loads outside scope/memo locks; publish only completed results and per-node metrics. Retry failed/cancelled loads; let waiters cancel independently. Retain at most 128 entries, 16,384 ranges and 16 MiB estimated range payload; oversized synchronous results are returned without retention. Streaming never enters the memo. - **Results:** Same-build alternating local ASAN/SF1 A/B: Q88 EXPLAIN 323.616 to 152.279 ms and execution 2268.353 to 2050.009 ms (-9.6%); Q80 EXPLAIN -23.9%, execution -1.9%; Q4 EXPLAIN -49.4%, execution essentially flat; Q22 has no hits and execution is flat (+0.5%). Three independent traces each show checkpoint GETs 160 to 55 / 90 to 60 / 90 to 30 for Q88/Q80/Q4; Q22 stays at 15. A million-file streaming fixture remains bounded in a 512 MiB JVM: enumeration median 1493.940 to 1484.218 ms. Sampled heap peak 112.921 to 125.744 MB is recorded, not claimed unchanged. - **Applicability and cost:** Helps identical synchronous scans within one statement. Does not share content across statements or change large-table streaming. The byte budget covers retained range payload, not whole-query or in-flight heap. Query results and FE/BE wire formats are unchanged; all four targeted queries match Hive reference results. This is not a full TPC-DS or RELEASE measurement. - **Behavior change:** Repeated Delta plans may share immutable ranges until planning ends when the existing flag is enabled. PlanningReuseHit identifies each reuse. Disable the session flag to bypass the memo. Shared FE/SPI/BE and other connectors are unchanged. ### Testing Local macOS ARM64: - build.sh --fe -j4: full FE build and Checkstyle passed; final test-only refinements also passed reactor validate/Checkstyle. - run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test', with follow-up memo/provider and opt-in scale runs: 102 distinct executed tests passed; the optional PlanningMetrics diagnostic class was not run. - Official regression runner: test_deltalake_basic (36 blocks), test_deltalake_scan_reuse (27), test_deltalake_parquet_deletion_vector (32), test_deltalake_hms_mixed_cast (14), all passed. The new .out was script-generated, independently audited, then compared without -genOut; existing .out files are unchanged. - Retained-data A/B: 10 alternating EXPLAIN pairs and 7 execution pairs per query, after three warmups per flag and readiness of all 24 table counts. All 88 Delta executions (including warmups/profiles) matched four Hive references. Separate three-sample storage traces verified requests and bytes. - Million-file streaming before/after: independent 512 MiB JVMs, three complete enumerations each, with FE/BE/Docker stopped for the matched comparison. - All 96 retained TPC-DS objects matched path/size/SHA-256 before and after. Temporary services were stopped and fixtures/volumes retained. ### Release note Repeated identical Delta Lake scans within one statement can plan faster, including Delta tables queried through an HMS catalog. The optimization uses the existing external scan task reuse setting and preserves streaming for large tables. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the connector README and local design/tasklist are updated. The broader Delta website documentation remains tracked by apache#69884. - Pre-submission status: Partially verified: local build, relevant tests and targeted ASAN measurements passed; Linux CI, RELEASE and a full TPC-DS rerun are not claimed for this commit. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, documentation and this description. - Human review: Pending review of this change.
Master added `16: optional TMongoScanDesc mongodb_params` to TTableFormatFileDesc (apache#70327), so `deltalake_params` moves to field 17. Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. Generated code accesses the field by name, so no Java or C++ code changes. build.sh keeps master's mongodb plugin entries and adds deltalake to both connector lists.
1 of 31 tasks
added 2 commits
November 13, 2026 00:02
### Background The Delta reader can stream file enumeration. While enumeration is in progress, EXPLAIN can show a file-count estimate for scan scheduling. The completed query profile records the selected files and generated splits. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta reader) PR Type: Test Problem Summary: The basic partition-pruning and Parquet deletion-vector regressions treated the streaming EXPLAIN estimate as the final split count. Read the counters from the completed profile of the query whose rows are checked. ### Details - **Failure:** With the default streaming threshold of 1, the estimate for the basic query was 3 when one split was generated. For the DV byte-range scan the estimate was 1 when five splits were generated. - **Cause:** Split scheduling uses a file-count hint before enumeration, including when one file generates multiple ranges. - **Stability evidence:** The basic check retains the partition fraction. The DV check retains the deletion-vector, row group, page-pruning, payload and row checks. Each scan is located by its unique SQL token and waits for complete Delta enumeration. ### Testing - Built the PR1 FE and regression framework and ran the existing golden outputs for test_deltalake_basic and test_deltalake_parquet_deletion_vector on local ASAN Doris, Hive Metastore and MinIO. - Both suites passed with default batch scheduling and threshold 1, and passed again with only session batch scheduling disabled. - Filesystem and HMS each reported one selected file for both queries, one generated split for the filtered basic query and five for the DV cross-split query. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Test changes and commit description - Human review: Pending
### Background Apache Doris PR apache#69904 adds the Delta Lake reader for the latest table state. Delta records each committed change as a numbered version of its transaction log. Reading an older version requires using that version for column binding, file selection and deletion vectors. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (required base, merge first) PR Type: Feature Problem Summary: Allow an ordinary Delta table reference to select a numeric version using `FOR VERSION AS OF`. This is a stacked draft based on `8ba8e67f29` from apache#69904. GitHub's Files changed tab includes the base PR until it merges. [Review only the changes for version reads](8ba8e67...codex/delta-version-as-of). ### Details - **Change map:** The Delta FE connector accounts for 65 added and 8 removed production lines. Unit tests and fixtures add 136 lines and remove 2. The Spark comparison adds 172 test lines and the framework generated 91 output lines. - **Scope:** Numeric versions, including zero, through Delta filesystem, Delta HMS and the HMS gateway. - **Design:** Load the requested Kernel snapshot, verify the target table UUID and reader features, and retain its schema. Reuse the checked handle for that table and version within the query. The file scan, projected columns and exact COUNT operate on the same version. - **Usage:** `SELECT * FROM catalog.db.t FOR VERSION AS OF 3`. - **Default behavior:** Ordinary reads continue to use the latest snapshot and the configured latest cache. - **Compatibility:** The table's current protocol must be readable when opening its handle. An unavailable log version fails explicitly. Timestamp selection and the history system table remain open in apache#69884. The HMS gateway still uses its existing statistics delegation for CBO estimates. Kernel reads a version while its required log and data files are available. Spark Delta 4.4.1 adds a separate, default-on age guard based on deleted-file retention, which can reject readable data. [Spark implementation](https://github.com/delta-io/delta/blob/v4.4.1/spark/src/main/scala/org/apache/spark/sql/delta/SnapshotManagement.scala#L1656-L1689) ### Testing - Full local macOS arm64 ASAN build with `FE_MAVEN_THREADS=4 ./build.sh --fe --be -j10`. Started the resulting FE and BE and verified a distributed query. - `bash run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test'` passed. The Delta connector and Delta metastore report 106 tests executed with zero failures, including all six new history tests. Two opt-in diagnostics were skipped. - Generated the golden output for `test_deltalake_version_as_of` via `run-regression-test.sh` and reran the unchanged suite without `-genOut`. Both runs passed with Spark 4.0.0, Delta 4.4.1, Hive Metastore and MinIO. All three catalog routes matched Spark on historical columns, types, ordered rows, counts, column mapping by name/ID, a physical deletion vector and multi-version joins. The new test exercises synchronous and streaming scheduling with the existing default threshold of 1. - The existing `test_deltalake_hive` passed. After rebasing on the split-profile test fix from apache#69904, reran `test_deltalake_basic`, `test_deltalake_parquet_deletion_vector` and `test_deltalake_version_as_of` with default streaming scheduling and `num_files_in_batch_mode=1`. All three passed against their existing golden outputs. ### Release note Delta Lake tables can be queried at a numeric version with `FOR VERSION AS OF`, including the schema and deletion vectors recorded at that version. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes, to accompany the Delta reader website documentation tracked in apache#69884. - Pre-submission status: Locally verified. Draft pending apache#69904. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests and PR description with AI assistance. - Human review: Pending.
xylaaaaa
force-pushed
the
codex/delta-version-as-of
branch
from
November 12, 2026 16:05
2365136 to
71897a6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Apache Doris PR #69904 adds the Delta Lake reader for the latest table state. Delta records each committed change as a numbered version of its transaction log. Reading an older version requires using that version for column binding, file selection and deletion vectors.
What problem does this PR solve?
Issue Number: #69884
Related PR: #69904 (required base, merge first)
PR Type: Feature
Problem Summary: Allow an ordinary Delta table reference to select a numeric version using
FOR VERSION AS OF. This is a stacked draft based on8ba8e67f29from #69904. GitHub's Files changed tab includes the base PR until it merges. Review only the changes for version reads.Details
SELECT * FROM catalog.db.t FOR VERSION AS OF 3.Testing
FE_MAVEN_THREADS=4 ./build.sh --fe --be -j10. Started the resulting FE and BE and verified a distributed query.bash run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test'passed. The Delta connector and Delta metastore report 106 tests executed with zero failures, including all six new history tests. Two opt-in diagnostics were skipped.test_deltalake_version_as_ofviarun-regression-test.shand reran the unchanged suite without-genOut. Both runs passed with Spark 4.0.0, Delta 4.4.1, Hive Metastore and MinIO. All three catalog routes matched Spark on historical columns, types, ordered rows, counts, column mapping by name/ID, a physical deletion vector and multi-version joins. The new test exercises synchronous and streaming scheduling with the existing default threshold of 1.test_deltalake_hivepassed. After rebasing on the split-profile test fix from [feature](external) Support reading Delta Lake tables natively with Delta Kernel #69904, rerantest_deltalake_basic,test_deltalake_parquet_deletion_vectorandtest_deltalake_version_as_ofwith default streaming scheduling andnum_files_in_batch_mode=1. All three passed against their existing golden outputs.Release note
Delta Lake tables can be queried at a numeric version with
FOR VERSION AS OF, including the schema and deletion vectors recorded at that version.Check List (For Author)
AI Disclosure