Repository navigation
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
1 of 31 tasks
Contributor
Author
|
run buildall |
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 2, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Ordinary external-table queries only consulted the table-keyed row-count cache, which could keep a cold estimate after versioned connector statistics completed. Prefer statistics at the statement's pinned snapshot for ordinary and versioned queries, reuse the existing size/row-width fallback, and forward the pinned snapshot through the Hive sibling gateway. This covers STAT-02. ### Release note External-table optimizer estimates can use completed statistics for the query's pinned snapshot without waiting for the table-only row-count cache to refresh. ### Check List (For Author) - Test: Full FE build and Checkstyle; 114 FE core and 267 Hive tests; focused compatibility run including 22 Paimon and 7 Iceberg tests; seven Delta p0 suites on the combined change, including new filesystem/HMS/Hive cardinality coverage. - Behavior changed: Yes, ordinary pinned queries prefer snapshot statistics. - Does this need documentation: Yes, Delta connector README and prepared EN/ZH website patch are updated in the accompanying statistics change; website PR pending.
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 2, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The native Delta connector did not supply basic table statistics to the optimizer. Cache logical row counts by user, table identity and version, summing active AddFile numRecords minus deletion-vector cardinality. Collect in a bounded background pool and return same-version CRC size while cold; missing row statistics stay unknown. Shared cache invalidation cancels pending work, and exact COUNT remains independent of optimizer estimates. Add shared-snapshot concurrency, real checkpoint/DV, lifecycle, authentication and million-file tests (STAT-01/03), plus filesystem/HMS/Hive regression coverage and connector documentation. ### Release note Native Delta Lake catalogs and Hive gateway queries gain basic versioned row/size statistics with asynchronous collection and a CRC size fallback. ### Check List (For Author) - Test: Full FE build and Checkstyle; 87 Delta and 2 metastore tests, including the opt-in million-file fixture in a 512 MiB JVM; focused compatibility/authentication rerun; seven Delta p0 suites passed on the final build. New .out generated by the official regression script and then verified without generation. Preserved all 1,227 MinIO objects, checked by path, size and SHA-256. - Behavior changed: Yes, Delta exposes basic optimizer statistics. Exact COUNT semantics and Delta data are unchanged; column NDV/histograms are not supplied. - Does this need documentation: Yes, connector README included; local design, tasklist and EN/ZH website patch updated. Website PR/full site build pending.
Contributor
Author
|
run buildall |
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 2, 2026
…stics ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Delta table row counts load asynchronously, so a post-statistics benchmark needs to distinguish ready statistics from a cold metadata lookup. Add an optional retained-manifest check that waits for Delta EXPLAIN cardinalities before timing and records Hive estimates. Keep the existing query texts, common controls, result comparator and three alternating measured pairs unchanged. Document the complete local TPC-DS rerun: the summed median gap narrows from 26.1% to 16.5%, with remaining regressions retained rather than replaced by focused samples. No production code or fixture data changes. ### Release note None ### Check List (For Author) - Test: Official regression script passed all 103 TPC-DS queries (824 executions) with matching results; all 24 Delta cardinalities match the independent manifest. A separate Q4 rerun passed, and eight complete diagnostic profiles were retained. All 1,227 MinIO objects match their original path/size/SHA-256 before and after. Source whitespace checks and code-review checkpoints passed. - Behavior changed: No production change. Optional benchmark warmup is disabled unless deltalakeBenchmarkStatisticsManifest is provided. - Does this need documentation: Yes, benchmark README includes warmup usage, before/after measurements, controls and remaining performance limitations.
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 2, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: COMPILE, FE UT, Cloud UT and BE UT all failed while merging the PR into master 5af2df3, before their build or test commands ran. Keep master's unified FileScannerV2 and split-producer lifecycle. Route Delta through the request-based streaming SPI, honor cancellation and partition selection, and validate the physical format on every Delta scan range. Propagate missing snapshot files and scheduling errors. Include Delta in the default FE unit-test reactor so the CI invocation exercises the connector. ### Release note Delta scans fail if a committed data or deletion-vector file cannot be found, instead of silently skipping a range when ignore_not_found_file is enabled. ### Check List (For Author) - Test: Full FE and ASAN BE builds; standard FE unit-test selection (857 passed, 2 opt-in cases skipped); FE Checkstyle, clang-format 16 and BE header hygiene. ASAN BE unit tests and a final FE lifecycle fixture recheck were still running or queued at commit time. - Behavior changed: Yes. Delta uses master's common scanner and streamed split lifecycle and fails on missing snapshot files. - Does this need documentation: Yes (connector README updated in this PR; the end-user documentation PR is tracked in apache#69884).
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 28313 ms |
Contributor
TPC-DS: Total hot run time: 151400 ms |
Contributor
ClickBench: Total hot run time: 25.2 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 3, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: File scans always use FileScannerV2. The basic and features Delta suites still exercised the same reader twice for both filesystem and HMS catalogs. Run them with enable_file_scanner_v2=true and preserve the retained query tags. The dedicated Hive gateway and native reader dispatch checks still exercise the deprecated false setting. ### Release note None ### Check List (For Author) - Test: Regenerated expected outputs with run-regression-test.sh and reran test_deltalake_basic and test_deltalake_features in comparison mode. Both suites passed both runs. All 146 retained query output blocks were byte-identical. Removed 32 redundant blocks (526 lines). - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 28448 ms |
Contributor
TPC-H: Total hot run time: 28753 ms |
Contributor
TPC-DS: Total hot run time: 152082 ms |
Contributor
ClickBench: Total hot run time: 24.35 s |
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 3, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Commit 598756e made ordinary plugin table reads resolve their pinned snapshot statistics on the planning thread. This changed the row-count path for Iceberg, Paimon and Hive as part of a Delta feature. Revert that commit's FE table and Hive statistics changes. Delta still collects logical rows asynchronously per version and supplies same-version CRC bytes when its count is unavailable. Exercise cold estimates in p0 and wait for the standard FE cache refresh before timing the p2 benchmark. ### Release note Ordinary catalog queries use the standard FE external-table row-count cache. Delta estimates in EXPLAIN may retain their initial value until an access triggers a refresh (10 minutes by default). Exact COUNT is independent. ### Check List (For Author) - Test: Full FE build and Checkstyle. All 102 targeted FE tests passed. test_deltalake_statistics passed for filesystem, HMS and Hive. With the default refresh interval, all 24 TPC-DS manifest counts matched and Q1 passed for both readers in test_deltalake_benchmark. - Behavior changed: Yes. Restore master's row-count policy for ordinary plugin table reads and its original Hive statistics delegation. - Does this need documentation: Yes (connector README and benchmark guide updated in this PR; website documentation is tracked in apache#69884).
Contributor
Author
|
run buildall |
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 3, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Merging master 7acfd9b left two Delta gaps. First, apache#69717 replaced table_reader.h's full ShardedKVCache definition with a forward declaration, but deltalake_reader_test.cpp constructs a ShardedKVCache and relied on the transitive include. Include format/format_common.h directly, as hudi_reader_test.cpp does. Second, apache#69300 made Parquet readers in one scan share their profile counters through TableReader's parquet profile cache. Hudi, Iceberg and the base reader pass that cache, but DeltaReader still built its ParquetReader without it, so every Delta file created its own set of counters. Pass the cache, and keep binary-UUID preservation disabled as before. ### Release note None ### Check List (For Author) - Test: No local build. The merge was conflict-free. Both changes follow the matching master code. COMPILE and BE UT are left to CI. - Behavior changed: No. Delta scan profiles reuse Parquet counters across files, as other formats do. - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 5, 2026
### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: Extracting decode_portable_deletion_vector_payload for inline Delta DVs left the framed Iceberg/Delta decoder checking the portable magic twice. Its first check runs before the CRC, and the payload function checked it again. The magic constant was also defined twice. Move the constant, the magic check and the Roaring decode into file-local helpers. The framed decoder again validates magic, then CRC, then decodes the bitmap once, with the same offsets and length it used before this PR. The inline payload decoder keeps its own size check and uses the same helpers. ### Release note None ### Check List (For Author) - Test: No local build. Existing Iceberg DV unit tests cover the magic and CRC rejection order. The Delta reader unit tests compare inline and on-disk decoding. Both are left to CI. - Behavior changed: No. Iceberg DV decoding is again identical to master's. - Does this need documentation: No
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 5, 2026
…n validation ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: validate_file_scan_support is the shared range gate for table markers that are no longer supported. The Delta integration also added a rule there that rejects non-Parquet Delta ranges. That state cannot occur. The Delta protocol stores data only as Parquet, and DeltaLakeScanRange always sends FORMAT_PARQUET. DeltaReader also asserts Parquet in create_file_reader. That runs for every data file, after prepare_split resets the reader format from the current range. Remove the shared rule and its unit test, so validate_file_scan_support is identical to master again. Keep the Delta entry in can_ignore_not_found_file. A missing file in a committed Delta snapshot must still fail the scan, as for Iceberg. ### Release note None ### Check List (For Author) - Test: No local build. The remaining Delta dispatch and missing-file unit tests are left to CI. - Behavior changed: No for FE-generated plans. A malformed non-Parquet Delta range now fails the DeltaReader assertion instead of returning NotSupported. - Does this need documentation: No
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 5, 2026
… the Delta PR ### What problem does this PR solve? Issue Number: close apache#69884 Related PR: apache#69904 Problem Summary: The Delta integration widened two fe-core catch blocks to Error. One is in executeWithProfileFinalization, on the batch planning path. The other is in the streaming profile finalizer. A lazily loaded plugin dependency then reported NoClassDefFoundError like any other failure. These are general hardening changes for every plugin connector, and Delta does not depend on them. Master's SplitProducer already catches Throwable while it iterates, closes the source and runs completion. The scan still fails without hanging and still calls finishSchedule. Of the four streaming tests added with the change, three cover existing master behavior. Only the profile-phase Error case needs the widened catch. Restore PluginDrivenScanNode and its scan profile test to master so the Delta PR leaves fe-core unchanged. The hardening can be proposed on its own. ### Release note None ### Check List (For Author) - Test: No local build. Both files are identical to master; left to CI. - Behavior changed: No relative to master. A Delta-only branch would have reported a profile-phase Error as a UserException. - Does this need documentation: No
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 28638 ms |
Contributor
TPC-DS: Total hot run time: 153387 ms |
Contributor
ClickBench: Total hot run time: 24.93 s |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
### Background A Delta deletion vector uses positions in the entire Parquet file. Page pruning can skip the first candidate page, and a scan split can start after earlier row groups. The id and its projected payload must stay aligned across both boundaries. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta Lake reader) PR Type: Test Problem Summary: The existing range queries retained a contiguous set of pages in one split. An incorrect payload could also have the expected length because every row in the fixture had the same value. ### Details - **Scenarios:** Store a distinct 64-character payload derived from each id. Check rows and profiles for disjoint selected pages, predicates matching only deleted rows, and file-global deletion positions during a 32 KiB split scan. Check COUNT, MIN and MAX on the empty selection. - **What existing tests miss:** Scanning one retained range does not require resuming after a fully rejected page. The previous projection could not distinguish payloads from different rows. - **Fails when broken:** Reviving a deleted row, omitting a live row or pairing an id with its neighbour's payload changes the golden result. The profile and plan checks require the intended pruning and splits. ### Testing Regenerated the Spark Delta fixtures on local MinIO and HMS. Ran the existing basic and Parquet baselines, generated this suite's golden file with the Doris regression runner, and reran it without -genOut. Both catalogs passed all 16 expected blocks each. An independent comparison verified all 9,990 surviving (id, payload) pairs per catalog. Both scan plans used five splits and the cross-split profiles read two row groups. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Codex updated the fixture and regression suite and ran the independent checks. The Doris regression runner generated the golden. - Human review: Pending review of the changes in PR apache#69904.
Contributor
Author
|
run buildall |
added 2 commits
November 5, 2026 22:24
### Background Delta Kernel uses file-level statistics to decide which Parquet files to schedule. Doris evaluates the original SQL predicates on the rows of the files that reach BE. A Hive catalog can dispatch Delta tables to the same Delta scan provider as a dedicated catalog. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (introduces the Delta reader) PR Type: Bugfix Problem Summary: A two-file Delta table containing 123.45 returned no rows for CAST(decimal_column AS BIGINT) = 123. Kernel skipped both files before Doris could evaluate the cast. Let the selected table's scan provider veto cast predicates while preserving independent conjuncts for file pruning. ### Details - **Frequency:** Reproducible with filesystem, dedicated HMS and Hive gateway catalogs in synchronous and streaming scans. - **Root cause:** FE unwraps CastExpr when converting a residual to a connector expression. Kernel compared the original decimal with 123. The catalog's default accepted the conversion even when its gateway routed a table to Delta. - **Why existing tests missed it:** Widening tests projected casts. Skipping tests compared columns directly. Neither used a decimal cast in WHERE that changes which files can contain matching rows. - **Fix:** Consult the selected provider's existing ConnectorPushdownOps capability while the original expression still contains its cast. Delta declines these predicates for pruning. BE retains the SQL predicate and an independent decimal conjunct can still prune. - **Other paths and branches:** The catalog's existing veto still takes precedence. Native Hive, Iceberg and Paimon readers use their existing policies. The connector API signatures and version stay the same. - **Behavior change:** Queries that previously skipped matching rows return those rows. ### Testing Built FE locally with build.sh. All 106 targeted FE tests passed, including Hive gateway routing, Iceberg, Paimon and JDBC controls. Reproduced the bug on the old FE and checked 36 queries against the same Parquet files on the fixed FE. Generated and reran the new regression on all three catalogs in both planning modes. All 24 golden blocks matched the fixture. All six mixed-predicate profiles selected one of two files. The existing basic, skipping and Parquet deletion vector regressions passed. ### Release note Delta readers return matching rows when CAST or TRY_CAST changes the value used in a filter, including when accessed through a Hive catalog. ### Check List (For Author) - Test: FE unit tests and Delta regression tests, detailed above - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Codex investigated the failure, prepared this change and its tests, and checked the regression-generated output. - Human review: Pending.
…catalog ### Background An HMS catalog can delegate different tables in one statement to the Delta and Iceberg connectors. Its scan nodes share catalog metadata, while Delta's scan provider rejects CAST-bearing file-pruning predicates and Iceberg retains its existing policy. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: None PR Type: Test Problem Summary: The existing Delta CAST regression tested each catalog entrance separately and did not mix Delta and Iceberg scans in one statement. Add mixed HMS queries and shared-metadata unit coverage so a catalog-wide CAST opt-out or a policy leaking between table scans cannot pass unnoticed. ### Details - **Scenarios:** UNION ALL and JOIN across Delta and Iceberg tables in one HMS catalog, both table orders, synchronous and streaming scans, and an independent ordinary Delta predicate. The FE test alternates two scan nodes sharing the same metadata instance in both orders. - **What existing tests miss:** A blanket HMS CAST opt-out returns correct rows for these fixtures but changes Iceberg's file-pruning behavior. Separate-catalog queries do not exercise the shared metadata boundary. - **Fails when broken:** Pushing the Delta CAST loses the matching 123.45 row. Disabling Iceberg's existing CAST pushdown removes its EXPLAIN predicate and makes the synchronous scan select both files instead of one. The ordinary Delta predicate must still prune one of its two files. ### Testing - Local run-regression-test.sh: test_deltalake_hms_mixed_cast with -genOut and an ordinary rerun both passed. Independently checked all 14 generated result blocks; the golden SHA-256 stayed unchanged. - Local run-regression-test.sh: test_deltalake_cast_pruning passed all 24 result blocks and test_deltalake_basic passed all 36 result blocks. - Local run-fe-ut.sh: 108 tests across PluginDrivenScanNodeCastPredicateTest, Hive routing, Delta, Iceberg, Paimon, JDBC and SPI passed, with no failures, errors or skipped tests. - Local FE Checkstyle reported zero violations; git diff --check passed. ### Notes for Reviewers Iceberg fixtures use integral decimal values to test its existing policy without relying on the separately known lossy CAST bug. Iceberg publishes SDK file counters only for synchronous planning; both modes check its EXPLAIN predicate and actual query results. Production code is unchanged in this commit. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Tests, generated regression output, and this description - Human review: Pending
Contributor
Author
|
run buildall |
### Background Delta's scan provider rejects CAST-bearing predicates before the shared converter strips their casts. The mixed HMS regression reads Delta and Iceberg tables in one statement, and originally asserted Iceberg's existing CAST pushdown behavior as well as query results. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: None PR Type: Test Problem Summary: The Delta connector unit tests did not directly guard its scan provider's CAST rejection. Meanwhile, the mixed regression would reject a future Iceberg correctness fix that disables unsafe CAST pushdown. Add a real-provider unit assertion and keep the mixed regression focused on query results and Delta pruning. ### Details - **Scenarios:** Obtain the real scan provider from the Delta connector and assert that it implements ConnectorPushdownOps and returns false for CAST pushdown. Keep the mixed HMS JOIN, UNION ALL, table-order and synchronous/streaming result checks. - **What existing tests miss:** Shared FE tests use provider stubs, so removing the real Delta override could pass them. Iceberg's current CAST pushdown policy is not a permanent correctness contract for a Delta regression. - **Fails when broken:** Removing Delta's interface opt-in or its false-returning override fails the connector unit test. Losing matching rows still fails the mixed golden results. Iceberg can change its own CAST policy without failing assertions about its plan text or selected file count. ### Testing - run-fe-ut.sh: DeltaLakeScanPlanProviderTest and PluginDrivenScanNodeCastPredicateTest passed all 32 tests, with no failures, errors or skips. - run-regression-test.sh: test_deltalake_hms_mixed_cast passed an ordinary rerun with all 14 result blocks. Its generated golden file is unchanged. - Delta connector Checkstyle reported zero violations; git diff --check passed. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex - Generated: Unit test, regression revisions, and this description - Human review: Pending
Resolve three conflicts with current master: - be/src/format_v2/table_reader.cpp: keep master's comment on the existing format_common.h include. - build.sh: keep master's MongoDB packaging step and the connector list that includes deltalake. - docker/thirdparties/docker-compose/iceberg/README.md: keep master's encrypted fixture text, followed by the Delta Lake regression section. Test: No local build. Build hygiene, git diff --check and bash -n build.sh passed. Compilation and tests are left to CI.
Contributor
Author
|
run buildall |
### Background Delta chooses synchronous or streaming split planning from the current snapshot's file count. Writers without a current checksum leave that count unknown, so even a two-file table streams and can start far more scanners than the selected file needs. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 reader) PR Type: Performance Problem Summary: Reuse complete file counts already available in the statement or in Delta's user/table/version-scoped statistics cache when no current checksum is available. Existing statistics collection counts files, and exhaustive normal scans publish their already-observed total without submitting a statistics task. This lets subsequent small-table queries use synchronous planning while keeping large and unknown tables streaming. Changes are confined to the Delta plugin, its tests, and its README. ### Details - **Why it was slow:** The streaming scanner path does not cap scanner creation by the actual split count. On the retained two-file fixture, the automatic path started 144 scanners to read one selected file and one row; forced synchronous planning started one. - **Optimization:** Check the current CRC, then the statement's complete count, then the same user/table identity/version cache entry. The gate never enumerates files, starts statistics work, or waits for storage. A fully consumed scan without partition-pruned replay publishes its total into the entry captured before scanning, so late reports cannot repopulate invalidated entries. Background row-count sessions can have a different identity from SQL sessions, making normal-scan publication necessary without relaxing user isolation. - **Results:** On local macOS ARM64 with the same ASAN BE and retained data, 16 alternating warm pairs per entrance reduced automatic median latency from 61.614 to 41.420 ms (filesystem), 66.370 to 38.509 ms (HMS), and 57.911 to 32.874 ms (HMS gateway). Scanner counts became one; all returned 123.45. Forced-sync controls were 35.978/41.056, 44.302/37.627, and 32.914/32.370 ms before/after. Million-file metadata enumeration medians were 1493.563/1451.023 ms with a 512 MiB heap; both scale tests passed. - **Applicability and cost:** Warm snapshots with an already-known complete small file count benefit. Cold unknown counts still stream. Reuse existing cache capacity, TTL, invalidation and executor ownership; add one count, not a file list. These are small local ASAN measurements, not a TPC-DS claim. Sampled scan heap peaks increased from 133327520 to 238172480 bytes while remaining within 512 MiB; actual million-file data IO was not measured. - **Behavior change:** Small Delta tables may select synchronous split planning after a complete count is available. Row results, public row-count cache behavior, BE scheduling, other connectors, and wire/storage formats are unchanged. ### Testing - Local `./build.sh --fe -j4`: passed, including Checkstyle. Existing ASAN BE binary and local source configurations were unchanged by SHA-256. - Official FE UT runner for `DeltaLake*Test,DeltaKernel*Test`: 92 actual tests passed, zero failures/errors; two optional classes skipped in that invocation. Separately enabled `DeltaLakePlanningScaleTest` with the retained million-file checkpoint and `-Xmx512m`: two tests passed. The diagnostic PlanningMetrics class remained disabled. - Tests cover cached/unknown/current-CRC counts, threshold equality, missing row statistics, user/version/UUID/path isolation, no work submission or IO wait, late invalidated reports, partial and partition-pruned scans, large-file splitting, and large-table streaming after count reuse. - Official regression runner: basic, statistics, cast_pruning, parquet_deletion_vector, hms_mixed_cast, and planning passed: six suites, 129 result blocks. Existing golden files were unchanged. The planning suite initially targeted the wrong local MinIO; using the retained fixture's endpoint resolved the missing-database error without changing the test or data. - `git diff --check`: passed. ### Release note Delta Lake queries can avoid excessive scanner startup for small tables without a checksum once a complete file count is available. Large tables and snapshots with unknown file counts continue to use streaming planning. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the Delta connector README documents selection, cache reuse and cold-query limitations. No new user configuration is introduced. - Pre-submission status: Partially verified: local tests and targeted performance checks passed; full TPC-DS and large-table data IO performance were not rerun. ### AI Disclosure - Tool: Codex (GPT-6) - Generated: Implementation, tests, README, local evidence report and this description. - Human review: Pending.
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 28591 ms |
Contributor
TPC-DS: Total hot run time: 154395 ms |
Contributor
ClickBench: Total hot run time: 23.94 s |
### Background Delta fixes a snapshot for each statement, then enumerates its active files to plan native Parquet ranges. Repeated scans currently reopen the same checkpoint and reconstruct the same ranges. Other connectors already use the existing statement planning scope and external scan task reuse session flag. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta M1 implementation) PR Type: Performance Problem Summary: Repeated identical synchronous Delta scans redo checkpoint IO and planning. Reuse completed immutable ranges and metrics within the existing planning scope, with typed request keys, bounded retention and cancellable single-flight loading. The change applies to dedicated Delta and HMS gateway reads. Production changes are confined to the Delta plugin; most added lines are tests and generated output. ### Details - **Change map:** Four production Java files (269 added / 50 removed lines), five test files (549 added / 4 removed), 244 generated expected-output lines and 33 README lines. The new memo owns keys, admission and concurrency; the existing scan body is extracted without changing its pruning semantics. Provider/scale tests and a three-route SQL suite cover reuse and bypass. - **Why it was slow:** Each scan independently enumerated the pinned snapshot, including repeated checkpoint range GETs, decode and range construction. - **Optimization:** Use the existing enable_external_scan_task_reuse flag and planning-only statement scope. Key by catalog/query/user, table coordinates/UUID/version, typed filter, partition selection, COUNT, split size and time zone. Coalesce identical loads outside scope/memo locks; publish only completed results and per-node metrics. Retry failed/cancelled loads; let waiters cancel independently. Retain at most 128 entries, 16,384 ranges and 16 MiB estimated range payload; oversized synchronous results are returned without retention. Streaming never enters the memo. - **Results:** Same-build alternating local ASAN/SF1 A/B: Q88 EXPLAIN 323.616 to 152.279 ms and execution 2268.353 to 2050.009 ms (-9.6%); Q80 EXPLAIN -23.9%, execution -1.9%; Q4 EXPLAIN -49.4%, execution essentially flat; Q22 has no hits and execution is flat (+0.5%). Three independent traces each show checkpoint GETs 160 to 55 / 90 to 60 / 90 to 30 for Q88/Q80/Q4; Q22 stays at 15. A million-file streaming fixture remains bounded in a 512 MiB JVM: enumeration median 1493.940 to 1484.218 ms. Sampled heap peak 112.921 to 125.744 MB is recorded, not claimed unchanged. - **Applicability and cost:** Helps identical synchronous scans within one statement. Does not share content across statements or change large-table streaming. The byte budget covers retained range payload, not whole-query or in-flight heap. Query results and FE/BE wire formats are unchanged; all four targeted queries match Hive reference results. This is not a full TPC-DS or RELEASE measurement. - **Behavior change:** Repeated Delta plans may share immutable ranges until planning ends when the existing flag is enabled. PlanningReuseHit identifies each reuse. Disable the session flag to bypass the memo. Shared FE/SPI/BE and other connectors are unchanged. ### Testing Local macOS ARM64: - build.sh --fe -j4: full FE build and Checkstyle passed; final test-only refinements also passed reactor validate/Checkstyle. - run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test', with follow-up memo/provider and opt-in scale runs: 102 distinct executed tests passed; the optional PlanningMetrics diagnostic class was not run. - Official regression runner: test_deltalake_basic (36 blocks), test_deltalake_scan_reuse (27), test_deltalake_parquet_deletion_vector (32), test_deltalake_hms_mixed_cast (14), all passed. The new .out was script-generated, independently audited, then compared without -genOut; existing .out files are unchanged. - Retained-data A/B: 10 alternating EXPLAIN pairs and 7 execution pairs per query, after three warmups per flag and readiness of all 24 table counts. All 88 Delta executions (including warmups/profiles) matched four Hive references. Separate three-sample storage traces verified requests and bytes. - Million-file streaming before/after: independent 512 MiB JVMs, three complete enumerations each, with FE/BE/Docker stopped for the matched comparison. - All 96 retained TPC-DS objects matched path/size/SHA-256 before and after. Temporary services were stopped and fixtures/volumes retained. ### Release note Repeated identical Delta Lake scans within one statement can plan faster, including Delta tables queried through an HMS catalog. The optimization uses the existing external scan task reuse setting and preserves streaming for large tables. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes; the connector README and local design/tasklist are updated. The broader Delta website documentation remains tracked by apache#69884. - Pre-submission status: Partially verified: local build, relevant tests and targeted ASAN measurements passed; Linux CI, RELEASE and a full TPC-DS rerun are not claimed for this commit. ### AI Disclosure - Tool: Codex - Generated: Implementation, tests, documentation and this description. - Human review: Pending review of this change.
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
xylaaaaa
changed the title
Master added `16: optional TMongoScanDesc mongodb_params` to TTableFormatFileDesc (apache#70327), so `deltalake_params` moves to field 17. Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1. Generated code accesses the field by name, so no Java or C++ code changes. build.sh keeps master's mongodb plugin entries and adds deltalake to both connector lists.
Contributor
Author
|
run buildall |
### Background The Delta reader can stream file enumeration. While enumeration is in progress, EXPLAIN can show a file-count estimate for scan scheduling. The completed query profile records the selected files and generated splits. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (Delta reader) PR Type: Test Problem Summary: The basic partition-pruning and Parquet deletion-vector regressions treated the streaming EXPLAIN estimate as the final split count. Read the counters from the completed profile of the query whose rows are checked. ### Details - **Failure:** With the default streaming threshold of 1, the estimate for the basic query was 3 when one split was generated. For the DV byte-range scan the estimate was 1 when five splits were generated. - **Cause:** Split scheduling uses a file-count hint before enumeration, including when one file generates multiple ranges. - **Stability evidence:** The basic check retains the partition fraction. The DV check retains the deletion-vector, row group, page-pruning, payload and row checks. Each scan is located by its unique SQL token and waits for complete Delta enumeration. ### Testing - Built the PR1 FE and regression framework and ran the existing golden outputs for test_deltalake_basic and test_deltalake_parquet_deletion_vector on local ASAN Doris, Hive Metastore and MinIO. - Both suites passed with default batch scheduling and threshold 1, and passed again with only session batch scheduling disabled. - Filesystem and HMS each reported one selected file for both queries, one generated split for the filtered basic query and five for the DV cross-split query. ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Test changes and commit description - Human review: Pending
xylaaaaa
pushed a commit
to xylaaaaa/doris
that referenced
this pull request
Nov 12, 2026
### Background Apache Doris PR apache#69904 adds the Delta Lake reader for the latest table state. Delta records each committed change as a numbered version of its transaction log. Reading an older version requires using that version for column binding, file selection and deletion vectors. ### What problem does this PR solve? Issue Number: apache#69884 Related PR: apache#69904 (required base, merge first) PR Type: Feature Problem Summary: Allow an ordinary Delta table reference to select a numeric version using `FOR VERSION AS OF`. This is a stacked draft based on `8ba8e67f29` from apache#69904. GitHub's Files changed tab includes the base PR until it merges. [Review only the changes for version reads](8ba8e67...codex/delta-version-as-of). ### Details - **Change map:** The Delta FE connector accounts for 65 added and 8 removed production lines. Unit tests and fixtures add 136 lines and remove 2. The Spark comparison adds 172 test lines and the framework generated 91 output lines. - **Scope:** Numeric versions, including zero, through Delta filesystem, Delta HMS and the HMS gateway. - **Design:** Load the requested Kernel snapshot, verify the target table UUID and reader features, and retain its schema. Reuse the checked handle for that table and version within the query. The file scan, projected columns and exact COUNT operate on the same version. - **Usage:** `SELECT * FROM catalog.db.t FOR VERSION AS OF 3`. - **Default behavior:** Ordinary reads continue to use the latest snapshot and the configured latest cache. - **Compatibility:** The table's current protocol must be readable when opening its handle. An unavailable log version fails explicitly. Timestamp selection and the history system table remain open in apache#69884. The HMS gateway still uses its existing statistics delegation for CBO estimates. Kernel reads a version while its required log and data files are available. Spark Delta 4.4.1 adds a separate, default-on age guard based on deleted-file retention, which can reject readable data. [Spark implementation](https://github.com/delta-io/delta/blob/v4.4.1/spark/src/main/scala/org/apache/spark/sql/delta/SnapshotManagement.scala#L1656-L1689) ### Testing - Full local macOS arm64 ASAN build with `FE_MAVEN_THREADS=4 ./build.sh --fe --be -j10`. Started the resulting FE and BE and verified a distributed query. - `bash run-fe-ut.sh --run 'DeltaLake*Test,DeltaKernel*Test'` passed. The Delta connector and Delta metastore report 106 tests executed with zero failures, including all six new history tests. Two opt-in diagnostics were skipped. - Generated the golden output for `test_deltalake_version_as_of` via `run-regression-test.sh` and reran the unchanged suite without `-genOut`. Both runs passed with Spark 4.0.0, Delta 4.4.1, Hive Metastore and MinIO. All three catalog routes matched Spark on historical columns, types, ordered rows, counts, column mapping by name/ID, a physical deletion vector and multi-version joins. The new test exercises synchronous and streaming scheduling with the existing default threshold of 1. - The existing `test_deltalake_hive` passed. After rebasing on the split-profile test fix from apache#69904, reran `test_deltalake_basic`, `test_deltalake_parquet_deletion_vector` and `test_deltalake_version_as_of` with default streaming scheduling and `num_files_in_batch_mode=1`. All three passed against their existing golden outputs. ### Release note Delta Lake tables can be queried at a numeric version with `FOR VERSION AS OF`, including the schema and deletion vectors recorded at that version. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes, to accompany the Delta reader website documentation tracked in apache#69884. - Pre-submission status: Locally verified. Draft pending apache#69904. ### AI Disclosure - Tool: OpenAI Codex (GPT-6). - Generated: Implementation, tests and PR description with AI assistance. - Human review: Pending.
Contributor
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: #69884
Problem Summary:
Add native read-only Delta Lake support through a Kernel 4.4.1 FE connector and
FileScannerV2's native Parquet reader. The reader preserves the transaction-log
snapshot, logical/physical column mapping and deletion-vector row positions.
Catalogs can discover tables through Hive Metastore or a filesystem warehouse;
existing Hive catalogs can route Delta tables to the same native path.
This is a draft for implementation review. Performance acceptance, the remaining
storage matrix, the website documentation PR and final acceptance are open.
Integrated master
5af2df3231infcc32e2885after the original TeamCity COMPILE, FE UT, Cloud UT and BE UT jobs all stopped at the same merge conflicts. Delta now uses the request-based streaming SPI and master's unified scanner; closing sources, stopping failed queries and publishing successful scan profiles follow the current split lifecycle. The standard FE UT reactor now includes Delta.Delta validates the physical format of every range and propagates missing committed data or deletion-vector files instead of returning a partial snapshot.
Local integration checks: full FE/Checkstyle and ASAN BE builds passed. Of 859 selected FE tests, 857 passed and two opt-in cases were skipped; the final real SplitAssignment fixture was also rerun (39/39 overlapping lifecycle/producer tests passed). The ASAN BE runner passed all 239 selected tests, including 3 scanner-dispatch/format/missing-file cases, 14 Delta reader and 3 deletion-vector cases. C++ clang-format 16, header hygiene and changed-line clang-tidy passed.
On integration commit
fcc32e2885, COMPILE, FE UT, Cloud UT and BE UT all passed. The separate FE coverage check failed on that commit.The November 3 test change runs the Delta basic and features suites with the deprecated scanner option set to true. The Hive gateway and native dispatch tests retain false-setting coverage. Both suites passed generation and comparison against the retained fixtures. Removing 32 duplicate result blocks dropped 526 expected-output lines. All 146 retained blocks were byte-identical. Full CI requested on
338108051b.The row-count change in
3d4899eb94restores master’s FE table-level row-count cache for ordinary reads and removes the extra Hive snapshot-statistics forwarding. Delta keeps versioned logical counts and same-version CRC fallback. An access after the FE refresh interval (10 minutes by default) triggers an asynchronous update. The full FE build and 102 targeted unit tests passed. The statistics p0 passed for filesystem, HMS and Hive. Under the default refresh interval all three routes converged to 3 and 35 rows for the two reference tables. All 24 TPC-DS manifest counts matched and Q1 passed on both readers in the focused p2 run. Full CI requested.CI diagnosis and verification. Benchmark, SQL and scale figures below are historical results at or before measurement head
f0a229f7b9.Implemented scope:
statement-pinned snapshots and catalog-owned filesystem clients.
timestamps/NTZ and the tested type-widening combinations.
across byte-range splits. Legacy native bitmap encoding is rejected.
unfiltered metadata COUNT when all required file statistics are present,
subtracting deletion-vector cardinality. Other cases scan normally.
Fixed latest-version TTL (hits do not extend it), user-scoped snapshot caching,
REFRESH invalidation and lazy streaming split enumeration.
by user/table identity/version and exclude deletion-vector cardinality. A bounded
background pool collects active-file statistics; cold lookups use same-version
CRC bytes when available. Ordinary Delta reads use the standard FE row-count
cache through direct and Hive catalogs. Exact COUNT is independent. Shared real
checkpoint/DV snapshots have concurrent scan coverage.
Streaming sources close before scheduling completion; close failures propagate.
shared-file Hive/Delta benchmark. Normal restarts reuse initialized fixtures.
Time travel, Unity/Glue catalogs, variant, Delta writes and column-level NDV /
histogram statistics are out of scope for this PR. Basic table row/size statistics are included. No additional log/checkpoint content cache is introduced: the
million-file measurements did not justify it here.
Compatibility: all new Thrift fields are optional.
deltalake_paramsusesTTableFormatFileDescfield 17: master took field 16 formongodb_params(#70327),so the merge of master at
e321ede256moved Delta to 17. Field 17 is unused on master andbranch-4.2/4.1/4.0/3.1 (highest existing IDs: 16/13/12/10/9); no ID is reused and
existing meanings are preserved. Native Delta queries require compatible FE/BE
versions; older BE versions do not implement this reader.
Outbound HTTP / SSRF surface: object-store endpoints supplied through catalog
properties cause FE log/checkpoint reads and BE Parquet/DV reads through existing
filesystem clients. Configuring endpoints requires catalog CREATE_PRIV or
ALTER_PRIV; SELECT_PRIV can trigger reads. This implementation reuses
fe-filesystem configuration and does not fetch/load runtime artifacts from
catalog-supplied URLs.
Release note
Add native read-only Delta Lake catalogs backed by Hive Metastore or filesystem
warehouses, plus native Delta reads through Hive catalogs. Support mapped/evolved
schemas, portable deletion vectors, V2 checkpoints, pruning, metadata COUNT and
streaming file planning, with asynchronous versioned basic optimizer statistics.
A missing file referenced by the Delta snapshot fails the query.
Check List (For Author)
Test
Docker environment; million-file planning p2 passed; MinIO and ordinary
HDFS 3.2.1 storage p2 passed using the same generated expected output.
75 Delta connector, 2 metastore, 25 FE core and 274 Hive-related tests passed.
clang-format 16, header hygiene and clang-tidy passed. Thirteen physical
Parquet fixtures matched PyArrow; semantic golden fixtures matched Spark.
Metadata COUNT profiles verified zero data reads. Three million-file
source passes completed in a 512 MiB JVM (first split 22–60 ms, enumeration
1.46–1.86 s, sampled heap peak about 200 MiB). This synthetic fixture has
no physical data objects and measures planning/count only. Restart checks
preserved all 1,227 MinIO objects by path, size and SHA-256.
cc12b56819): final full FE buildand Checkstyle; 87 Delta + 2 metastore tests, broad 267 Hive / 114 FE core
tests, and focused 22 Paimon / 7 Iceberg compatibility tests passed. Eight
concurrent scans share one real checkpoint/DV snapshot. Seven Delta p0
suites passed on the final FE build, including the new statistics suite
through filesystem/HMS/Hive; expected output was generated by the official
script and then verified without generation. Background auth/classloader,
invalidation, cancellation, retry, identity/version isolation and missing
stats are covered. Existing 1,227 MinIO objects still match their retained
path/size/SHA-256 manifest.
completion 2.170 s, sampled heap peak 100,255,200 bytes (about 96 MiB).
This synthetic fixture measures metadata planning/statistics, not data scans.
executions passed in one official suite, with three alternating measured
pairs and unchanged controls. All 24 Delta cardinalities matched the
independent fixture manifest before timing. Eight complete diagnostic
profiles and a separate Q4 repeat are retained without replacing the full
samples. All 1,227 data objects still match the retained checksum manifest.
Behavior changed:
streaming source-close errors now fail scheduling instead of being logged
after completion. Delta provides logical row counts and CRC size estimates
through the existing table statistics API.
Does this need documentation?
Local design/tasklist and paired EN/ZH website patches are updated. Documentation PR and
full website build are pending.
Performance evidence and remaining gates
On the November 2 build with the earlier FE statistics path, all 103 TPC-DS queries
passed and the measured gap narrowed from 26.1% to 16.5%. Performance acceptance remains open.
The full rerun uses the same retained SF1 data, ASAN build, query settings and
Q85/Q30 controls. An optional independent-manifest check waits for Delta's
asynchronous statistics before timing and records Hive estimates. Medians still
come from three alternating pairs, excluding first executions. Delta's total
falls 10.46% and Hive's 3.07%; this before/after observation does not isolate the
cause of every timing change.
Separate complete profiles confirm Q43's build side falls from 2.75M rows to
366 and Q72 no longer builds from 11.745M inventory rows. Q22 is still about 40%
slower; Q4/Q11/Q74 lost earlier Delta advantages. A separate Q4 repeat still shows
a 44.2% gap and is excluded from the 103-query aggregate. Detailed samples and
before/after CSV/JSON remain in the local benchmark report; the benchmark README
contains the reproducible setup and result summary.
Earlier baseline, retained below: TPC-H has not been rerun. The following
125-query comparison predates basic statistics; it is not a new result.
All 22 TPC-H and 103 TPC-DS queries completed cross-reader result comparison and
three alternating paired repetitions on the same retained SF1 Parquet objects.
This combines several official regression invocations, with original failures
retained; it is not an unmodified full-suite pass. Both paths use FileScannerV2,
four pipeline tasks, disabled SQL/file caches, UTC, 8 GiB/query and 120 seconds.
These are local relative observations, not RELEASE performance, concurrent
throughput or certified TPC results. TPC-DS uses the repository's SF100 query
parameters on SF1 data; 12 queries have empty results. Excluding them, the
TPC-DS ratio is 1.215.
Explicit controls: Q7 and Q85 use the same fixed join order on both readers after
default Hive plans exceeded memory; Q15 uses an equivalent CTE; Q30 projects the
current generator's review-date string column. Results use the framework's typed
comparator (exact integer/decimal, floating tolerance 1e-8). Q39's original exact
comparison differed by one ULP in one cell; the original difference is retained.
Nine complete profile pairs show different join build sides, runtime-filter
placement and scanner parallelism. For example, Q72's Delta plan builds from
11.745M inventory rows versus Hive's 62,699-row filtered build side; Q43 builds
from 2.750M sales rows versus Hive's 366 date rows. Missing statistics were one observed planning limitation; basic row/size
statistics are now available. Column statistics remain absent, and the full gap
has not been attributed solely to this omission. Q22 variability and small-table streaming startup still
need isolation; performance parity is not accepted yet.
available. MinIO does not establish acceptance against live cloud services.
5af2df3231and resolve the four conflicts against the current scanner and connector SPI.The local tasklist currently records 63/67 completed items, including the three
basic-statistics/concurrency follow-ups. Environment configs,
local toolchains, large generated datasets and diagnostic logs are excluded from
this PR; reusable fixture generators and regression suites are included.
Check List (For Reviewer who merge this PR)