Repository navigation
[Tracking] Native Delta Lake support #69884
Description
Goal
Doris should read and write Delta Lake tables natively, so users can analyze lakehouse data written by Spark and Databricks without copying it. Today Doris can only reach Delta tables through the Trino connector (JNI, read-only, no time travel) or, for tables with UniForm enabled, through an Iceberg catalog.
This issue tracks the whole plan, from basic reads to writes and table maintenance. Every milestone follows the same rules:
- Query results match Spark on the same table and SQL.
- A Delta feature Doris does not support yet fails with its name; Doris never returns partial or wrong results.
- Anything Doris writes must be readable by Spark, which can keep writing to the table.
Roadmap
Reads come first. Writes open in the order INSERT → DELETE → UPDATE / MERGE, because each step builds on the previous one.
M1 Basic read ──► M2 Advanced read
│
└─────────► M3 INSERT ──► M4 DELETE ──► M5 UPDATE / MERGE
│
└───────► M6 Maintenance and DDL
Gates: M1 must match Spark before later milestones start; each write milestone must pass "Spark reads back what Doris wrote" before the next one opens.
| Milestone | Scope | Why it is here | Acceptance | Status |
|---|---|---|---|---|
| M1 Basic read | Latest snapshot; partition pruning and data skipping; deletion vectors, column mapping, timestamp_ntz, type widening, V2 checkpoints; HMS (including Delta tables in Hive catalogs) and filesystem catalogs |
Reads are the prerequisite for everything else and cover most tables Spark and Databricks write | Results match Spark; unsupported features fail clearly; performance close to Hive over the same Parquet files | Implemented in #69904, under review |
| M2 Advanced read | Time travel and $history; Unity Catalog and AWS Glue; variant; Change Data Feed reads; optimizer statistics; materialized view partition refresh |
Widens the tables and scenarios Doris can read; Databricks users depend on Unity Catalog. Does not block writes | Time travel matches Spark; Unity Catalog managed tables are readable | Numeric version reads in draft #70536 |
| M3 INSERT | CREATE / DROP TABLE, CTAS; INSERT INTO and INSERT OVERWRITE (including dynamic partition overwrite); writes to Unity Catalog managed tables |
Appends are the simplest writes with the fewest conflicts | Spark reads the data back and keeps writing; concurrent appends with Spark lose nothing | Not started |
| M4 DELETE | DELETE, writing deletion vectors when enabled, otherwise rewriting files |
The simplest row-level change; validates deletion vector writes | Spark sees the same result as Doris | Not started |
| M5 UPDATE / MERGE | UPDATE, MERGE INTO |
Depends on M4's deletion vector writes; the most complex semantics | Same result as Spark running the same statement | Not started |
| M6 Maintenance and DDL | OPTIMIZE, VACUUM, ALTER TABLE (add and rename columns), RESTORE |
Usually done by Spark or Databricks, so lowest priority | Spark still reads and writes normally afterwards | Not started |
Delta table features by milestone
| Feature | Read | Write |
|---|---|---|
columnMapping, timestampNtz, typeWidening, v2Checkpoint |
M1 | M3 |
deletionVectors |
M1 | M4 |
vacuumProtocolCheck |
M1 (recognized) | M6 |
catalogManaged |
M2 | M3 |
variantType, variantShredding |
M2 | M3 |
changeDataFeed |
M2 (table_changes) |
After M5 |
inCommitTimestamp |
M2 (time travel) | M3 |
appendOnly, invariants, rowTracking, domainMetadata, materializePartitionColumns |
— (writer-only) | M3 |
clustering |
— (writer-only) | M3 (append), M6 (OPTIMIZE) |
checkConstraints, generatedColumns, allowColumnDefaults, identityColumns |
— (writer-only) | After M3 |
icebergCompatV1 / icebergCompatV2 (UniForm) |
— (writer-only) | Not planned |
M1 · Basic read
| Step | Scope | Status |
|---|---|---|
| M1.1 Minimal read | Delta catalog on HMS or a filesystem path; latest snapshot; scalar, decimal, date, timestamp and nested types; partitioned tables; classic and multipart checkpoints; clear errors for unsupported reader features | Done |
| M1.2 Pruning | Partition pruning; data skipping on file min/max/null counts; Parquet predicate pushdown; COUNT(*) from file row counts |
Done |
| M1.3 Schema-evolution features | Column mapping (name and id, including renamed and dropped columns); type widening; timestamp_ntz |
Done |
| M1.4 Deletion vectors | Relative, absolute and inline deletion vectors, combined with column mapping, partitions and checkpoints | Done |
| M1.5 Metadata formats and large tables | V2 checkpoints with sidecars; log compaction (ignored by readers); vacuumProtocolCheck; snapshot caches; streaming split generation for very large tables |
Done |
| M1.6 Integration and coverage | Delta tables in Hive catalogs; storage on S3, OSS, COS, OBS, HDFS, Azure and GCS; performance comparison with Hive | Hive routing done; MinIO (S3-compatible) and HDFS verified, other storage pending; performance acceptance open |
Design
- FE: a new
fe-connector-deltalakeplugin built on Delta Kernel Java 4.4.1. It resolves tables, loads snapshots, prunes partitions and files, and generates splits. Doris FE itself only gained a per-table veto for CAST predicate pushdown (PluginDrivenScanNode,ConnectorPushdownOps). - BE: reads through FileScannerV2. A thin
DeltaReadersubclassesTableReaderand uses the existing V2ParquetReader; it adds column mapping and deletion vectors. There is no JNI path, because Delta updates and deletes only produce new Parquet files and row-index deletion vectors. - Thrift: two optional structs,
TDeltaLakeFileDescandTDeltaLakeDeletionVectorDesc, carried inTTableFormatFileDescfield 17 (deltalake_params). Field 17 is unused on master, branch-4.2, branch-4.1, branch-4.0 and branch-3.1; field 16 went to MongoDB ([feature](mongodb) Add native catalog, reader and BSON SerDe #70327). - Safety: reader features outside an allowlist fail at planning time. An older BE that receives a
deltalakerange returnsNotSupportedinstead of reading it as plain Parquet, so a rolling upgrade cannot return deleted rows.
What #69904 delivers
| Area | What works |
|---|---|
| Catalogs | type=deltalake with deltalake.catalog.type=hms or filesystem (warehouse/<db>/<table>/_delta_log). An existing type=hms catalog detects tables with spark.sql.sources.provider=delta and routes them to the same reader. |
| SQL | SHOW DATABASES, SHOW TABLES, DESC, information_schema and SELECT on partitioned and unpartitioned tables. Writes are rejected until M3. |
| Reader features | columnMapping (none, name, id), deletionVectors (inline and on disk), timestampNtz, typeWidening and typeWidening-preview (validated conversions only), v2Checkpoint, vacuumProtocolCheck. Writer-only features are ignored. |
| Types | Scalars, decimals, timestamp (optionally TIMESTAMPTZ), timestamp_ntz, binary (optionally VARBINARY), arrays, maps and structs, including renamed and dropped columns. |
| Snapshot consistency | Each statement pins one snapshot per table for its schema, partitions, files and count. The table UUID is checked so a table recreated at the same path is detected. Latest-version and snapshot caches are isolated per user and honor REFRESH. |
| Pruning | Partition pruning and file skipping on Delta statistics. Predicates are used only to skip files and are re-evaluated in BE; CAST predicates are never pushed (they could skip matching rows). The Parquet reader also uses row-group statistics and the page index. |
COUNT(*) |
An unfiltered count is answered from file row counts minus deletion-vector cardinalities when every file has statistics; otherwise Doris scans. |
| Statistics | Row count and size for the optimizer, collected asynchronously per snapshot version; a same-version CRC supplies the size while collection runs. Column statistics are left to M2. |
| Scan planning | Large tables stream splits with cancellation and cleanup. Known file counts let small tables plan synchronously when below num_files_in_batch_mode. With enable_external_scan_task_reuse, identical synchronous scans in one statement reuse their ranges. EXPLAIN and profiles show the snapshot version, files, partitions and splits. |
| Failure handling | A missing data or deletion-vector file fails the query instead of returning a partial snapshot. |
| Tests | FE and BE unit tests; 14 p0 regression suites under regression-test/suites/external_table_p0/deltalake; p2 suites for million-file planning, storage and a Hive comparison benchmark. Fixtures come from Delta v4.4.1 golden tables plus Spark 4.0 written tables (optional docker profile). |
Known limitations in #69904
- The FE default
num_files_in_batch_modeis1, so ordinary non-empty tables stream splits even when their file count is known. - Checkpoint contents are not cached across statements; repeated streaming scans of one table still repeat checkpoint reads.
- Delta regression suites are opt-in and are not yet part of the external CI pipeline.
- Kernel internal APIs are used only in
DeltaKernelCompat, pinned to Kernel 4.4.1. - Performance is close to, but not yet at, parity with Hive over the same files: the latest full TPC-DS SF1 run (local ASAN, threshold 1024) was 4.69% slower overall. See the benchmark comment.
Before #69904 can merge
- Resolve the merge conflicts with master (merged in
e321ede256;deltalake_paramsmoved to Thrift field 17) - Linux CI passes on the latest head (
run buildallrequested fore321ede256) - Code review
- English and Chinese website documentation PR, with a full website build
Remaining M1 work after #69904
- Validate OSS, COS, OBS, Azure, GCS and Kerberos HDFS when environments are available
- Run the Delta regression suites in the external CI pipeline, using a prebuilt fixture package instead of downloading jars and golden tables
- Plan small tables synchronously under the default batch-mode setting, using a Delta-side small-table rule (for example, files ≤
split_batch_size); measure first - Reuse completed small streaming plans within one statement (assessment)
- Bounded cross-statement cache for checkpoint and log contents
- Serve
COUNT(*)from the versioned row counts already in the statistics cache - Close the remaining TPC-DS gaps (multi-scan FE planning, Q1 early termination); rerun full TPC-DS at the default setting, TPC-H and a RELEASE build (diagnosis)
M2 · Advanced read
The five steps are independent of each other.
- M2.1 Time travel:
FOR VERSION AS OFandFOR TIME AS OF, resolving times with in-commit timestamps; a$historysystem table. Acceptance: any historical version matches Spark. Numeric version reads are implemented in draft #70536, stacked on [feature](external) Support reading Delta Lake tables natively with Delta Kernel #69904. Filesystem, dedicated HMS and the HMS gateway matched Spark for historical schemas, ordered rows and counts across deletion vectors and schema changes. Timestamp reads and the history table remain open. - M2.2 Unity Catalog: open-source and Databricks Unity Catalog (token auth, later OAuth machine-to-machine); temporary storage credentials; reading catalog-managed tables. Acceptance: external and managed tables match Spark. The catalog URI is user-supplied, so the PR must call out this outbound HTTP surface and the privilege needed to reach it.
- M2.3 AWS Glue: Glue as a metadata source. Acceptance: Delta tables in Glue are queryable.
- M2.4 New types and change reads: variant (including shredding); Change Data Feed through
table_changes. Acceptance: results match Spark. - M2.5 Optimizer and materialized views: column statistics from the Delta log for the CBO; incremental partition refresh for async materialized views. Acceptance: sensible join orders; materialized views refresh only changed partitions.
M3 · INSERT
M3.1 builds the write path and comes first; the other steps build on it.
- M3.1 Append:
INSERT INTOfor partitioned and unpartitioned tables; write file statistics; retry on concurrent commits; write checkpoints periodically; reject unsupported writer features. Acceptance: Spark reads the data back and keeps writing; concurrent appends with Spark neither lose nor duplicate rows. - M3.2 Overwrite:
INSERT OVERWRITEfor whole tables and dynamic partitions. Acceptance: Spark sees the same data as Doris. - M3.3 DDL:
CREATE TABLE(with partitions and table properties),DROP TABLE, CTAS. Acceptance: Spark can read and write tables Doris creates. - M3.4 Writer features: appends to tables with
appendOnly,invariants, row tracking, in-commit timestamps, domain metadata, liquid clustering and column mapping. Acceptance: all of these tables accept writes and stay readable and writable by Spark. - M3.5 Unity Catalog managed tables: commit through Unity Catalog for catalog-managed tables. Acceptance: managed tables in Databricks and open-source Unity Catalog accept Doris writes.
M4 · DELETE
M4.1 comes first; M4.2 and M4.3 can then proceed in parallel.
- M4.1 Whole-file delete: when the condition covers whole files or partitions, change only metadata. Acceptance: matches Spark, including tables with Change Data Feed.
- M4.2 Rewrite delete: for tables without deletion vectors, rewrite the files that contain deleted rows. Acceptance: Spark sees the same result.
- M4.3 Deletion-vector delete: write deletion vectors, merging with existing ones. Acceptance: Spark reads Doris deletion vectors; concurrent deletes of the same file with Spark report a conflict and lose nothing.
M5 · UPDATE / MERGE
In order; M5.2 reuses M5.1's "delete old rows and write new rows".
- M5.1 UPDATE: delete the old rows (deletion vectors or rewrite) and write the updated rows. Acceptance: same result as Spark.
- M5.2 MERGE INTO:
WHEN MATCHEDupdate or delete,WHEN NOT MATCHEDinsert; error when one target row matches several source rows. Acceptance: same result as Spark. - M5.3 MERGE extensions:
WHEN NOT MATCHED BY SOURCE; schema evolution during MERGE. Acceptance: same semantics as Spark.
M6 · Maintenance and DDL
The four steps are independent of each other.
- M6.1 ALTER TABLE: add columns, rename columns (requires column mapping), change comments and table properties. Acceptance: Spark reads and writes normally afterwards.
- M6.2 OPTIMIZE: compact small files; Z-order; re-cluster liquid clustering tables. Acceptance: fewer files, unchanged results.
- M6.3 VACUUM: remove expired data and deletion-vector files, honoring retention and
vacuumProtocolCheck. Acceptance: never deletes a file referenced by a retained version. - M6.4 RESTORE: restore a table to a version or timestamp. Acceptance: results match that version.
Not before M5
- Row-level changes on tables with Change Data Feed enabled.
- Writes to tables that define CHECK constraints, generated columns, identity columns or column defaults.
- Writes to UniForm (
icebergCompat) tables. OPTIMIZEandVACUUMon Unity Catalog managed tables (Databricks does not allow external clients to run them).
Out of scope: streaming reads and Delta Sharing.
Risks and maintenance
- Concurrent writes with Spark or Databricks: Delta Kernel fully handles conflicts only for concurrent appends; DELETE and UPDATE need dedicated tests.
- Fast protocol evolution: Delta ships roughly every two months; new features such as geometry, collations and nanosecond timestamps must fail clearly until supported.
- Very large tables: million-file tables with frequent commits make metadata loading part of query latency.
- Kernel upgrades: when moving past Delta Kernel 4.4.1, update
DeltaKernelCompatand recheck the behaviors pinned to 4.4.1 (old DATE statistics after widening totimestamp_ntz, legacy-protocol feature filtering, snapshot memory estimates).
Confirmed decisions: development happens on master; DELETE and UPDATE are feasible with Delta Kernel 4.4 (the official Flink connector commits deletion vectors this way); open-source and Databricks Unity Catalog share one API; read compatibility is tested with Delta's golden tables.
History
Earlier status reports, CI investigations and benchmark details for M1 are in the comments, including the CI integration, the full TPC-DS benchmark, the checkpoint-read diagnosis and the scan-planning reuse and streaming assessment.
Activity
xylaaaaa commentedon Oct 30, 2026
Development update — 2026-10-30
The implementation worktree and local M1 task checklist are ready. PRE-01 (dependency resolution and real-snapshot loading with Kernel 4.4.1) is in progress; no implementation task has passed its acceptance checks yet.
Confirmed implementation details:
Progress policy: update the local checklist and this issue after each task group or implementation commit, new validation result, or blocker change, and before ending a development session. Only mark tasks complete after their acceptance checks pass; do not post unchanged status repeatedly.
xylaaaaa commentedon Oct 30, 2026
Draft implementation PR opened: #69904 (
xylaaaaa:codex/delta-m1-reader→master).The PR contains 19 commits and is intentionally still a draft; the local tasklist
records 60/64 completed items.
Implemented: Kernel 4.4.1 FE connector, native FileScannerV2 reader, HMS/filesystem
catalogs and Hive routing, mapped/evolved schemas, portable deletion vectors,
checkpoint sidecars, pruning/count, fixed-TTL snapshot caching, streaming planning
and EXPLAIN/profile diagnostics. Thrift field 16 was rechecked against current
master and branch-3.1/4.0/4.1/4.2 before opening the PR.
Verified locally: full FE/ASAN BE builds; 165 BE, 75 Delta connector, 2 metastore,
25 FE core and 274 Hive-related unit tests; eight p0 suites twice on the optional
Spark/HMS/MinIO environment; million-file planning in a 512 MiB JVM; MinIO and
ordinary HDFS storage regression. Test data is retained for reuse, and restarts
preserved all 1,227 formal MinIO objects by path, size and SHA-256.
Performance measurement: all 125 TPC-H/TPC-DS queries completed cross-reader
comparison and three paired repetitions across official regression runs. Q7/Q85
use the same fixed join order on both readers after default Hive memory failures;
Q30 adapts the generator's column, and floating values use the framework comparator.
These controls and the original failures are recorded in the PR. On local ASAN
-O0 SF1, summed per-query medians are 28.4% lower for Delta on TPC-H and 26.1%
higher on TPC-DS. Nine profile pairs show join/filter/parallelism differences;
performance parity is not yet accepted.
Remaining:
with current master and renewed build/test validation. A merge-tree check found
conflicts in file_scanner_v2.cpp, PluginDrivenScanNode.java and PlanNodes.thrift;
current master is 371 commits beyond the original base and includes connector
SPI/scan-lifecycle changes. Existing results apply to head ef4b7b2.
The tracking issue stays open while these items are completed.
[fix](fe) Read connector statistics at the query snapshot
[feature](fe) Cache Delta Lake table statistics by snapshot version
xylaaaaa commentedon Nov 2, 2026
Basic statistics follow-up — 2026-11-02
Implemented and pushed to draft PR #69904:
Cache tests cover concurrent single-flight, user/version/table identity isolation, invalidation/close, late completion, retry, missing/invalid stats and background authentication/classloader handling. Eight concurrent scans share one real cached checkpoint/DV snapshot; early-closing iterators do not affect the other scans.
Validation on the final source:
Connector README, local design/tasklist and paired EN/ZH website patch are updated. The checklist is 63/67; the issue body now reflects verified work and remaining gates.
The previous TPC-DS +26.1% measurement predates this change. A full post-statistics benchmark has not been rerun, and the entire gap has not been attributed to statistics. Column NDV/histograms remain later work; performance acceptance, cloud/Kerberos environments, website PR/build and master integration/final acceptance remain open.
[test](regression) Measure Delta benchmarks with prepared table stati…
xylaaaaa commentedon Nov 2, 2026
TPC-DS rerun after basic statistics — 2026-11-02
All 103 query files passed in one official regression suite. Delta's summed-median gap to Hive narrows from 26.1% to 16.5%; performance acceptance remains open.
The gap shrinks by 9.6 percentage points. Delta's total falls 10.46%, while Hive's falls 3.07%. These are paired local ASAN/SF1 observations, not RELEASE/TPC-certified results or an isolated attribution of every change to statistics.
The rerun reuses exactly the retained Parquet objects, the same 103 SQL files, four pipeline tasks, disabled SQL/file caches, 8-GiB/query, 120-second timeout and UTC. Q85 keeps the existing common fixed join order; Q30 keeps the same generator-column adaptation. All 24 Delta EXPLAIN cardinalities matched the independent converter manifest before timing; Hive's existing estimates were recorded without changing them. All 824 executions were compared through the existing typed result comparator. TPC-H was not rerun.
Observed improvements and remaining gaps:
Benchmark warmup and public documentation are committed in f0a229f7b9 and pushed to PR #69904. Full raw samples, before/after CSV/JSON, complete profiles and a reproduction report are preserved locally under
docs/delta-m1-preflight/benchmark/post-statistics-2026-11-02/. The tasklist and design/acceptance records are updated; 63/67 remains the completed count and END-03 remains in progress.All 1,227 MinIO objects match their original paths, sizes and SHA-256 before and after the run. FE/BE, the temporary HMS/PostgreSQL/MinIO services and Docker Desktop are stopped again; all 11 container objects, data and build artifacts are retained.
xylaaaaa commentedon Nov 2, 2026
CI investigation for PR #69904, head
f0a229f7b9:file_scan_operator.cpp,file_scanner_v2.cpp,PluginDrivenScanNode.java, andPlanNodes.thriftwhen merging with master5af2df3231. No Cloud-specific test failure was reported.Local integration checks: full FE/Checkstyle and ASAN BE builds passed. Of 859 selected FE tests, 857 passed and two opt-in cases were skipped; the final real SplitAssignment fixture was also rerun (39/39 overlapping lifecycle/producer tests passed). The ASAN BE runner passed all 239 selected tests, including 3 scanner-dispatch/format/missing-file cases, 14 Delta reader and 3 deletion-vector cases. C++ clang-format 16, header hygiene and changed-line clang-tidy passed.
All four retriggered CI jobs passed the old merge point on this head. As of 2026-11-02 14:53 +0800: COMPILE running, FE UT running, Cloud UT passed, BE UT running. The default Linux FE reactor completed the Delta connector (85 passed, 2 opt-in skips, 0 failures/errors). FE also recorded a pre-existing muted non-Delta test,
statistics.cache.CacheTest.testEvict(line 345, mute in place since 2026-09-07). The same assertion failed in five other PR runs earlier that day. It is not counted as passing or as a new blocking failure. Cloud UT finished SUCCESS (856 passed, 27 skipped, zero failures).The local tasklist is updated. Historical SQL/TPC-DS results are unchanged; overall M1 acceptance remains open.
[fix](deltalake) Integrate the Delta reader with current scan APIs
[test](regression) Trim duplicate Delta scanner compatibility cases
xylaaaaa commentedon Nov 3, 2026
Compared PR head
338108051bwith a local build that reverses only the shared statistics-consumption change. Delta's versioned asynchronous row count, DV subtraction and same-version CRC fallback were present in both builds.With the stock FE table-level row-count cache, the filesystem, HMS and Hive gateway catalogs all showed the same results on retained data. For the 3-row basic table and the 35-row checkpoint table with deletion vectors,
EXPLAINestimated 74 and 1, respectively, both initially and after 65 seconds. SQLCOUNT(*)returned 3 and 35 throughout. The effective default refresh interval was 10 minutes. In a separate run with a local-only 1-minute refresh interval, all three catalogs reported the precise estimates by 72 seconds. The FE refreshes on access asynchronously.The existing shared change controls when ordinary CBO plans see completed statistics. Restoring master's policy for every connector would also restore that delay for Delta and would not meet the current 60-second statistics-readiness test. The comparison build and all 81 focused FE tests passed. The approved M1 policy uses the standard FE refresh interval.
The row-count change in
3d4899eb94restores master’s FE table-level row-count cache for ordinary reads and removes the extra Hive snapshot-statistics forwarding. Delta keeps versioned logical counts and same-version CRC fallback. An access after the FE refresh interval (10 minutes by default) triggers an asynchronous update. The full FE build and 102 targeted unit tests passed. The statistics p0 passed for filesystem, HMS and Hive. Under the default refresh interval all three routes converged to 3 and 35 rows for the two reference tables. All 24 TPC-DS manifest counts matched and Q1 passed on both readers in the focused p2 run. Full CI requested.[fix](fe) Restore standard row-count caching for Delta catalog reads
[fix](deltalake) Adapt the Delta reader to current master
[refactor](be) Share one portable deletion vector magic check
[refactor](be) Drop the redundant Delta Parquet check from shared sca…
45 remaining items
[fix](fe) Keep cast predicates out of Delta file pruning
[test](regression) Cover Delta and Iceberg CAST isolation in one HMS …
[test](deltalake) Guard CAST rejection without pinning Iceberg policy
xylaaaaa commentedon Nov 6, 2026
Review follow-up in
661bcbda52: strengthen the Delta unit guard and stop pinning Iceberg's current CAST policy in the mixed HMS regression. This changes tests only.ConnectorPushdownOpsopt-in, and asserts that CAST pushdown is rejected. Removing the override or the interface opt-in will fail the module test without requiring the Docker regression environment.icebergPredicatePushdown=and Icebergdata_files: 1assertions. The earlier isolation run documented current behavior, but that behavior should not be a permanent contract that blocks a future Iceberg correctness fix.The known Iceberg lossy-CAST issue remains in the local TODO for a separate fix. Test data is retained and the local services have been stopped again.
xylaaaaa commentedon Nov 6, 2026
Progress update — Delta-only small-table scheduling (2026-11-06)
Pushed on user approval as
fb4e695560tocodex/delta-m1-reader/ PR #69904. The original local commitc571356e07was rebased onto the remote branch's existing master merge3f893caae7; all nine Delta changed files are byte-identical to the tested version, and the push was a fast-forward. After rebasing, the official FE UT runner passed all 95 selected tests, including million-file scale and planning diagnostics, with zero failures/errors/skips. The latency, full-build and SQL regression results below remain measurements of the original pre-rebase build.The change is confined to four production files in the Delta plugin, plus its tests and README. With no current CRC, scheduling reuses a complete count from the statement or the same user/table identity/version statistics-cache entry. Existing statistics collection counts files; exhaustive normal scans can also publish their already-observed total without starting a statistics task. This is necessary because FE background row-count sessions and SQL sessions can have different identities; user isolation is preserved. A late report only updates its captured entry and cannot republish an invalidated entry. There are no shared FE/BE, SPI, or wire-format changes.
The scheduling gate does not enumerate files, submit work, or wait for statistics. A known small count selects synchronous planning; unknown and large counts keep streaming. Cold snapshots without CRC may still stream.
Retained-data comparison, using the same local macOS ARM64 ASAN BE: a two-file Delta table, one selected file, one returned decimal value (
123.45), SQL cache disabled, five warmup pairs and sixteen alternating measured pairs per entrance. Profiles were collected separately from timing.All three automatic paths went from 144 scanners to one, with the same one file/split/row read and correct result. The final automatic and forced-sync medians differ by 0.4–0.9 ms. These are warm, small-table measurements; the controls also vary across runs, and this is not a new TPC-DS performance result.
Large-table guard: the retained million-file checkpoint was tested before/after in separate 512 MiB JVMs. All three runs enumerated exactly 1,000,000 ranges; enumeration medians were 1493.563 / 1451.023 ms. Both completed background statistics and a complete count reused by subsequent queries still selected streaming. Sampled scan heap peaks were 133,327,520 / 238,172,480 bytes: the bounded-heap tests passed, but this does not establish unchanged memory use. The fixture contains planning metadata, not the corresponding million data files, so this does not measure full data IO throughput.
Validation:
./build.sh --fe -j4passed, including Checkstyle; the ASAN BE binary was unchanged..outwas edited or regenerated.delta_bench; it passed after selecting the retained fixture's original endpoint, without changing the test/data.README, local design notes and tasklist are updated. Raw samples, profiles, test logs and the checkpoint-by-checkpoint self-review are retained in the worktree under
docs/delta-m1-preflight/small-table-scheduling-20261106/. Test services are stopped and data is preserved. Full TPC-DS parity remains open.[opt](deltalake) Reuse complete file counts for small-table scheduling
xylaaaaa commentedon Nov 6, 2026
Full TPC-DS rerun after small-table scheduling — 2026-11-06
Tested fb4e695560, including a fresh complete FE/BE build on local macOS ARM64, ASAN. 103/103 query files passed, with 824 executions and no result mismatches or failed queries. The basic p0 suite also passed all 36 result blocks.
The same retained SF1 data was reused: 24 tables, 19,557,335 rows, 96 objects. Object paths, sizes and SHA-256 values matched the original export before and after testing. Hive reads the same Parquet objects as Delta; all 24 Delta tables lack CRC files. The 103 SQL hashes match the November 2 run.
The recorded overall gap is 11.79 percentage points smaller. This is not an isolated before/after measurement of one patch: master changed and the earlier shared FE row-count consumption was reverted in between. Hive's total increased 6.58% and Delta's decreased 4.21%. The current same-build Delta/Hive comparison is the primary result.
Controls remain: SQL/file caches disabled, four pipeline tasks, 8 GiB/query, 120-second timeout, UTC, default external batch mode and threshold 1024. Before timing, all 24 Delta cardinalities converged to the independent manifest under the default 10-minute FE refresh interval; Hive estimates were recorded unchanged. Q85 retains the same fixed join order on both sides, and Q30 retains the generator column-name adaptation. First executions are excluded; profiles were disabled during timing. Integer/decimal values compare exactly and floating results use the existing 1e-8 tolerance.
Remaining differences: 32 queries are still over 10% slower; two are over twice as slow. The geometric mean of per-query ratios is 1.124, so the +4.69% aggregate does not mean uniform per-query parity. Largest absolute penalties in the full run:
Ten complete plans/profiles were captured after timing for Q1/Q88/Q80/Q22/Q4, with results checked again. They do not replace any full-run sample:
Limits and delivery: This is local ASAN / SF1 with SF100 query parameters, not RELEASE, certified TPC results, cold-cache or concurrent throughput. TPC-H and million-file data IO were not rerun. END-03 and overall M1 acceptance remain open. The local tasklist and full report are updated under
docs/delta-m1-preflight/benchmark/post-small-table-scheduling-2026-11-06/, including raw samples, comparisons, build logs, artifact hashes, profiles and data verification. The local PCH/float.h build conflict was handled withENABLE_PCH=OFF; no product/test/golden-file changes were needed. FE/BE, the three started dependency containers and Docker Desktop are stopped; all data and artifacts are retained.xylaaaaa commentedon Nov 6, 2026
November 6 follow-up: repeated checkpoint reads during FE planning
Read-only diagnosis at
fb4e695560bafc1862ac398fe22bfe6338c7f02e, using the same local ASAN build and retained SF1 data as the full TPC-DS measurement. No production code, regression cases or expected outputs changed. The full-suite result remains +4.69% overall, not a new number derived from the following targeted probes.After all 24 Delta cardinalities matched the fixture manifest, ten alternating pairs of uninstrumented
EXPLAIN VERBOSEproduced:These measure planning plus plan-text transfer, without executing BE data scans. Separate query profiles also place the extra FE work in Translate: Hive/Delta 6/286 ms for Q88 and 3/156 ms for Q80, with no Delta planning GC in these samples.
Confirmed path: the translator finalizes scan nodes sequentially; DeltaLakeScanPlanProvider.planScan opens a fresh Kernel scan-file iterator, which rereads the checkpoint through Hadoop S3A. The cached snapshot does not contain the active AddFile list or checkpoint bytes.
Profile interpretation correction: Get Splits start/finish overwrite one timestamp pair. The summary contains the last scan node's time, not the sum. Q88's 7 ms Get Splits therefore does not contradict its 286 ms Translate; split enumeration is included in Translate. No shared FE timer change was made.
Separate Q1 issue: both readers still return the correct empty result. Delta prunes store to zero files but still reports RowsRead 100,000 / 55,820 / 366 for customer / store_returns / date_dim. Hive's RowsRead counters are all zero, although customer RawRowsRead is 32. Different join/build and CTE arrangements allow Hive to terminate earlier. A diagnostic sample shows wait/fetch 14/432 ms for Hive/Delta; identifying the exact optimizer or execution rule needs a separate controlled-plan experiment. These single execution samples do not replace paired full-suite timings.
Filesystem table-location HEAD/LIST is another, smaller foreground cost. A longer JFR also observed repeated location checks on the independent Statistics Job Appender thread; its time was not charged to the query.
Next candidate: bounded reuse of small committed checkpoint content inside the Delta plugin, with identity/version/object correctness and large-table streaming preserved. This is a proposal, not an implemented cache or a promise to eliminate the entire 4.69% gap. END-03 remains open (63/67 tasks complete).
Validation: official basic regression 36 result blocks passed; Q1/Q88/Q80 actual results match across readers; all 96 retained TPC-DS objects still match path/size/SHA-256. FE/BE and temporary Docker services are stopped, data retained. Local tasklist and the diagnosis report, full plans/profiles, raw JFR and sanitized request traces are preserved under
docs/delta-m1-preflight/benchmark/planning-diagnosis-2026-11-06/. Initial JFR was taken before row-count refresh; its first trace-parser failure is explicitly retained as unavailable, with subsequent successful traces and warm-statistics verification kept separately.xylaaaaa commentedon Nov 6, 2026
Follow-up assessment: prefer statement-local scan-plan reuse first
The requested follow-up was read-only: no production/test code changes, no new build or cluster run, and no new performance claims. This refines the recommendation in the checkpoint diagnosis.
Recommended first step: wire Delta into the existing statement-local reuse of equivalent synchronous scan plans. The current branch already has this in Iceberg, Hive, Paimon and Hudi.
ConnectorStatementScope.computeIfAbsentForPlanningandenable_external_scan_task_reusealready exist; Delta currently re-enumerates every scan. This needs no shared FE/SPI/BE change and can avoid both repeated IO and Kernel parsing, rather than only network reads.Offline grouping of the saved plans by table and predicate, removing slot IDs only, gives these candidate groups:
These are not observed cache hits: the full request key still needs tests. Q88's store_sales/store/household_demographics each repeat eight equivalent filters, while its eight time_dim predicates remain distinct. If other request dimensions match, the measured five-GET-per-enumeration pattern suggests 160→55 checkpoint GETs for Q88 and 90→60 for Q80; this is workload arithmetic, not a latency prediction.
The Delta key must cover table path/UUID/version, typed filter, required partitions and the pruned-to-empty flag, COUNT mode, split size and timezone. Reuse only complete immutable ranges plus per-node metrics; keep cancellation/validation on hits. The existing range descriptor copy behavior is a suitable base. Preserve a per-statement retention budget and release at planning end. Streaming/unknown-size scans must keep their current path without materialization. The scope loader must only create a private memo: scan planning re-enters the statement scope for counts, so doing IO directly inside that loader would introduce map re-entry.
Checkpoint byte caching remains a second candidate. The retained 24 checkpoint objects total only 1,002,655 bytes; Q88's four unique objects total 165,159 bytes, versus 3,331,960 transferred per EXPLAIN. A scoped Engine/DefaultParquetHandler/FileIO wrapper is possible through public Kernel 4.4.1 APIs, but the catalog-shared
newInputFile(path, size)alone lacks user/table identity. Snapshot discovery should stay outside that cache until the table UUID is established. Also, the pinned Delta protocol allows classic checkpoints to be overwritten by a competing writer with equivalent content; byte-layout immutability must not be assumed. Whole-object consistency, refresh fencing, concurrent-load memory and interruptible waits require explicit design. Framework retained-weight limits do not bound byte arrays being downloaded or still held by readers.Acceptance for the first candidate: identity/filter/partition/count/size/timezone and failure/lifecycle tests; filesystem/dedicated HMS/HMS gateway and mixed-format results; same-build paired reuse-on/off probes with actual hit/GET/profile evidence; the retained million-file streaming fixture under 512 MiB; then full TPC-DS. No claim that this fixes Q22, Q1 early termination, or the full +4.69% gap. END-03 remains open. Local tasklist and
optimization-assessment.mdcontain the scoped design and review checklist; services remain stopped and retained fixtures untouched.xylaaaaa commentedon Nov 6, 2026
2026-11-07: Delta-only statement-local scan planning reuse
Implemented and locally verified on top of
fb4e695560. Commit8b5d68c46b, pushed to PR #69904 on 2026-11-09. Production changes are confined to four Java files in the Delta plugin; shared FE/SPI/BE, Hive, Iceberg and Paimon are unchanged.The existing
enable_external_scan_task_reuseflag and planning scope now share completed synchronous ranges and per-node metrics. Keys include catalog/query/user, table coordinates/UUID/version, typed predicate, partition selection, COUNT, split size and time zone. Identical concurrent loads coalesce; waiting callers can cancel independently; failed/cancelled loads are retried. The memo retains at most 128 entries, 16,384 ranges and 16 MiB of estimated range payload. This is a retention budget, not a whole-query/in-flight heap bound. Streaming never materializes a list to enter this memo.Validation
build.sh --fe -j4and Checkstyle passed. BE is unchanged and uses the preceding ASAN build at the same base revision.Same-build targeted A/B
Scope correction, 2026-11-11: The FE factory default of
num_files_in_batch_modeis 1 (source). The saved localcluster-ready.jsonrecords an effective value of 1024 for this benchmark. Delta uses synchronous planning only when the known file count is strictly below that threshold; ordinary nonempty scans stream under the factory default, so this synchronous memo does not apply to that ordinary path. The following timings, including Q88 -9.6%, are evidence for threshold 1024, not factory-default performance. At the time of this correction default-1 validation was pending; the targeted results are now appended below, while full-workload default-1 validation remains pending. No product policy or shared default was changed in this correction.Retained SF1 data; macOS ARM64 ASAN, four pipeline tasks, 8 GiB query limit, SQL/file cache disabled, UTC. All 24 row counts were ready through the existing default refresh interval before timing. Each setting had three warmups, ten alternating EXPLAIN pairs and seven execution pairs. All 88 Delta executions, including warmups and post-measurement profiles, matched four Hive reference executions.
Q22 has no reuse hits or IO savings; its approximately 5 ms EXPLAIN difference is not attributed to this optimization. Q88 was faster in all seven execution pairs; Q80 in six, Q4 in five, Q22 in three. Q4's post-measurement profile still spends about 8.8 seconds waiting/fetching execution results, so halving planning time barely changes end-to-end time.
Separate MinIO traces, collected after timing, give identical counts in all three samples per setting, all HTTP 206. Checkpoint body bytes per EXPLAIN: Q88 3,331,960 → 969,558; Q80 2,329,190 → 1,530,260; Q4 2,360,120 → 793,405; Q22 stays at 338,203.
Large-table streaming and retention
The retained million-file planning fixture was tested in independent 512 MiB JVMs, three complete enumerations each. With FE/BE/Docker stopped on both sides, enumeration median was 1493.940 → 1484.218 ms; first split 23.286 → 20.804 ms. Sampled heap peak was 112.921 → 125.744 MB, so memory is not claimed unchanged. The bounded-heap test passed and no full range list was materialized. An initial after-run with idle services present measured 1552.093 ms; that sample is retained separately rather than discarded.
This fixture validates metadata planning, not real million-file data IO. This is a targeted ASAN/SF1 A/B, not a full TPC-DS/Hive or RELEASE rerun; the November 6 +4.69% remains the latest full-workload result. END-03 and M1 acceptance remain open.
README, local design/tasklist and the per-checkpoint self-review are updated. Evidence is retained in
docs/delta-m1-preflight/scan-plan-reuse-20261107/(report, samples, plans/profiles, sanitized storage traces, JFR, unit-test XML and generated-output audit). All 96 TPC-DS objects match their original path/size/SHA-256. Temporary services are stopped, data/volumes/builds retained, and local configuration changes are not committed.Push and CI request (2026-11-09)
The commit is now the PR head. run buildall was requested. The trigger workflow failed while querying TeamCity queue/running-build endpoints, before submitting compiler or unit-test builds. One retry is in progress. New Linux CI results remain pending; the previous head's passing checks do not cover this commit. The validation figures above remain the local November 7 measurements.
Default-1 streaming measurements and assessment (2026-11-11)
The PR keeps the factory
num_files_in_batch_mode=1; no product, shared FE/SPI/BE, test or expected-output code was changed in this assessment. Every measured session explicitly set and read back 1, with batch mode enabled. This is a read-only measurement/design task, not a new implementation.Using the retained SF1 data and the same validated ASAN artifacts, all 24 Delta cardinalities were ready. Three interleaved variants (Hive, Delta reuse off, Delta reuse on) produced matching results. Actual-query medians from five rounds per variant:
All Delta profile nodes streamed and had zero synchronous planning-reuse hits; on/off differences were within approximately 1%. Separate actual-query traces still show 160 checkpoint GETs for Q88, 90 for Q80 and 15 for Q22 in each of three samples. Q4 recorded 90/100/90, with the extra 10 reads disclosed rather than discarded. JFR confirms planning waits for background split generation. Both readers use 64 scanners per node under this configuration; this is not a Delta-only scanner-count difference.
The Delta-only candidate is bounded replay of completed small streaming results. Publish only after successful EOF/close; a pending/missing entry continues independent streaming without waiting. Ordinary streaming does not forward
isBlocked, so it cannot safely use that hook for pending followers. The design also needs actual streaming COUNT mode in the key, a statement lifetime that outlives planning, and budgets covering in-flight capture. This candidate is not implemented or benchmarked; large-table non-regression remains future implementation acceptance.Official basic regression passed 36 blocks; all measured/diagnostic query results matched Hive. The 96 retained objects passed path/size/SHA-256 checks and owned services were stopped. This is four targeted queries, not a new full TPC-DS result. The earlier +4.69% full-workload result and synchronous-reuse gains used the local 1024 threshold. END-03 remains open. Raw local evidence (report, source assessment, self-review, samples, profiles, JFR, sanitized storage traces and cleanup verification):
docs/delta-m1-preflight/streaming-reuse-assessment-20261111/.[opt](deltalake) Reuse identical scans during statement planning
[-][Tracking] Native Delta Lake read support[/-][+][Tracking] Native Delta Lake support[/+][test](deltalake) Check actual splits in Delta reader regressions
[feature](deltalake) Read Delta tables at a numeric version
xylaaaaa commentedon Nov 12, 2026
The Delta partition-pruning and Parquet deletion-vector regressions now check the actual split counts from the completed scan profiles. The fix is in #69904, commit 8ba8e67f29.
num_files_in_batch_mode=1. They also passed on a connection using synchronous scheduling. For both filesystem and HMS, the basic partition filter selected one file and generated one split. The deletion-vector boundary query selected one file and generated five splits, with the existing row, payload and pruning checks passing.