[Feature] Support incremental view maintenance (IVM) for materialized tables (MTMV)

[yujun777]

Search before asking

• I had searched in the issues and found no similar issues.

Description

This issue tracks the development of Incremental View Maintenance (IVM) for materialized tables (MTMV) in Apache Doris.

Today, refreshing a materialized table recomputes the full result. With IVM, when a base table of an MTMV changes, only the affected rows are derived from the base-table deltas and transactionally applied to the materialized table, so that near-real-time aggregation and join results can be maintained at a fraction of the cost of a full refresh.

IVM is the MTMV incremental-maintenance layer of the end-to-end incremental computation stack being built in Doris:

OLAP table DML
  -> row binlog: durable row-level changes, before images, operation type, LSN/TSO
  -> Table Stream: change-query semantics, per-partition consumption offsets, snapshots and reset
  -> MTMV incremental maintenance (this issue): derive and transactionally apply only the affected result changes

This issue is the focused tracker for the MTMV layer. The overall stack is tracked by #65418, which supersedes the stale umbrella issue #57921.

Scope

• Define supported MTMV query shapes and IVM eligibility/fallback rules.
• Derive delta plans from Table Streams, including snapshot sides for joins.
• Define incremental semantics for projection, filter, join, union, aggregation, and supported expressions.
• Integrate stream creation and lifecycle with MTMV creation, alteration, pause/resume, and drop.
• Execute result changes and stream-offset advancement atomically.
• Support full-refresh bootstrap/reset and documented fallback when incremental maintenance is not valid.
• Add observability for refresh mode, consumed offsets, lag, fallback reason, and failures.
• Add FE and BE unit tests plus end-to-end regression tests for correctness and recovery.

Progress

The implementation is developed on the ivm branch (240+ commits) and is being submitted through PR #62606. The IVM incremental refresh for MTMV is basically complete, including:

PR: #62606 — [ivm] Add mv ivm test for pipeline (implementation PR from the ivm branch, ready for review)

• Supported operators: projection, filter, aggregate (COUNT/SUM/AVG/MIN/MAX, bitmap aggregates, expression arguments, bare GROUP BY), joins (INNER/CROSS JOIN, LEFT OUTER JOIN, FULL OUTER JOIN, nested outer joins), UNION ALL, subquery aliases, and OneRowRelation.
• Refresh: REFRESH MATERIALIZED VIEW ... INCREMENTAL/PARTITIONS, refresh explain, dry-run, and automatic full-refresh fallback when incremental maintenance is not valid (e.g. binlog broken).
• Lifecycle: stream creation/lifecycle integrated with MTMV, chained IVM MTMV support, and ivm_use_full_keys property for MOW materialized views.
• Testing: FE unit tests and end-to-end regression tests under mtmv_p0/ivm.

Related

• [Feature] to support incremental computation in Doris #57921 — [Feature] to support incremental computation in Doris (stale umbrella, superseded by [Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418)
• [Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418 — [Feature] Track incremental computation with row binlog, Table Stream, and MTMV (overall tracking issue)

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct