Storages: introduce trim min-max index for DATE/DATETIME/TIMESTAMP rough set filtering

[JaySon-Huang]

backport #10982 to master branch

What problem does this PR solve?

Issue Number: close #10989

Problem Summary:

Applications sometimes use a far-future temporal value, such as 2100-01-01 00:00:00, as a sentinel for an unsettled record while normal DATE, DATETIME, or TIMESTAMP values remain concentrated in a recent range.

Although such sentinels are sparse, they pollute the ordinary pack-level min-max index. With 8,192 rows per pack and an independently distributed sentinel probability of 1/10000, about 55.9% of packs contain at least one sentinel. Their ordinary maximum is extended to 2100, so narrow recent-time predicates cannot exclude those packs and must read and filter more data.

The ordinary min-max index cannot simply ignore a particular application value because queries may intentionally access it. TiFlash needs an optional temporal index that improves pruning for eligible predicates without changing SQL semantics or weakening the ordinary fallback path.

What is changed and how it works?

Storages: introduce trim min-max index for DATE/DATETIME/TIMESTAMP rough set filtering

This PR adds an optional pack-level trim min-max index for user DATE, DATETIME, and TIMESTAMP columns in DeltaMerge DMFile V3 / MetaV2.

The effective interval for format V1 is the half-open range [1900-01-01 00:00:00, 2099-12-01 00:00:00). The ordinary min-max index remains unchanged and continues to cover every non-NULL, non-deleted value. The trim index covers only values inside the effective interval and records directional per-pack marks when values are trimmed below or above it.

Write path and disk format

• Build the ordinary and trim min-max indexes in the same pack scan when dt_enable_trim_minmax=true.
• Keep the ordinary .idx payload and its byte format unchanged.
• Persist trim data in a separate deterministic <column-stream>.trim.idx merged subfile only when the DMFile contains trimmed outliers.
• Store format version, the type-packed lower/upper bounds, and pack count in the independent optional protobuf field ColumnStat.trim_minmax_index = 105.
• Reuse the existing min-max pack-mark byte: bit 0 is has_null, bit 1 is has_trimmed_low, and bit 2 is has_trimmed_high.
• Treat logical metadata absence or mismatch as a soft fallback, while preserving the existing hard-failure policy for impossible merged-file geometry, checksum failure, or corrupt/reserved pack-mark bits.

Query-domain analysis and read path

• Normalize supported temporal predicates into a query domain, including eligible Equal, IN, bounded ranges, and one-sided ranges.
• Merge same-column top-level AND range bounds without sharing eligibility across OR or NOT branches.
• Select trim min-max per predicate and per DMFile using the bounds persisted by that DMFile. A different predicate on the same column can still load and use the ordinary min-max index.
• Convert TIMESTAMP constants through the request time zone into UTC-packed values before eligibility checks; compare DATETIME and DATE using calendar-value semantics.
• Conservatively correct trim rough-check results using the directional pack marks, including None -> Some and All -> Some when trimmed values can invalidate the uncorrected result.
• Fall back to ordinary min-max when the switch is disabled, metadata or the trim subfile is absent, the version or pack count is unsupported, the predicate is outside the persisted interval, or the expression shape is not supported.

Configuration and observability

• Add the unified dt_enable_trim_minmax setting, defaulting to false. It controls both generation and query use of trim min-max indexes.
• Add Prometheus counters for trim selection/fallback reasons, trim rough-check All/Some/None/AllNull results, and conservative correction counts.
• Do not add SQL syntax or DDL. The feature is transparent to SQL results and is enabled only through the TiFlash profile setting.

Compatibility

• Old readers ignore protobuf field 105 and the additional merged subfile, then continue using the unchanged ordinary min-max index.
• New and old DMFiles can coexist; each predicate independently selects trim or falls back to ordinary min-max for each DMFile.
• An old-version metadata rewrite may drop field 105. A new reader then ignores the orphan trim subfile and safely falls back to ordinary min-max.
• Downgrading to a binary that predates dt_enable_trim_minmax requires removing that unknown setting from tiflash.toml before startup.

See the design document for the correctness proof, format details, and rollout constraints.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Unit coverage includes:

• trim pack marks, protobuf round trips, unknown-field compatibility, metadata validation/fallback reasons, file naming, and cache-key separation;
• ordinary/trim index construction in one pass, NULL/delete/empty packs, invalid masks, persistence only when outliers exist, and disabled-write behavior;
• DATE, fractional DATETIME, and TIMESTAMP packed bounds and time-zone conversion;
• query-domain eligibility and normalization, inclusive/exclusive and empty ranges, per-column parse failures, unsupported/NULL-sensitive shapes, mixed AND/OR trees, and per-predicate eligibility;
• the complete RSResult correction matrix, trim/ordinary pack-filter selection, missing attribute types, and DMFile MetaV2 persistence/drop behavior.

The PR unit-test and integration-test CI jobs pass.

Manual validation used the same generated 500-million-row data set in two tables with different import/DMFile layouts: test.bc_bet_records_500m and test.bc_bet_records_500m_stream.

1. Correctness: Ran Q01-Q33 through TiKV, TiFlash with trim disabled, and TiFlash with trim enabled. Q01-Q31 compare exact row counts and checksums, Q32 compares grouped rows, and Q33 compares a deterministic complete 100-row TopN result. All 133 normalized result rows matched exactly. The enabled run exercised trim selection, outside-range fallback, All/Some/None, none_to_some, and all_to_some; the disabled run exercised only fallback_disabled. See the 500M correctness report.
2. Performance: For each table, ran one warm-up and five measured P01-P06 suites per setting, then disabled trim again for three reverse-order confirmation suites. The complete-suite median improved by 24.1% (2,125.5 ms -> 1,613.2 ms) on the regular-import table and by 24.9% (2,066.3 ms -> 1,552.7 ms) on the stream-import table. P04 physical scanned rows decreased by 52.4% and 52.6%, respectively. The P05 outside-range control had unchanged scanned rows and no meaningful benefit, as expected. See the 500M benchmark report.
3. Downgrade read compatibility: After newer code generated and read trim indexes, downgraded the TiFlash node to 5d56de051244ac7c893de5ac029f0013098f95dc, removed the unknown dt_enable_trim_minmax setting, and forced both tables through TiFlash MPP. Both 500-million-row invariant scans and complete Q01-Q33 suites passed with zero differences from the TiKV oracle, and no post-start DMFile/index/checksum/corruption error was logged. This validates read compatibility only; it does not cover old-version writes or metadata rewrites. See the downgrade report.

The data generator, distributions, import paths, and SQL suites are documented in the dbgen trim min-max example.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

The feature is disabled by default. When enabled, DMFile writing performs additional min/max comparisons and may persist one optional trim subfile per eligible temporal column with outliers. Eligible reads load the trim index in preference to ordinary min-max; fallback queries retain the ordinary path.

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Introduce trim min-max index to enhance tiflash's table scan filtering performance when DATE/DATETIME/TIMESTAMP defaults to future times

Summary by CodeRabbit

• New Features

  • Added optional trim min-max indexing for DATE, DATETIME, and TIMESTAMP data.
  • Improved rough-set filtering for eligible temporal equality, IN, and range predicates.
  • Added conservative fallback handling to preserve query correctness.
  • Added observability metrics for filtering, index selection, and corrections.
  • Added compatibility support for reading data with or without trim-index metadata.

• Documentation

  • Added design documentation covering trim min-max indexing, compatibility, validation, and rollout considerations.

• Tests

  • Added coverage for index persistence, temporal predicate normalization, fallback behavior, compatibility, and end-to-end filtering.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lidezhu, likidu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• dbms/src/Interpreters/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: bacb59bc-7196-46fb-9df1-93b36d4dbb2d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[JaySon-Huang]

default to false in master branch