ddl, importer: use captured collation mode for background encoding

[joechenrh]

What problem does this PR solve?

Background ADD INDEX and IMPORT INTO encoding can run outside the submitting keyspace. In next-gen deployments, the worker current new-collation default can differ from the table or task setting, so common handle, index key, restored data, and generated expression encoding need a task-level snapshot instead of reading the worker-global value.

What changed?

• Persist a use_new_collate snapshot in DDL reorg metadata and IMPORT INTO task plans.
• Build ADD INDEX txn and ingest indexes, handles, restored data, and decode column maps with the captured value.
• Build IMPORT INTO tables, KV encoders, sampled KV encoders, generated column expressions, and column assignment expressions with the captured value.
• Keep old task metadata compatible by falling back to the current process default when the snapshot field is missing.

Tests

• go test -p=1 ./pkg/ddl/copr

• go test -p=1 ./pkg/table/tables ./pkg/lightning/backend/kv ./pkg/executor/importer ./pkg/planner/core -run '^$'

• Local next-gen mismatch validation where the submitting user keyspace and the worker/system keyspace use different new-collation defaults.

• ADD INDEX coverage:

  • Distributed ADD INDEX backfill with task-captured collation mode.
  • Index key encoding and common handle encoding/decoding.
  • Prefix/restored-data index path.
  • Generated column expression path.
  • Multiple indexes added by one ALTER statement.
  • Max-handle scan and handle rebuild from coprocessor chunks.
  • ADMIN CHECK TABLE after backfill, with DDL job verification that the work ran through DXF.

• IMPORT INTO coverage:

  • File import with task-captured collation mode.
  • Table construction in planner/executor/importer paths.
  • SQL-to-KV encoding, sampled KV encoding, and raw row decoding.
  • Primary handle and secondary index consistency after import.
  • Generated column and column assignment expression paths.
  • Restored-data/extra-payload consistency checks.
  • ADMIN CHECK TABLE and query assertions after import.

• Baseline comparison: the same local validations fail before this change with index/table inconsistency and pass on this branch.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ti-chi-bot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e18b94c5-4a42-4e81-ae9d-1f4855ec432f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-dxf-collation-snapshot-on-refactor

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[D3Hunter]

NewCollate Is actually an cluster(keyspace) attribute, not a table attribute, i think we should get it from current source of truth, i.e. the global variable, not from tbl

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ailinkid, gmhdbjd, zanmato1984 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• lightning/OWNERS
• pkg/ddl/OWNERS
• pkg/dxf/OWNERS
• pkg/executor/importer/OWNERS
• pkg/expression/OWNERS
• pkg/infoschema/OWNERS
• pkg/lightning/OWNERS
• pkg/meta/OWNERS
• pkg/planner/OWNERS
• pkg/table/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment