refactor(meta): deprecate dispatcher table and compose dispatcher by fragment relation

[wenym1]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

With FragmentRelation table introduced, we don't have to persist the dispatcher information. Therefore, in this PR, we will deprecate the dispatcher table. In all places that used the dispatcher table, we will change to use the FragmentRelation table.

Previously, the dispatcher is used in either resolving the logical connection between fragments, or loading the existing physical dispatchers. For resolving the logical connection between fragments, we can use the FragmentRelation directly. For loading existing physical dispatchers, we can actually recompose the physical dispatchers by combining information in FragmentRelation table and the vnode distribution of upstream and downstream actors.

Checklist

• I have written necessary rustdoc comments.
• I have added necessary unit tests and integration tests.
• I have added test labels as necessary.
• I have added fuzzing tests or opened an issue to track them.
• My PR contains breaking changes.
• My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
• My PR contains critical fixes that are necessary to be merged into the latest release.

Documentation

• My PR needs documentation updates.

Release note

[wenym1]

• refactor(meta): deprecate persisted fragment upstream ids #20548
• refactor(meta): deprecate dispatcher table and compose dispatcher by fragment relation #20541 👈 (View in Graphite)
• refactor(meta): deprecate persisted actor upstreams and get upstreams from dispatcher info #20387
• feat(meta): let stream actors in fragment share same stream node #20310
• refactor: make actor graph builder fragment aware #20385
• refactor(meta): do not store upstream actors in merge node #20222
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[yezizp2012]

Rest LGTM

[yezizp2012]

Can we create a migration to drop the table and also remove this file?

[wenym1]

Shall we keep the dispatcher table for a release? The backfill of fragment_relation depends on the dispatcher table. I'm afraid that if something wrong happened after we drop the dispatcher table and the fragment_relation table gets rollbacked and also dropped in the down, then we will lose the dispatcher information forever and cannot backfill the fragment_relation table.

[yezizp2012]

I'm afraid that if something wrong happened after we drop the dispatcher table and the fragment_relation table gets rollbacked and also dropped in the down

🤔 The backfill of fragment_relation is done in previous migration m20250106_072104_fragment_relation.rs, if it fails actor_dispatcher won't be dropped. Actually the logic of "down" is mostly untrustworthy and irreversible, we will not execute it or rely on it to perform a downgrade. Currently, the only consistency that can be guaranteed in rw is backup and restore. The only issue I can think of is that fragment_relation is not included in the backup, which has been fixed in this PR.

It's fine to keep it for now, as it doesn't have any negative effects and aids in recovering abnormal data and pinpointing issues. Shall we leave a TODO for it?

[wenym1]

Sure. I will create an issue to track it.

[wenym1]

#20643