Data consistency when replicting tpcc workload to mysql sink

[fubinzh]

What did you do?

1. Upstream: TiDB cluster with 1 cdc node; CDC downstream: MySQL
2. Create mysql sink changefeed
3. Run tpcc workload

/go-tpc tpcc --db workload --warehouses 1000 -T 32 --host upstream-tidb.cdc-testbed-tps-7593068-1-644 --port 4000 --parts 1 prepare --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062'
/go-tpc tpcc --db workload --warehouses 1000 -T 32 --host upstream-tidb.cdc-testbed-tps-7593068-1-644 --port 4000 --parts 1 --time 5h0m0s run --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062'

4. Wait sync finished and do data consistency check between upstrem and downstream

What did you expect to see?

Data should be consistent

What did you see instead?

Data inconsistency, and cdc panic seen.

[2024/06/25 01:21:47.816 +00:00] [PANIC] [shared_region_worker.go:286] ["The CommitTs must be greater than the resolvedTs"] [EventType=COMMITTED] [CommitTs=450698541640712196] [resolvedTs=450698544943726600] [regionID=676] [stack="github.com/pingcap/tiflow/cdc/kv.handleEventEntry\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:286\ngithub.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).handleEventEntry\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:237\ngithub.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).processEvent\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:197\ngithub.com/pingcap/tiflow/cdc/kv.(*sharedRegionWorker).run\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_region_worker.go:164\ngithub.com/pingcap/tiflow/cdc/kv.(*SharedClient).Run.func1\n\tgithub.com/pingcap/tiflow/cdc/kv/shared_client.go:346\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/[email protected]/errgroup/errgroup.go:78"]

Versions of the cluster

Release Version: v8.2.0-alpha
Git Commit Hash: c001638
Git Branch: heads/refs/tags/v8.2.0-alpha
UTC Build Time: 2024-06-21 11:36:49
Go Version: go version go1.21.10 linux/amd64
Failpoint Build: false

[fubinzh]

/severity critical

[hicqu]

Duplicated with tikv/tikv#17233.

[flowbehappy]

Remove the affected versions, as we use another issues to track.