Checksum keeps failling with commands out of sync. You can't run this command now

[DevMasterSoonToBe]

Hi Spirit team!

We are trying to run checksum after dropping sentinel table and it tries to repair a lot of incosistencies.

However, after like 10mins of running spirit always fails with below output:

WARN[0425] inspection revealed row does not exist in target for pk: xxxxxxxxxxx
WARN[0425] inspection revealed row does not exist in target for pk: xxxxxxxxxxx
WARN[0425] recopying chunk: Primary_key >= "xx" AND Primary_key < "xx"
INFO[0428] finished periodic flush of binary log: total-duration=26.67µs batch-size=1000
[mysql] 2025/01/10 11:40:02 packets.go:446: busy buffer
[mysql] 2025/01/10 11:40:02 packets.go:427: busy buffer
INFO[0435] closing canal
INFO[0435] syncer is closing...
[mysql] 2025/01/10 11:40:02 packets.go:446: busy buffer
[mysql] 2025/01/10 11:40:02 packets.go:427: busy buffer
[mysql] 2025/01/10 11:40:02 connection.go:150: tls: failed to send closeNotify alert (but connection was closed anyway): write tcp 10.26.14.19:44518->10.26.15.72:3306: write: broken pipe
[mysql] 2025/01/10 11:40:02 packets.go:427: busy buffer
[mysql] 2025/01/10 11:40:02 packets.go:427: busy buffer
[mysql] 2025/01/10 11:40:02 packets.go:427: busy buffer
WARN[0435] releasing metadata lock for xx.xxxxxxxxxxx
INFO[0435] kill last connection id 2692950
INFO[0435] syncer is closed
spirit: error: commands out of sync. You can't run this command now

After doing some google search it says something about go drivers for mysql. We don't see any metrics as issues on the DB itself.

Our command looks like this:
./spirit --host=${DB_HOST} --username=${DB_USERNAME} --password=${DB_PASSWORD} --database=x --table=xxx --alter="PARTITION BY KEY() PARTITIONS 8" --defer-cutover=true --checksum=true --skip-drop-after-cutover=true --threads=16 --target-chunk-time=1s --lock-wait-timeout=300s

[morgo]

Thank you for the bug report. Can you please report the git hash of your version?

There was an issue a while back where changes might not have been synced, causing checksum failures and re-copies. If you are ahead of #332 it should be okay.