Skip to content

gh-ost idling on apply error #1730

Description

@krime777

I was encountering problems when, at random times, gh-ost would just idle while backlogging was filling up, lag and heartbeatlag increasing, no new rows would get copied:

2026-06-26 09:05:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m0s(total), 3h6m59s(copy); streamer: dal13-sql21-bin.025288:95132715; Lag: 9236.81s, HeartbeatLag: 9236.81s, State: migrating; ETA: 157h7m35s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m30s(total), 3h7m29s(copy); streamer: dal13-sql21-bin.025288:120527768; Lag: 9266.80s, HeartbeatLag: 9266.81s, State: migrating; ETA: 157h32m48s
2026-06-26 09:05:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h7m30s(total), 3h7m29s(copy); streamer: dal13-sql21-bin.025288:120527768; Lag: 9266.80s, HeartbeatLag: 9266.81s, State: migrating; ETA: 157h32m48s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m0s(total), 3h7m59s(copy); streamer: dal13-sql21-bin.025288:149082002; Lag: 9296.80s, HeartbeatLag: 9296.81s, State: migrating; ETA: 157h58m1s
2026-06-26 09:06:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m0s(total), 3h7m59s(copy); streamer: dal13-sql21-bin.025288:149082002; Lag: 9296.80s, HeartbeatLag: 9296.81s, State: migrating; ETA: 157h58m1s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m30s(total), 3h8m29s(copy); streamer: dal13-sql21-bin.025288:172653166; Lag: 9326.71s, HeartbeatLag: 9326.81s, State: migrating; ETA: 158h23m13s
2026-06-26 09:06:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h8m30s(total), 3h8m29s(copy); streamer: dal13-sql21-bin.025288:172653166; Lag: 9326.71s, HeartbeatLag: 9326.81s, State: migrating; ETA: 158h23m13s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m0s(total), 3h8m59s(copy); streamer: dal13-sql21-bin.025288:194205311; Lag: 9356.80s, HeartbeatLag: 9356.81s, State: migrating; ETA: 158h48m26s
2026-06-26 09:07:27 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m0s(total), 3h8m59s(copy); streamer: dal13-sql21-bin.025288:194205311; Lag: 9356.80s, HeartbeatLag: 9356.81s, State: migrating; ETA: 158h48m26s []
Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m30s(total), 3h9m29s(copy); streamer: dal13-sql21-bin.025288:217244680; Lag: 9386.80s, HeartbeatLag: 9386.81s, State: migrating; ETA: 159h13m38s
2026-06-26 09:07:57 INFO Copy: 7775000/399786193 1.9%; Applied: 39572; Backlog: 1000/1000; Time: 3h9m30s(total), 3h9m29s(copy); streamer: dal13-sql21-bin.025288:217244680; Lag: 9386.80s, HeartbeatLag: 9386.81s, State: migrating; ETA: 159h13m38s []

I tried find out what is happening, but it wasn't throttled, couldn't find any issue for this idling.
So i enabled --verbose and found out that when processing the applied rows it encountered an error which made it just stay there:

2026-06-26 15:25:18 ERROR Error 1406 (22001): Data too long for column 'title' at row 1; query=
                  insert /* gh-ost.....

I think reporting that it encountered an error would be a good idea, and maybe aborting the copying, since there doesn't seem to be any method to resume the process.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions