[chriskirkland]

A Pull Request should be associated with an Issue.

We wish to have discussions in Issues. A single issue may be targeted by multiple PRs.
If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues,
and potentially we'll be able to point development in a particular direction.

Related issue: None; reported while investigating move-tables WriterLag flapping between approximately 0.1s and 3300s.

Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md
Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

Description

Replace the move-tables streamer-silence heuristic with ordered source-position probes. Previously, a full apply queue could block event delivery for a heartbeat interval, causing the idle timer to overwrite the last-applied timestamp with time.Now() despite a substantial writer backlog. Completing another old batch restored the high lag reading, producing the observed flapping.

The reader now periodically queries the hooked source's binlog position and emits a catch-up marker only when its fully processed position matches. That marker passes through the same stream and apply queues as DML; the writer updates lag only after preceding work completes. Markers retain the timestamp from before the source query, so query, queue, and writer delays remain visible. GTID probes wait for transaction completion, and file-position startup events and standalone GTID DDL are handled explicitly. Standard-mode streaming does not enable these probes.

This adds a read-only source-position query at the configured heartbeat interval (100ms by default; up to 1000ms). A failed probe follows the existing logged streamer-reconnect/error path rather than claiming the writer is caught up. Before the first DML batch or catch-up marker is processed, lag retains its existing zero initialization.

Added race-tested unit regressions for unread source positions across file/MySQL/MariaDB coordinates, idle streams, partial GTID transactions, standalone DDL, probe errors/cancellation, and a blocked writer. Extended the existing MySQL integration regression to ensure backpressure does not reset lag and to account for ordered markers. Documented the updated metric semantics.

Validation (local Go 1.26.4):

• go test -race ./go/base ./go/binlog — passed.
• go test -race ./go/logic -run '^Test(WriterLagProgress|MoveTablesWritableColumns|HooksExecutorMoveTablesEnvironmentVariables)' -count=10 — passed.
• go test -race ./go/binlog -run '^TestWriterLagProgress' -count=25 — passed.
• go vet ./go/binlog ./go/logic — passed.
• go build ./go/... — passed.
• git diff --check — passed.
• script/cibuild and Docker-backed integration suites — not run: the Docker daemon is unavailable locally. The repository build script also expects Go 1.20–1.25, whereas the installed toolchain is Go 1.26.4. The extended integration regression still needs CI validation.

In case this PR introduced Go code changes:

• contributed code is using same conventions as original code
• script/cibuild returns with no formatting errors, build errors or unit test errors.