logpuller: realtime large transactions can OOM before reaching eventstore

[asddongmen]

Problem

TiCDC can OOM when it is already running and receives a very large transaction
from the realtime TiKV CDC stream. In one reproduced case, a single 20GiB
transaction caused the CDC process to be killed by cgroup OOM around commit
time. After CDC restarted, the same transaction could be synchronized
successfully.

This issue focuses on the logpuller realtime ingestion path before rows are
persisted into eventstore.

Current logic

In logservice/logpuller/region_event_handler.go, TiKV CDC row events are
handled differently by event type:

• Event_COMMITTED is converted to common.RawKVEntry and appended to
  span.kvEventsCache directly.
• Event_PREWRITE is stored in state.matcher.putPrewriteRow.
• Event_COMMIT calls state.matcher.matchRow to match the cached prewrite
  row and fill Value / OldValue.

The matcher in logservice/logpuller/txn_matcher.go keeps unmatched prewrite
rows in unmatchedValue, keyed by (startTs, key).

For a realtime large transaction, CDC can receive many PREWRITE events first
and keep their row values in txnMatcher until the corresponding COMMIT
events arrive. At the same time, memory can also be held by gRPC/protobuf
unmarshal, kvEventsCache, the eventstore write queue, and Pebble batch growth.
For a 20GiB transaction, these holders can overlap and push the process over
the memory limit.

After CDC restarts, the upstream transaction has already committed. The same
history no longer stays in the "realtime PREWRITE waiting for COMMIT" state, so
CDC can continue from committed data without holding the whole transaction in
txnMatcher. This explains why the process can OOM in the realtime path but
successfully finish after restart.

Evidence

Raw pprof profiles are uploaded here:

https://gist.github.com/asddongmen/f9c40dfa1e52166408e3ea84c2144b61

The files are base64 encoded because GitHub gist does not accept binary pprof
files through gh gist create. Restore one with:

base64 -d highfreq-heap-31g-20260703-102906.pb.gz.b64 > highfreq-heap-31g-20260703-102906.pb.gz
go tool pprof -top highfreq-heap-31g-20260703-102906.pb.gz

Key profile observations:

• highfreq-heap-31g-20260703-102906.pb.gz
  • github.com/pingcap/kvproto/pkg/cdcpb.(*Event_Row).Unmarshal: 8.20GiB flat, 74.18%
  • github.com/pingcap/ticdc/logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 8.48GiB cumulative, 76.70%
  • github.com/cockroachdb/pebble.(*Batch).grow: 0.56GiB flat, 5.09%
• heap-28g-20260703-101449.pb.gz
  • github.com/pingcap/kvproto/pkg/cdcpb.(*Event_Row).Unmarshal: 5.93GiB flat, 47.07%
  • github.com/cockroachdb/pebble.(*Batch).grow: 4.25GiB flat, 33.74%
  • github.com/pingcap/ticdc/logservice/logpuller.(*regionRequestWorker).receiveAndDispatchChangeEvents: 6155.65MiB cumulative, 47.72%

Expected behavior

The logpuller realtime path should have bounded memory behavior for very large
transactions. A very large transaction may be slow, but it should not require a
process OOM and restart to finish syncing.

Possible areas to investigate:

• bound or spill txnMatcher prewrite bytes;
• add backpressure based on bytes held by kvEventsCache / eventstore write
  queue;
• avoid building very large Pebble write batches for a single realtime burst.