Skip to content

[ISSUE #10966] Fix TransactionalOpBatchService busy loop when deleteContext is drained - #10967

Open
waterWang wants to merge 1 commit into
apache:developfrom
waterWang:fix/transaction-op-batch-busy-loop
Open

[ISSUE #10966] Fix TransactionalOpBatchService busy loop when deleteContext is drained#10967
waterWang wants to merge 1 commit into
apache:developfrom
waterWang:fix/transaction-op-batch-busy-loop

Conversation

@waterWang

Copy link
Copy Markdown

What

Fixes #10966 — the TransactionalOpBatchService thread busy-loops at ~100% CPU after a transaction-message workload goes idle.

Root cause

deleteContext entries are only ever putIfAbsent in deletePrepareMessage() and never removed. Once a context has been fully drained (all offsets batched and sent), it stays in the map forever with a stale lastWriteTimestamp.

In batchSendOpMessage(), when every context is empty, the scan falls through to the stale-firstTimestamp branch:

  • firstTimestamp = min(startTime, staleLastWrite) → stale value
  • wakeupTimestamp = stale + interval < startTime → the wakeupTimestamp > startTime guard fails
  • method returns 0L

Back in TransactionalOpBatchService.run():

long interval = wakeupTimestamp - System.currentTimeMillis();   // 0 - now < 0
if (interval <= 0) { interval = 0; wakeup(); }
this.waitForRunning(interval);   // waitForRunning(0) returns immediately

→ tight loop, one CPU core pinned at ~100% indefinitely.

Fix

When no op message was batched in the round (sendMap == null), return System.currentTimeMillis() + transactionOpBatchInterval instead of 0L, so the thread sleeps until the next scheduled scan. New deletes still wake it promptly via deletePrepareMessage() -> transactionalOpBatchService.wakeup().

Verification

  • New regression test testBatchSendOpMessage_noPendingDataReturnsFutureWakeup — places a drained context (stale lastWriteTimestamp = 0) in deleteContext and asserts the returned wakeup time is in the future.
  • mvn -am -pl broker test -Dtest=TransactionalMessageServiceImplTestTests run: 9, Failures: 0, Errors: 0 (8 pre-existing + 1 new).
  • Standalone loop simulation of TransactionalOpBatchService.run() + batchSendOpMessage(): before fix 12,126,584 iterations/2s (busy loop), after fix 13 iterations/2s (sleeps normally).

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR fixes a critical busy-loop bug in TransactionalOpBatchService where the thread consumes ~100% CPU after transaction-message workloads go idle. The fix is minimal, well-documented, and includes a unit test that reproduces the issue.

Analysis

Root Cause (Accurate):

  • deleteContext entries are added via putIfAbsent but never removed
  • After draining, contexts remain with stale lastWriteTimestamp
  • batchSendOpMessage() returns 0 when all contexts are empty
  • Returning 0 causes immediate re-execution → busy loop

Fix Quality:

  • ✅ Minimal change (11 lines + clear comments)
  • ✅ Returns currentTime + interval instead of 0, breaking the loop
  • ✅ New deletes still wake the thread promptly via wakeup()
  • ✅ No breaking changes, backward compatible
  • ✅ Unit test reproduces the bug and validates the fix

Code Review:
The fix is correct and addresses the root cause. The comment clearly explains the "why" (not just "what"), which helps future maintainers understand the edge case. The test case properly validates the fix by ensuring the return value is a future timestamp.

Performance Impact:
This eliminates unnecessary CPU consumption in production environments with intermittent transaction-message workloads. A significant operational improvement.

Verdict

LGTM — Solid bug fix with proper test coverage. Ready to merge.


Automated review by github-manager-bot

…eleteContext is drained

When every entry in deleteContext has already been drained (offsets batched
and sent) but is still present in the map, batchSendOpMessage() falls through
to the stale lastWriteTimestamp branch and returns 0L. TransactionalOpBatchService
treats 0 as "run again immediately", so waitForRunning(0) returns at once and the
thread busy-loops at ~100% CPU after any transaction-message workload goes idle.

Fix by returning System.currentTimeMillis() + transactionOpBatchInterval when
no op message was batched in the round (sendMap == null), so the thread sleeps
until the next scheduled scan. New deletes still wake it promptly via
deletePrepareMessage() -> wakeup().

Signed-off-by: waterWang <waterwang@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RocketMQ 5.5.0 的事务消息批处理线程陷入忙循环,CPU 10%

2 participants