Enhancements from Kafka Mirrormaker2 forlog truncation detection and topic reset handling.

[anurag-ris90]

MirrorMaker2: Detect log truncation (fail-fast) and auto-recover on topic reset

Summary

This PR extends MirrorMaker 2 to handle two important failure scenarios for cross-cluster replication:

1. Silent data loss (log truncation)

   • Detect when a source partition's earliest available offset has advanced past the expected offset (messages were truncated due to retention).
   • On detection, MirrorMaker logs a clear error (TRUNCATION-DETECTED) with topic/partition/expected/beginning offsets and fails-fast to surface the issue to operators.

2. Topic reset (deletion & recreation)

   • Detect topic recreation by observing a backward change in the beginning offset or partition disappearance/reappearance.
   • On detection, MirrorMaker logs (TOPIC-RESET-DETECTED with timestamp and topic) and automatically resubscribes / seeks to the beginning offset and clears offset translation state as needed.

Design rationale

• Uses consumer.beginningOffsets(...) to compare expected offsets before processing a batch.
• Minimal, localized changes within connect/mirror classes to avoid broad impact.
• Maintains clear SLF4J logging markers to help automation scripts identify events.

Test plan

• Included integration demonstration via docker-compose and run_challenge.sh in the project repo:
  • Normal replication: producer -> MM2 -> DR consumer
  • Truncation simulation: pause MM2, wait for retention.ms to expire, resume -> expect TRUNCATION-DETECTED and failure
  • Topic reset simulation: delete and recreate topic -> expect TOPIC-RESET-DETECTED and auto-resubscribe

Notes

• Small, focused changes (aim ~<500 LOC).
• Signed-off-by: Anurag Gupta [email protected]

[LiamClarkeNZ]

You have a massive amount of diffed files in here. Looking at the changes, I think you need to merge in the latest apache:trunk branch as your actual changes are lost in the noise.

Once you've done so, ping me and I'll re-review.