fix(messagequeue): remainder-aware fair share and orphan sweep - #526
Merged
Conversation
## Summary ### Why? The MySQL queue subscriber capped every node at ceil(P/N) partitions, computed independently per node. The caps sum to more than P, so uneven splits can settle into stable starvation: with 12 partitions and 5 subscribers every cap is 3 (sum 15), and 3/3/3/3/0 is a legal end state where every owner sits "at cap" and nothing ever obliges anyone to shed for the empty subscriber. Separately, nothing guaranteed an unleased partition would eventually be picked up if the share arithmetic misfired (divergent heartbeat views during the staleness window, or a subscriber that heartbeats without ever acquiring), and every shedding rebalance tick logged a spurious `lease renewal failed: ErrLeaseExpired` because rebalance sorted the shared slice in place and renewal then ran over the just-released tail. ### What? - `fairShareCap` is now remainder-aware: each subscriber ranks itself in the sorted `ActiveSubscribers` list; the first `P mod N` ranks get `floor(P/N)+1` and the rest `floor(P/N)`, so per-rank caps sum to exactly P. A starved subscriber or an unclaimed partition now implies a peer over/under its cap that rebalance and discovery resolve — neither is a stable state. The signature, the `0 = unlimited` contract, and the minimum-1 floor are unchanged; a subscriber missing from its own active view falls back to `ceil` over N+1 contenders (never unlimited, never zero). - Orphan sweep: every `2 x LeaseDurationMs` the discovery tick runs one uncapped acquisition pass. `TryAcquireLease` cannot steal a valid lease, so the sweep is a no-op in a healthy group, but a partition left unleased for any reason is picked up by whichever subscriber sweeps first; the next rebalance sheds any over-cap grab once a peer has capacity. - `rebalance` sheds from a sorted copy instead of reordering the caller's slice, and returns the released partitions; the lease tick renews only the kept set, eliminating the spurious `ErrLeaseExpired` error log on every shedding tick. ## Test Plan - ✅ `make test` — new `TestSubscriber_FairShareCap` covers the 12/5 starvation case (caps 3,3,2,2,2), P<N flooring, the missing-heartbeat fallback, and a property subtest asserting per-rank caps sum to exactly P for all N≤6, P≤13; `TestSubscriber_Rebalance*` cover the released-tail contract and that the input slice is not reordered. - ✅ `bazel test //test/integration/extension/messagequeue/...` — full Docker suite, including new `TestRebalance_NoStarvation_UnevenSplit` (12 partitions / 5 subscribers converge with every subscriber owning 2–3, total 12) and `TestRebalance_OrphanSweep` (phantom heartbeats cap the real subscriber at 1 of 3 partitions; all 3 messages are still delivered and acked via the sweep).
behinddwalls
force-pushed
the
messagequeue
branch
from
August 6, 2026 14:40
9dced9b to
1db0efc
Compare
behinddwalls
marked this pull request as ready for review
August 6, 2026 14:41
mnoah1
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why?
The MySQL queue subscriber capped every node at ceil(P/N) partitions, computed independently per node. The caps sum to more than P, so uneven splits can settle into stable starvation: with 12 partitions and 5 subscribers every cap is 3 (sum 15), and 3/3/3/3/0 is a legal end state where every owner sits "at cap" and nothing ever obliges anyone to shed for the empty subscriber. Separately, nothing guaranteed an unleased partition would eventually be picked up if the share arithmetic misfired (divergent heartbeat views during the staleness window, or a subscriber that heartbeats without ever acquiring), and every shedding rebalance tick logged a spurious
lease renewal failed: ErrLeaseExpiredbecause rebalance sorted the shared slice in place and renewal then ran over the just-released tail.What?
fairShareCapis now remainder-aware: each subscriber ranks itself in the sortedActiveSubscriberslist; the firstP mod Nranks getfloor(P/N)+1and the restfloor(P/N), so per-rank caps sum to exactly P. A starved subscriber or an unclaimed partition now implies a peer over/under its cap that rebalance and discovery resolve — neither is a stable state. The signature, the0 = unlimitedcontract, and the minimum-1 floor are unchanged; a subscriber missing from its own active view falls back toceilover N+1 contenders (never unlimited, never zero).2 x LeaseDurationMsthe discovery tick runs one uncapped acquisition pass.TryAcquireLeasecannot steal a valid lease, so the sweep is a no-op in a healthy group, but a partition left unleased for any reason is picked up by whichever subscriber sweeps first; the next rebalance sheds any over-cap grab once a peer has capacity.rebalancesheds from a sorted copy instead of reordering the caller's slice, and returns the released partitions; the lease tick renews only the kept set, eliminating the spuriousErrLeaseExpirederror log on every shedding tick.Test Plan
make test— newTestSubscriber_FairShareCapcovers the 12/5 starvation case (caps 3,3,2,2,2), P<N flooring, the missing-heartbeat fallback, and a property subtest asserting per-rank caps sum to exactly P for all N≤6, P≤13;TestSubscriber_Rebalance*cover the released-tail contract and that the input slice is not reordered.bazel test //test/integration/extension/messagequeue/...— full Docker suite, including newTestRebalance_NoStarvation_UnevenSplit(12 partitions / 5 subscribers converge with every subscriber owning 2–3, total 12) andTestRebalance_OrphanSweep(phantom heartbeats cap the real subscriber at 1 of 3 partitions; all 3 messages are still delivered and acked via the sweep).