ring_doorbells() runs after every registry mutation (registry.cpp#L395-L470). It does two things that scale with the size of the system rather than with the size of the change:
- Walks all slots from 0 to
high_water_slot looking for doorbell entries. On a system with ~7400 slots in use and 159 processes that is 7400 slot reads to find 159 doorbells. The reads are strided 1168 bytes apart, so almost every one is a cache miss.
- Sends a datagram to each doorbell, so all 159 processes wake up and re-check the registry.
Creating or destroying any node, publisher, subscription, service or client triggers this.
During startup, when many nodes register at once, we measured 127.8 mutations/s. With 159 processes that is roughly 20k wakeups/s across the machine, plus 127.8 x 7400 slot reads/s in the mutating processes. An 8-core machine drops to 0% idle and stays there until the launch settles.
There is a second effect in the EAGAIN path (registry.cpp#L452-L465). When a peer is slow to drain its doorbell, sendto fails, and the sender closes its socket, creates a new one and retries. That is three extra syscalls per slow peer per mutation, so a single saturated process makes every other process slower.
For context, this replaced a 200 ms poll in 1ad02b5. The doorbell is the better design and that commit fixed two real bugs. But the poll also happened to cap this work at 5 Hz, and removing the poll removed the cap.
Suggested fix
- Keep doorbells in a compact structure, a small array or a bitmap in the header, so ringing is O(doorbells) instead of O(high_water_slot).
- Give each doorbell socket a small
SO_RCVBUF. The doorbell carries no payload, it is only an edge signal, so redundant datagrams are safe to drop. A small buffer lets the kernel coalesce them and the drain loop then reads one or two messages instead of hundreds.
- Once the receive queue is bounded,
EAGAIN unambiguously means a wakeup is already queued for that peer, so the right action is to skip it rather than recreate the socket.
If that is not enough, debouncing the generation check on the receiving side would bound what is left. The timeout contract fixed in 1ad02b5 constrains what rmw_wait returns, not how often it re-reads the registry, so a debounce there does not bring that bug back.
ring_doorbells()runs after every registry mutation (registry.cpp#L395-L470). It does two things that scale with the size of the system rather than with the size of the change:high_water_slotlooking for doorbell entries. On a system with ~7400 slots in use and 159 processes that is 7400 slot reads to find 159 doorbells. The reads are strided 1168 bytes apart, so almost every one is a cache miss.Creating or destroying any node, publisher, subscription, service or client triggers this.
During startup, when many nodes register at once, we measured 127.8 mutations/s. With 159 processes that is roughly 20k wakeups/s across the machine, plus 127.8 x 7400 slot reads/s in the mutating processes. An 8-core machine drops to 0% idle and stays there until the launch settles.
There is a second effect in the EAGAIN path (registry.cpp#L452-L465). When a peer is slow to drain its doorbell,
sendtofails, and the sender closes its socket, creates a new one and retries. That is three extra syscalls per slow peer per mutation, so a single saturated process makes every other process slower.For context, this replaced a 200 ms poll in 1ad02b5. The doorbell is the better design and that commit fixed two real bugs. But the poll also happened to cap this work at 5 Hz, and removing the poll removed the cap.
Suggested fix
SO_RCVBUF. The doorbell carries no payload, it is only an edge signal, so redundant datagrams are safe to drop. A small buffer lets the kernel coalesce them and the drain loop then reads one or two messages instead of hundreds.EAGAINunambiguously means a wakeup is already queued for that peer, so the right action is to skip it rather than recreate the socket.If that is not enough, debouncing the generation check on the receiving side would bound what is left. The timeout contract fixed in 1ad02b5 constrains what
rmw_waitreturns, not how often it re-reads the registry, so a debounce there does not bring that bug back.