Found while running the ros2-benchmark-container comparison on devel + #61 (7e39aa3), SYSTEM_EXECUTOR=EventsExecutor: every cross-process suite recorded received_msgs = 0 for this RMW — multi-process pub/sub (all payload sizes), mix-process pub/sub, cli/srv, actions, and scalability — while single-process ipc_on runs (delivery via rclcpp intra-process) produced normal numbers. The publisher side sends normally until the subscriber's socket fills, then logs subscriber recv buffer full once per second for the rest of the test. Nothing crashes and nothing errors: the data is silently never delivered.
Mechanism
The ingestion pump only runs inside rmw_wait (drain_socket) and the take family (drain_subscription), and on_new_message_cb fires only from those drains:
- EventsExecutor never calls
rmw_wait for subscriptions/services/clients — it waits for rmw_*_set_on_new_*_callback notifications.
- The callback only fires during a drain; a drain only happens in
rmw_wait or rmw_take; rmw_take is only called after a callback.
So under a pure EventsExecutor the first cross-process datagram is never noticed: no drain → no callback → no take → no drain. There is no reader thread anywhere in the RMW to break the cycle.
Note #61 is what made this visible: before it, EventsExecutor aborted at subscription creation, so no run ever got far enough to lose data silently. The July benchmark numbers were collected with wait-based executors for the same reason.
Production impact
None for wait-based executors (SingleThreadedExecutor / MultiThreadedExecutor) — rmw_wait drains everything. Any node opting into EventsExecutor gets 100% silent cross-process message loss.
Fix direction
A per-context listener thread epolling all subscription/service/client sockets, draining into the existing mutex-guarded queues and firing the armed callbacks — the analogue of a DDS listener. Wait-based executors are unaffected (level-triggered epoll + the queue-non-empty fast path in rmw_wait already tolerate a concurrent drainer). Until then, EventsExecutor should be treated as unsupported: arguably rmw_*_set_on_new_*_callback should return an error rather than accept a callback the transport cannot honor, so rclcpp fails fast instead of losing data silently.
Benchmark evidence: benchmark_results/jazzy/rmw_comparison_20260819/ (EventsExecutor run) on the benchmark host; e.g. pub-sub_multi_process/10b/unix_socket_ipc_off/sub_10b_log/latency_total.txt → received_msgs,mean_us = 0,nan.
Found while running the ros2-benchmark-container comparison on devel + #61 (
7e39aa3),SYSTEM_EXECUTOR=EventsExecutor: every cross-process suite recordedreceived_msgs = 0for this RMW — multi-process pub/sub (all payload sizes), mix-process pub/sub, cli/srv, actions, and scalability — while single-processipc_onruns (delivery via rclcpp intra-process) produced normal numbers. The publisher side sends normally until the subscriber's socket fills, then logssubscriber recv buffer fullonce per second for the rest of the test. Nothing crashes and nothing errors: the data is silently never delivered.Mechanism
The ingestion pump only runs inside
rmw_wait(drain_socket) and the take family (drain_subscription), andon_new_message_cbfires only from those drains:rmw_waitfor subscriptions/services/clients — it waits forrmw_*_set_on_new_*_callbacknotifications.rmw_waitorrmw_take;rmw_takeis only called after a callback.So under a pure EventsExecutor the first cross-process datagram is never noticed: no drain → no callback → no take → no drain. There is no reader thread anywhere in the RMW to break the cycle.
Note #61 is what made this visible: before it, EventsExecutor aborted at subscription creation, so no run ever got far enough to lose data silently. The July benchmark numbers were collected with wait-based executors for the same reason.
Production impact
None for wait-based executors (
SingleThreadedExecutor/MultiThreadedExecutor) —rmw_waitdrains everything. Any node opting intoEventsExecutorgets 100% silent cross-process message loss.Fix direction
A per-context listener thread epolling all subscription/service/client sockets, draining into the existing mutex-guarded queues and firing the armed callbacks — the analogue of a DDS listener. Wait-based executors are unaffected (level-triggered epoll + the queue-non-empty fast path in
rmw_waitalready tolerate a concurrent drainer). Until then,EventsExecutorshould be treated as unsupported: arguablyrmw_*_set_on_new_*_callbackshould return an error rather than accept a callback the transport cannot honor, so rclcpp fails fast instead of losing data silently.Benchmark evidence:
benchmark_results/jazzy/rmw_comparison_20260819/(EventsExecutor run) on the benchmark host; e.g.pub-sub_multi_process/10b/unix_socket_ipc_off/sub_10b_log/latency_total.txt→received_msgs,mean_us = 0,nan.