multi-robot in simulation and the limitation of namespacing

[moutalibbadr]

Describe the the feature you would like
We would like to have a multi-tb4 system running on Gazebo Fortress and our first approach was namespacing. This seemed to be the only possible one since we needed to be on the same Domain-ID of the gz server in order to have the robots simulated. However, it seems that each ROS_DOMAIN_ID can only support 120 nodes and each tb4 with the nav stack needs up to 70 nodes if not more, so the multi-robot scenario on simulation will not be solved this way.

Solving this issue could mean providing researchers who want to try and test a multi-robot system and who can not come up with the budget to get a fleet of robots and solve all the networking issues, with a first step solution that will empower OSS users.
Most the discussion link2 I've seen revolves around real robots and the use of DDS partitions or Domain_ID's to limit conflict between robots and enhance security but this, for now, doesn't map to the issue of multi-robot in gazebo.
Unless I am missing something, it seems that there is room for improvement on this front and we would like to progress on this discussion and help out on creating a stable solution.

PS
Please tell me if I'm on the wrong space for this type of discussion.

[roni-kreinin]

By nav stack are you referring to Nav2 and slam_toolbox? If so, I have been able to run two robots each with their own nav stack in simulation using just namespacing.

A lot of the nodes created for the sim are actually just ros_gz_bridge nodes that bridge ignition messages to ROS 2. If you are having issues with the number of nodes being created, I would suggest you modify the two ros_gz_bridge launch files (create3_sim)(turtlebot4_simulator), and use a single bridge node for all topics. This may cause reduced performance and there may be limits to how many bridges can be made in one node, but I haven't tested this.

[moutalibbadr]

By nav stack are you referring to Nav2 and slam_toolbox? If so, I have been able to run two robots each with their own nav stack in simulation using just namespacing.

What about 4 or more robots. In my side it seems that the CPU is bottlenecking around two robots so i can not verify if indeed 4 robots with nav2 and slam-toolbox will run. But a CPU can be upgraded, what i'm more worried about is the domain_id limitations. I might be wrong, Let me know what you think.

[roni-kreinin]

For me the bottleneck is usually with my GPU and all of the lidars being rendered (There are 13 in total for each turtlebot4). I have also not been able to spawn more than 3 robots without gazebo crashing. This might be caused by CPU/GPU limitations. You can try disabling some of the ray sensors if you are not using them (ir intensity, cliff sensors) to improve performance.

Regarding the domain ID, https://docs.ros.org/en/humble/Concepts/About-Domain-ID.html mentions that If it is known that the computer will only ever be on a single domain ID at a time, and the domain ID is low enough, it is safe to create more ROS 2 processes than this. So I think it should be ok to exceed 120 processes on the same domain ID.