Low image publishing rate

[wyattrees]

I have the Spot connected to our lab's WiFi network with the ROS driver running in a Docker container, but the publisher for all camera topics is only publishing at about 2 Hz. This is much lower than when I use the Spot Python SDK to retrieve the camera feed with the same setup, indicating that it's not a network bandwidth issue. Using the SDK isn't really an option for me, since I need a subscriber written in C# for Unity.

Could this be an error in the setup, or is this publishing rate for camera topics expected?

[heuristicus]

I don't think this is the expected rate. in my experience the rate is usually something slightly above 8Hz.

You can check the default rates that the driver uses in https://github.com/clearpathrobotics/spot_ros/blob/master/spot_driver/config/spot_ros.yaml, but note that in practice the actual rate that you will get is lower.

The main loop is also limited to 50Hz so you cannot receive messages faster than that https://github.com/clearpathrobotics/spot_ros/blob/master/spot_driver/src/spot_driver/spot_ros.py#L492, but this is an easy change.

[civerachb-cpr]

8Hz does seem excessively low.
Are the ROS master and the main nodes running on the Spot backpack PC and connected directly to Spot via ethernet cable? Or is ROS operating over the wireless?

[wyattrees]

The ROS master and main nodes are running in a Docker container on a separate PC, communicating with the Spot over a wireless network. However, I've been able to get enough bandwidth with the same setup to stream the cameras using the Spot SDK at a much higher frame rate than 2 Hz. I've tried setting the publish rate up to 50 in spot_ros.yaml, but I'm still only getting 2 Hz.

[heuristicus]

If you change the rate of the main loop at

https://github.com/clearpathrobotics/spot_ros/blob/a09e6add6b0ed192ffd60b19c64055d08505d96d/spot_driver/src/spot_driver/spot_ros.py#L492

does that help at all?

[civerachb-cpr]

The ROS master and main nodes are running in a Docker container on a separate PC, communicating with the Spot over a wireless network

This may actually be contributing to the problem. Rather than running roscore inside the Docker, can you try running it on the Spot itself, and use the ROS_MASTER_URI envar inside your docker container to connect to the ROS master running on the robot?

[gchustz]

I am having a similar issue, I am only getting ~2.5 FPS on the cameras, but I am running the ROS node on a PC connected to the Ethernet port on the back of the spot. The network usage is about 7 MB/s. I have also tried modifying the config yaml and the main loop rate with no improvement.

Any suggestions?

[heuristicus]

Were you able to find any solution for this issue?

[Hakaino]

@heuristicus I'm using the humble release and I'm also facing the same problem of 2.5Hz rates.
I tried both with Ethernet and cable and got consistent results.
I'm not being able to increase the main loop rate. Since different computers are achieving the approximately the same rates, I wander where is the main loop bottleneck.

I noticed that after commenting out some "time.sleep" the rate improved slightly, which confirms that the problem is in the main loop, but not exactly where you define its rate.

Can you advice?

[heuristicus]

If you're using ros2, I recommend that you use the ros2 driver at https://github.com/bdaiinstitute/spot_ros2. That driver implements the image publishing in a separate node, which apparently improves performance. I hope to implement a similar approach in the ros1 driver soon as well.

[Telios]

I'm experiencing the same problem on SpotCORE with ROS melodic running directly on SpotCORE and on ROS noetic inside a docker container.
In docker container:
~2.2Hz from the start even when changing rates.
Directly on SpotCORE:
Start with "high" refresh rate ~10Hz (depending on config) and gradually converge to ~2.5Hz (2min). I also get this error from time to time, when running directly on SpotCORE with ROS melodic.