CPU load too high when streaming video

[robbiedood]

Dear webrtc_ros friends,

We are concerning that the CPU usage is too high when using webrtc_ros. For example, webrtc_ros_serv takes over 160% CPU usage when streaming a 720p video. Moreover, we found it would go double if there is two browsers watching the same streaming.

Maybe we miss something, so hope to hear your expertise in order to optimize CPU loading.

Thanks,
-- Luke

[mjsobrep]

WebRTC isn't exactly an efficient protocol, but I too would like to see improvements here. Have you tried using another webrtc system to access the camera on your robot? https://appr.tc/ can be tried as a good benchmark, this will of course require being able to open a browser on your robot...

The expected behavior of WebRTC is that it will scale computation linearly with the number of connections. It re-encodes the media for each connection. To get around that, you need a media server. Setting that up is hard... I would love to see a plug and play solution for ROS with a full messaging server w/ authentication, media server, turn server, all ready to go for new users. Maybe I will work on that this winter.

The best thing to do is to down sample your video, you probably don't really need 720p or full frame rate.

I think there are other optimizations that could be made in the code, but I'm not really sure.

[robbiedood]

@mjsobrep thanks for your reply. Nice, https://appr.tc/ seems a good benchmark to us.

And Yes, we tried another webrtc system by webrtc-streamer. The cpu loading could be better but the linear behavior is the same.

Fortunately, so far our use cases require just few connections (but really need a high-reso video up to 1080p@30fps, and low-latency down to 100ms).

Although it's not CPU efficient, webrtc really gives a fantastic latency. We can't find other protocols which could be better.

[mjsobrep]

I've been wondering for a while if there are some compile time tricks that could make this all work better because I agree, WebRTC can't be beat for latency which is really important for the stuff I do. It seems like in general my cell phone handles webrtc better than my desktop/laptop, which makes me think there are things that the phones are doing on their chips. Wonder if we could get some of that working on robots.

I would expect webrtc-streamer to be a bit better. It doesn't have to deal with the ROS encoding/decoding of the images, which is a non-trivial task. Another possible way to improve your webrtc-ros performance would be to use nodelets. I haven't yet gotten that working.

There are multiple different codecs supported by webrtc. It might be worth looking into how they each perform. I don't think that webrt-ros supports codec selection, but maybe that is a worthwhile path for a contribution?