Poor Streaming Performance

[RhysDeLoach]

I've got a Yolov11 object detection model that performs very well when analyzing recordings from a camera. However, when I stream from that same camera with the same settings, my model performs very poorly. I've set the stream parameter to true, and I have not seen much of a difference. What are some steps I can take to fix this? Is this something that can be solved with a better camera?

Whether there is a wired connection or not, I see about the same performance. I've also tried threading, but this doesn't seem to make a big difference.

[UltralyticsAssistant]

👋 Hello @RhysDeLoach, thank you for sharing your experience with Ultralytics 🚀! We’re here to help.

We recommend checking out our Docs where you’ll find extensive information on both Python and CLI usage. There may already be pre-existing solutions or optimizations to improve streaming performance.

If this is a 🐛 Bug Report regarding streaming performance, please provide a minimum reproducible example including:

1. Your code snippet (especially how you are setting the stream parameter and any threading configurations).
2. Details regarding your hardware setup (camera specs, wired/wireless connection types, etc.).
3. Logs or outputs showcasing the performance discrepancy between recording analysis and live streaming.

If this is a ❓ Question regarding improving streaming performance, some additional context such as camera specs, frame rate, resolution, and network bandwidth (for wireless setups) might help us better understand your setup.

In the meantime, ensure you are leveraging the latest features and updates by upgrading your ultralytics package and dependencies in a Python>=3.8 environment with PyTorch>=1.8:

pip install -U ultralytics

For those exploring GPU-accelerated environments, consider running YOLO in one of these environments, which are preconfigured for optimal performance:

• Notebooks with free GPU:
  
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

For real-time collaboration or exchanging ideas with the Ultralytics community, you might also find these platforms useful:

• Join Discord 🎧 for instant help.
• Engage with others via our Discourse forum.
• Discuss and share on the Subreddit.

Your inquiry has also been flagged for support, and an Ultralytics engineer will assist you shortly. Thank you for your patience and for being part of the Ultralytics community! 😊

[glenn-jocher]

@RhysDeLoach for live streaming performance improvements with YOLOv11, consider these optimizations:

1. Use solutions.Inference() instead of base prediction - it's specifically optimized for streaming (see Streamlit Inference Docs)
2. Try smaller models like YOLOv11n and reduce input resolution
3. Enable GPU acceleration with device=0 parameter
4. Implement frame skipping (process every Nth frame)

If using RTSP streams, ensure your network latency is <100ms. For code examples, see our Live Inference Guide.

[RhysDeLoach]

@glenn-jocher I appreciate you getting back to me. I've implemented all of these approaches except for solutions.Inference(), which I will try. However, I am starting to think that it is an issue with the stream quality of the camera because it is a cheap USB 2.0 camera, and my model is processing at about 5x the speed of the FPS. Am I right in assuming this?

[RhysDeLoach]

Also, it should be mentioned that I am counting the objects.

[glenn-jocher]

Thank you for the additional context. For object counting in streaming scenarios, ensure you're using our optimized solutions.ObjectCounter class which includes dedicated methods for crowd management. While camera quality can impact performance, first verify your actual capture FPS vs expected using cv2.CAP_PROP_FPS - sometimes USB2 cameras achieve sufficient throughput when resolution is reduced. The solutions module includes frame processing optimizations that might help bridge any hardware limitations.

[RhysDeLoach]

Yes, I tried the ObjectCounter. I couldn't get it to count properly. It would correctly identify my objects, but the counting logic wouldn't work (only counting 22 of 200 objects in one video). As of right now, I am using the track function, and I am counting the objects with my own logic. By adjusting some parameters, I've got my model back under 10% counting error (it is less than 1% when analyzing recordings), and the last bit of error is being caused by the track function assigning different objects the same track id when one comes in the frame at the same trajectory that the other left. I am guessing that this is a symptom of lower fps, so I am exploring other options.

I am curious as to why I saw a performance drop in the model by switching to streaming. Intuitively, I understand how this could effect the tracking mechanism because frames are being dropped, but I don't understand why my model is having a harder time identifying the objects themselves.

[glenn-jocher]

For tracking ID swaps during streaming, ensure you're using persist=True in model.track() and consider adjusting conf/iou thresholds. The detection quality drop during streaming could stem from frame timestamp inconsistencies - verify actual capture FPS with cv2.CAP_PROP_FPS. For robust counting implementations, we recommend revisiting the ObjectCounter with proper region configuration as shown in our Object Counting docs, which handles track interpolation.

[RhysDeLoach]

Update. I was able to achieve real-time object counting with about 1% error by doing the following. Switching to a computer with a better GPU, optimizing the model by adjusting parameters such as IOU and setting half to True, setting custom tracking parameters (this was a big one), and using threading to handle streaming issues such as inconsistent FPS and dropped frames.

[glenn-jocher]

@RhysDeLoach that's great to hear you've significantly improved your real-time object counting performance, and thank you for sharing your optimizations with the YOLO community!

[RhysDeLoach]

No problem. Thank you for your help as well!

[glenn-jocher]

You're welcome, and the Ultralytics team appreciates your positive feedback and contributions to the YOLO community!