OpenVino, YOLO-nas, N150 Beelink EQ14, number of detectors and performance

[daubman]

I am migrating from a Beelink EQ12 N100 with m.2 Coral to an EQ14 N150 and trying Openvino to compare. I am using the latest 0.15.0 Beta 4 on both systems.

I have 10 cameras, detect is always using low-quaity substreams at 5-8fps. With the coral, everything was performant, speeds were ~10ms, coral never seemed busy.

I am currently using the basic config recommendations with two ov detectors and the yolo-nas model built from the (modified for new url) colab notebook:

detectors:
  ov_0:
    type: openvino
    device: GPU
  ov_1:
    type: openvino
    device: GPU
model:
  model_type: yolonas
  width: 320 # <--- should match whatever was set in notebook
  height: 320 # <--- should match whatever was set in notebook
  input_tensor: nchw
  input_pixel_format: bgr
  path: /config/yolo_nas_s.onnx
  labelmap_path: /labelmap/coco-80.txt

This seem to be working OK (is there any discussion around detection accuracy/performance vs coral and default builtin model?) except I get sporadic (and especially startup) spikes very high for the ov detectors.

I also didn't see any discussion of how many detectors, and especially how to tell how many iGPU like N150 could/should support... The graphs look identical, but should I add more detectors? How many is enough vs too many, etc...

At 45 detections per second, I'm skipping 9 per second:

[daubman]

As a followup, it would be great to have adjustable metric time windows, or at least to ensure the same time slices are used across tabs to compare between the camera and general metrics screens (the ability to see the Overview Frames/Detections graph in the System metrics view would be super helpful).

Update: now my metrics tabs are mostly in sync, whereas they weren't when I was first grabbing screenshots 🤷‍♂️

[NickM-27]

There is no discussion about these things you mention because it is very specific to your circumstances. If there is no activity then the only limiting factor is how much RAM you have and how many instances of the model can fit in the GPU RAM. But of course if there is a lot of activity the GPU may be a bottleneck. This is specific to the GPU, system, etc. and most of the time with recent hardware the possibilities far outreach what is practically necessary.

The OV detector does require more CPU as it is more work to prepare the frames for inference

[daubman]

I will try with a 3rd ov detector and see how it does, GPU doesn't seem to go above 50% currently...
Are there any more details or places I should look to learn about tuning width/height for yolonas (or other models)?
Are there any details or comparisons of accuracy/performance of default model vs yolonas? (is there a way for me to compare usefully?)

I'm trying to figure out if I should stick with the Coral, which seems to be the older rec and still all over the docs, or if Openvino should be sufficient, which seems to be the newer rec but still not mentioned nearly as much as Coral.

[NickM-27]

whether or not openvino provides an advantage in accuracy / performance will depend very much on your specific scenario and use case. There's not reliable way for us to predict what the best decision is.

[daubman]

Thanks for all the details so far @NickM-27 (and happy new year!).

I dropped from 10 to 9 cameras but am still hitting bottlenecks and dropped detections any time I get to ~40 detections per second. I'm trying to learn how to reason better about what is going on. All of my cameras are set to detect at 5fps using substreams, but I have various different substream resolutions.

I noticed some cameras will generate 20 detections per second from a 5fps detect stream. Is this because the model is currently set at 320x320, and so a camera with a 1920x1280 detect stream resolution could generate 24 individual detections per frame, so up to 120 detections per second?

Is there a general process/flow-chart I should go through when attempting to tune and determine bottlenecks - e.g. if I see a camera generate more detections than others, see if I can drop the resolution of the detect stream if I don't need small object detection, or if it is an indoor camera, etc?

EDIT:
And going back to my previous question, when attempting to compare model performance, detection needs, etc, re:

performance will depend very much on your specific scenario and use case. There's not reliable way for us to predict what the best decision is.

Are there steps I can/should take to compare setups to see what works better? Do I just run for a day one way and a day another way and then make my best guess from reviewing alerts and assuming events will be similar enough across a large enough time frame, or are there other better ways to test models, detections sizes, frame rates, etc...

[NickM-27]

A single frame can lead to multiple inferences being run. Frigate will scale down a region so if there is a larger area it doesn't mean it has to fit in 320x320. The best way to debug this is to look at the debug view. That will tell you if there is errant motion detection, just general activity, or some other issue going on. In general 1280x720 is the sweet spot for the detect stream to run on.

In general yes it is easiest to let things run and compare the results, you want to avoid skipped fps.

[nukefrenzy]

I'm trying to setup my new Beelink EQ14 N150 and I cannot get the device to recognize the iGPU. /dev/dri is empty so there's nothing to pass along to the frigate container. How did you get this up and running?

[nukefrenzy]

Crap, thought I was talking to OP. Thanks for trying to help tho!

[daubman]

In case it helps, here's what I did. I am running docker in a Proxmox LXC, so I followed these directions on the proxmox host, and then in the docker LXC. I think it was really the "compile and install latest drivers and utils from source" steps at the end that were necessary. I saw in some random thread that the n150 likely won't see fully builtin support until kernel 6.12.5. I also just ordered a Beelink EQi12 for ~$70 more than the EQ14 - more and better ram, nvme, CPU, iGPU, etc... I'll compare against the EQ14 that has been disappointing so far and may return the EQ14. (Wish there were a cheap N305 or 125H minipc). With the EQ14, even dropping from 10 to 9 cameras, tuning resolutions and attempting other tricks, I will still often get dropped detects with yolonas.

Installed Proxmox VE 8.3.2 with kernel 6.11
https://wiki.debian.org/HardwareVideoAcceleration#VA-API
https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_debian_firmware_repo

# vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

# update-grub

# vi /etc/modules
vfio
vfio_iommu_type1
vfio_pci

# update-initramfs -u -k all

# echo "options i915 enable_guc=2" >> /etc/modprobe.d/i915.conf
# update-initramfs -u -k all
# apt install intel-gpu-tools

To be able to install packages from this component, run editor /etc/apt/sources.list, append non-free-firmware [and non-free] to the end of each .debian.org repository line and run apt update.

# apt install intel-media-va-driver-non-free
# apt install vainfo

A helpful quote from a redditor in a similar situation:
…I followed the build and install steps here https://github.com/intel/media-driver?tab=readme-ov-file#building step by step, starting with building and installing libva based on the instructions on that github page (using the "For debian family, you could use autogen" step), followed by gmmlib based on the instructions on that page (using -DCMAKE_BUILD_TYPE=Release ). Finally I built and installed media-driver.

I then also installed: https://github.com/intel/libva-utils/tree/master?tab=readme-ov-file#build-and-install-libva-utils

[nukefrenzy]

I knew I should have stuck with the N100 or gone with a more powerful one as you suggest. I'm returning this thing as I'm not looking for a research project. The EQi12 processor has been out since Q2 2022 so that seems like a safe bet and will do much better with my 7 cameras with room for more. Thanks for taking the time to respond.

[daubman]

Just following up to say that the EQi12 has been so so so much better for frigate (and in general) than the EQ14. Graphics support didn't require anything special, and OpenVino yolonas inference is down to 15-30ms, but most impressively, CPU and GPU utilization mostly remain in the single digits, rather than jumping to the red and generating warnings frequently with the EQ14. For the $70 extra dollars and marginal power use increase (EQi12 seems to run cooler, I haven't stuck a power monitor on it yet though) it seems totally worth it and a way better option. I wonder if I can drop down my number of detectors seeing as they don't appear to get busy and nothing is being dropped now (still lots of drops with exact same config on the EQ14).