AMD CPU XDNA(NPU) better than iGPU, use openvino get 3ms

[Vincent5220]

Im new here,
I was using reolink independently.
And want to use Frigate NVR for more flexibility.

I now truly love the Frigate NVR.
Want to share setting for AMD CPU XDNA(NPU) user.
You wont need iGPU or dGPU at all.

System:
Miniforum UM790
AMD 7940HS which include XDNA(NPU)
Running Frigate on Ubuntu 24.04 using Portainer

Camera detect at
1x Reolink Doorbell 2K
2x Reolink RLC810A 4K

Set the Frigate config to use Openvino as Official Doc described.
Just remember to us CPU as detector type.
DONT USE Auto and GPU, Frigate will failed to Start.

detectors:
  ov:
    type: openvino
    device: CPU

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

And you will get Inference Speed about 3-5ms
Also you can use YoloNAS model, you will get 7-9ms.

Truly Amazing, excellent job Frigate with NPU.
For everyone with AMD CPU XDNA(NPU).

[gaynetdinov]

How do you know that detection is done using NPU and not CPU?

[Vincent5220]

Truly speaking, I don't know.
But from the official doc
i5-12600K Inference speeds on CPU were ~ 35 ms
i5-12600K Inference speeds on CPU&GPU were ~ 15 ms

The real world CPU benchmark of 12600K and 7940hs is about same.
But 7940HS get ~3ms. (default openvino setting)

The only reason for this speed would be the NPU with 10TOPS...
Also another user with 7940HS #9417

AMD XDNA supports ONNX, so maybe that is why it support openvino detector.
Thinking XDNA2 with 50TOPS will be an ultimate AI server for home user...

[nschimme]

This is likely using the CPU and not the NPU. The XDNA driver isn't mainlined yet and I doubt you installed the driver? https://github.com/amd/xdna-driver

[Vincent5220]

If you think pure CPU can get such result, fine.(maybe it's iGPU LOL)

No, I do not install XDNA driver.
I installed the AMD united driver and get ROCm.
(No software installed, just this page of installation)
https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/native_linux/install-radeon.html

The point is

1. OpenVINO do support x86 CPU.
2. 7940HS can handle it very well

[mikealanni]

I have 7945HX and I have tried openvino on SSDlite and YoloNas_s both worked but they are good in detection. As I have stop sign on my door to know if it is closed or open. While using rocm on yolonas_s way better and can detect the stop sign. But my infernance speed on rocm is like 70-90ms. The strange thing is if I use 0.13 rocm my infernance speed is 10-15ms on yolo8.