[Support]: v15 Beta1: OpenVINO 2024.3 Fails to run INT8 YOLO-NAS Models correctly on the GPU

[aeozyalcin]

Checklist

• I have updated to the latest available Frigate version.
• I have cleared the cache of my browser.
• I have tried a different browser to see if it is related to my browser.
• I have tried reproducing the issue in incognito mode to rule out problems with any third party extensions or plugins I have installed.

Describe the problem you are having

I found that OpenVINO 2024.3 is not running INT8 YOLO-NAS models correctly with the GPU. Running the INT8 model on the CPU, or using the default FP16/32 images on the GPU does not exhibit this issue. See shared image below - the inference generates a bunch of noise.

I found that hopping into the container, and updating the openvino python package to openvino-2024.4.0 fixes this issue. I recommend updating OpenVINO to 2024.4 going forward.

Steps to reproduce

1. Export Yolo-NAS model with the following parameters:

   model.export("yolo_nas_s_int8.onnx",
                 output_predictions_format=DetectionOutputFormatMode.FLAT_FORMAT,
                 quantization_mode=ExportQuantizationMode.INT8,
                 max_predictions_per_image=20,
                 confidence_threshold=0.4,
                 input_image_shape=(416,416),
                )

2. Load onto computer and configure Frigate to use this model, with GPU

...

Version

v15 Beta 1 (0.15.0-33825f6)

In which browser(s) are you experiencing the issue with?

No response

Frigate config file

detectors:
  ov:
    type: openvino
    device: GPU

model:
  model_type: yolonas
  width: 416 # <--- should match whatever was set in notebook
  height: 416 # <--- should match whatever was set in notebook
  input_tensor: nchw
  input_pixel_format: bgr
  path: /media/frigate/openvino-models/yolo_nas_s_int8_11_4.onnx
  labelmap_path: /media/frigate/openvino-models/coco_80cl.txt

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate_beta
    privileged: true # this may not be necessary for all setups
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:0.15.0-beta1
    shm_size: "512mb" # update for your cameras based on calculation above
    devices:
      # - /dev/bus/usb:/dev/bus/usb # passes the USB Coral, needs to be modified for other versions
      # - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
      # - /dev/dri/renderD128 # for intel hwaccel, needs to be updated for your hardware
      - /dev/dri/renderD128:/dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/raid1_hdd/frigate_v15/config:/config
      - /mnt/raid1_hdd/frigate_v15/media:/media/frigate      
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    cap_add:
      - CAP_PERFMON
    ports:
      - "5000:5000"
      - "1984:1984"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    environment:
      FRIGATE_RTSP_PASSWORD: "blah"

Relevant Frigate log output

No errors/messages logged

Relevant go2rtc log output

No errors/messages logged

Operating system

Other Linux

Install method

Docker Compose

Network connection

Wired

Camera make and model

Reolink & Amcrest

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

This is not a bug and is working as expected, openvino does not support quantized models on GPU in the current version.

[aeozyalcin]

It's possible the newer dep fixes this, but until onnx runtime is updated we can not update the openvino dep.
https://pypi.org/project/onnxruntime-openvino/

pypi onnxruntime-openvino runtime has been updated to version 1.20 (currently the dep in Frigate point to 1.19). If I'm not mistaken 1.20 supports 2024.4.x

Yep!

I'll update this on my container running 2024.4 and report back - although onnx-runtime-1.19 also worked just fine with OpenVino 2024.4.

[aeozyalcin]

Oof this is worrying, from the release notes. Looks like Frigate still uses 3.9...
ONNX Runtime packages will stop supporting Python 3.8 and Python 3.9.

[NickM-27]

Ah, in that case we will have to wait until 0.16 to support this most likely

[xavan83]

Oof this is worrying, from the release notes. Looks like Frigate still uses 3.9... ONNX Runtime packages will stop supporting Python 3.8 and Python 3.9.

Do you try v0.16 as suggested? I try it (INT8) and it seems to work with the colab in frigate documentation.

[aeozyalcin]

Oof this is worrying, from the release notes. Looks like Frigate still uses 3.9... ONNX Runtime packages will stop supporting Python 3.8 and Python 3.9.

Do you try v0.16 as suggested? I try it (INT8) and it seems to work with the colab in frigate documentation.

Not sure I follow. V16 isn't out yet.