[Detector Support]: Truenas Scale - Openvino Crashing App

[daviddreher2]

Describe the problem you are having

Frigate TrunNAS App is crashing randomly and getting stuck on "Deploying"

Error Message: [GPU] clWaitForEvents, error code: -14

Detector is otherwise working correctly with inference ~25ms

TrueNAS Scale version: ElectricEel-24.10.1

Version

0.14.1-f4f3cfa

Frigate config file

telemetry:
  # Optional: Configure system stats
  stats:
    amd_gpu_stats: false
    intel_gpu_stats: true
    network_bandwidth: true

ffmpeg:
  hwaccel_args: preset-intel-qsv-h264

detectors:
  ov:
    type: openvino
    device: GPU

record:
  enabled: true
  retain:
    days: 5
    mode: all
  events:
    retain:
      default: 30
      mode: motion

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  path: /openvino-model/ssdlite_mobilenet_v2.xml
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt

docker-compose file or Docker CLI command

TrueNAS generated docker run command

docker run \
  --name "/ix-frigate-frigate-1" \
  --runtime "runc" \
  --mount type=bind,source=/mnt/storage/Config/certs/selfsigned.crt,destination=/etc/letsencrypt/live/frigate/fullchain.pem,bind-propagation=rprivate \
  --mount type=bind,source=/mnt/storage/Config/certs/selfsigned.key,destination=/etc/letsencrypt/live/frigate/privkey.pem,bind-propagation=rprivate \
  --mount type=bind,source=/mnt/storage/Frigate/media,destination=/media,bind-propagation=rprivate \
  --mount type=volume,destination=/tmp \
  --mount type=bind,source=/mnt/storage/Frigate/cache,destination=/tmp/cache,bind-propagation=rprivate \
  --mount type=bind,source=/mnt/storage/Frigate/config,destination=/config,bind-propagation=rprivate \
  --log-driver "json-file" \
  --restart "unless-stopped" \
  --cap-add "CHOWN" \
  --cap-add "DAC_OVERRIDE" \
  --cap-add "FOWNER" \
  --cap-add "PERFMON" \
  --cap-add "SETGID" \
  --cap-add "SETUID" \
  --cap-drop "ALL" \
  --device "/dev/dri":"/dev/dri":rwm \
  --publish "0.0.0.0:30058:8971/tcp" \
  --network "ix-frigate_default" \
  --network-alias "ix-frigate-frigate-1" \
  --network-alias "frigate" \
  --hostname "47f2d0e1a9dc" \
  --expose "5000/tcp" \
  --expose "8554/tcp" \
  --expose "8555/tcp" \
  --expose "8555/udp" \
  --expose "8971/tcp" \
  --user "0:0" \
  --env "UMASK_SET=002" \
  --env "NVIDIA_VISIBLE_DEVICES=void" \
  --env "UMASK=002" \
  --env "PATH=/usr/lib/btbn-ffmpeg/bin:/usr/local/go2rtc/bin:/usr/local/tempio/bin:/usr/local/nginx/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" \
  --env "NVIDIA_DRIVER_CAPABILITIES=compute,video,utility" \
  --env "S6_LOGGING_SCRIPT=T 1 n0 s10000000 T" \
  --env "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0" \
  --label "com.docker.compose.config-hash"="de65ff0e954bbd5822efc2c61d37634f6ec01b7236fe6fe1e79b86bed12c9a0e" \
  --label "com.docker.compose.container-number"="1" \
  --label "com.docker.compose.depends_on"="" \
  --label "com.docker.compose.image"="sha256:d1502760b3b1eb627b63bb4f7f44b786228930d98403a12852f9708f1a6fd4c0" \
  --label "com.docker.compose.oneoff"="False" \
  --label "com.docker.compose.project"="ix-frigate" \
  --label "com.docker.compose.project.config_files"="/mnt/.ix-apps/app_configs/frigate/versions/1.1.12/templates/rendered/docker-compose.yaml" \
  --label "com.docker.compose.project.working_dir"="/mnt/.ix-apps/app_configs/frigate/versions/1.1.12/templates/rendered" \
  --label "com.docker.compose.service"="frigate" \
  --label "com.docker.compose.version"="2.29.1" \
  --attach stdout \
  --attach stderr \
  --entrypoint "/init" \
  "ghcr.io/blakeblackshear/frigate:0.14.1" \

Relevant Frigate log output

2025-01-05 06:05:12.172415+00:002025-01-05 06:05:12.172203428  127.0.0.1 - - [05/Jan/2025:14:05:12 +0800] "GET /api HTTP/1.1" 301 169 "-" "curl/7.74.0" "-"
2025-01-05 06:05:22.441855+00:002025-01-05 06:05:22.441678992  Process detector:ov:
2025-01-05 06:05:22.441954+00:002025-01-05 06:05:22.441684985  Traceback (most recent call last):
2025-01-05 06:05:22.441970+00:002025-01-05 06:05:22.441686663    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2025-01-05 06:05:22.441982+00:002025-01-05 06:05:22.441687864      self.run()
2025-01-05 06:05:22.441994+00:002025-01-05 06:05:22.441689386    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2025-01-05 06:05:22.442006+00:002025-01-05 06:05:22.441690742      self._target(*self._args, **self._kwargs)
2025-01-05 06:05:22.442018+00:002025-01-05 06:05:22.441692200    File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
2025-01-05 06:05:22.442032+00:002025-01-05 06:05:22.441693419      detections = object_detector.detect_raw(input_frame)
2025-01-05 06:05:22.442044+00:002025-01-05 06:05:22.441694783    File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
2025-01-05 06:05:22.442058+00:002025-01-05 06:05:22.441696195      return self.detect_api.detect_raw(tensor_input=tensor_input)
2025-01-05 06:05:22.442071+00:002025-01-05 06:05:22.441697581    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 160, in detect_raw
2025-01-05 06:05:22.442085+00:002025-01-05 06:05:22.441698694      infer_request.infer(input_tensor)
2025-01-05 06:05:22.442097+00:002025-01-05 06:05:22.441700126    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 132, in infer
2025-01-05 06:05:22.442110+00:002025-01-05 06:05:22.441701350      return OVDict(super().infer(_data_dispatch(
2025-01-05 06:05:22.442122+00:002025-01-05 06:05:22.441702684  RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:223:
2025-01-05 06:05:22.442133+00:002025-01-05 06:05:22.441704027  Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_event.cpp:56:
2025-01-05 06:05:22.442145+00:002025-01-05 06:05:22.441705217  [GPU] clWaitForEvents, error code: -14

Install method

Docker CLI

Object Detector

OpenVino

Screenshots of the Frigate UI's System metrics pages

Any other information that may be helpful

No response

[NickM-27]

This may be improved in 0.15 with the newer openvino version

[daviddreher2]

I updated to the 0.15.0 beta5 and still have the same issue, not sure if everything planned has been updated for the beta yet