Possible regression in Larod model compatibility in Axis OS > 10.12.270

[JsGjKJzi]

I am working on a Tensorflow Lite model to run on Axis cameras, using the Artpec 8 DLPU of the P1468-LE as a development target. I am successfully able to follow the tensorflow-to-larod-artpec-8 example to convert my model to tflite format.

However, when attempting to run it on the camera with Axis OS 11.11.73 I get the following error Error: Could not run job: Failure when invoking interpreter.

Digging into the logs a little bit yields:

root@axis-b8a44f9ebaea:~# journalctl
...
Feb 11 15:45:39 axis-b8a44f9ebaea lldpcli[1474]: system name set to new value axis-b8a44f9ebaea
Feb 11 15:45:39 axis-b8a44f9ebaea lldpcli[1474]: description set to new value AXIS P1468-LE Bullet Camera 11.11.73
...
root@axis-b8a44f9ebaea:~# journalctl -u larod
...
Feb 11 15:47:13 axis-b8a44f9ebaea systemd[1]: Starting Machine learning service...
Feb 11 15:47:13 axis-b8a44f9ebaea systemd[1]: Started Machine learning service.
Feb 11 15:47:13 axis-b8a44f9ebaea larod[2364]: Created a new session ID: 0, client: :1.473
Feb 11 15:47:13 axis-b8a44f9ebaea larod[2364]: Service started
Feb 11 15:47:18 axis-b8a44f9ebaea sh[2364]: ERROR: Failed to run graph
Feb 11 15:47:18 axis-b8a44f9ebaea larod[2364]: Node number 36 (Vx Delegate) failed to invoke.
Feb 11 15:47:18 axis-b8a44f9ebaea larod[2364]: Session 0: Could not load model: Unable to run warmup job: Failure when invoking interpreter
Feb 11 15:47:18 axis-b8a44f9ebaea larod[2364]: Could not send delete model result: Session has expired
Feb 11 15:47:18 axis-b8a44f9ebaea larod[2364]: Session 0 killed since client's (:1.473) connection has been lost
Feb 11 15:47:49 axis-b8a44f9ebaea larod[2364]: Stopping service due to inactivity
Feb 11 15:47:49 axis-b8a44f9ebaea larod[2364]: Service stopped
root@axis-b8a44f9ebaea:~# ls /lib | grep larod
liblarod.so.1
liblarod.so.3.2.181
liblarodremote.so.1
liblarodremote.so.3.2.181

Utilizing the included larod-client yields similar results the following result:

root@axis-b8a44f9ebaea:~# larod-client -g frozen_quantized_model.tflite -c  axis-a8-dlpu-tflite
2025-02-12T10:06:28.051 Connecting to larod...
2025-02-12T10:06:28.061 Connected
2025-02-12T10:06:30.083 ERROR: When loading model synchronously (-6): Could not load model: Asynchronous connection has been closed
root@axis-b8a44f9ebaea:~# journalctl 
...
Feb 12 10:05:59 axis-b8a44f9ebaea dbus-daemon[585]: [system] Activating via systemd: service name='com.axis.Larod1' unit='dbus-com.axis.Larod1.service' requested by ':1.506' (uid=0 pid=150165 comm="larod-client -l")
Feb 12 10:05:59 axis-b8a44f9ebaea systemd[1]: Starting Machine learning service...
Feb 12 10:05:59 axis-b8a44f9ebaea dbus-daemon[585]: [system] Successfully activated service 'com.axis.Larod1'
Feb 12 10:05:59 axis-b8a44f9ebaea systemd[1]: Started Machine learning service.
Feb 12 10:05:59 axis-b8a44f9ebaea larod[150166]: Created a new session ID: 0, client: :1.506
Feb 12 10:05:59 axis-b8a44f9ebaea larod[150166]: Service started
Feb 12 10:05:59 axis-b8a44f9ebaea larod[150166]: Session 0 killed since client's (:1.506) connection has been lost
Feb 12 10:06:08 axis-b8a44f9ebaea larod[150166]: Created a new session ID: 1, client: :1.508
Feb 12 10:06:08 axis-b8a44f9ebaea larod[150166]: Session 1 killed since client's (:1.508) connection has been lost
Feb 12 10:06:28 axis-b8a44f9ebaea larod[150166]: Created a new session ID: 2, client: :1.509
Feb 12 10:06:29 axis-b8a44f9ebaea kernel: Adding 32764k swap on /mnt/flash/var/lib/larod-swap/swap.  Priority:0 extents:1 across:32764k SS
Feb 12 10:06:30 axis-b8a44f9ebaea sh[150166]: double free or corruption (out)
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: larod: singleprocq: potentially unexpected fatal signal 6.
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: CPU: 2 PID: 150171 Comm: singleprocq Kdump: loaded Tainted: G           O      5.15.13-axis9 #1
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: Hardware name: AXIS P1468-LE Bullet Camera (DT)
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: pstate: 60000000 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: pc : 0000007f9a3f2988
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: lr : 0000007f9a3f2974
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: sp : 0000007f97efb070
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x29: 0000007f97efb070 x28: 0000007f97efb7f8 x27: 0000007f97efb4f8
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x26: 0000000000000000 x25: 0000000000000020 x24: 0000007f97efb240
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x23: 0000007f97efb240 x22: 0000007f97efb240 x21: 0000000000000006
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x20: 0000007f9a500a10 x19: 0000000000024a9b x18: 00000000f00acf3f
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 0000000029f5f437
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000028
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x11: 0101010101010101 x10: 000000000000000a x9 : 0000007f97efb400
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x8 : 0000000000000083 x7 : 048c9b225e57f67e x6 : 0000000000000020
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x5 : 0000000000000001 x4 : 0000007f9a500a10 x3 : 0000007f97efd080
Feb 12 10:06:30 axis-b8a44f9ebaea kernel: x2 : 0000000000000006 x1 : 0000000000024a9b x0 : 0000000000000000
Feb 12 10:06:30 axis-b8a44f9ebaea dbg-cgi[1310]: Core dump ID: axis-b8a44f9ebaea_1739372790_150166.core
Feb 12 10:06:30 axis-b8a44f9ebaea systemd[1]: larod.service: Main process exited, code=killed, status=6/ABRT
Feb 12 10:06:30 axis-b8a44f9ebaea systemd[1]: larod.service: Failed with result 'signal'.
Feb 12 10:06:30 axis-b8a44f9ebaea systemd[1]: larod.service: Scheduled restart job, restart counter is at 1.
Feb 12 10:06:30 axis-b8a44f9ebaea systemd[1]: Starting Machine learning service...
Feb 12 10:06:30 axis-b8a44f9ebaea systemd[1]: Started Machine learning service.
Feb 12 10:06:30 axis-b8a44f9ebaea systemd[1]: mnt-flash-var-lib-larod\x2dswap-swap.swap: Deactivated successfully.

I upgraded to Axis OS 11.11.124 with similar results:

Feb 12 10:41:23 axis-b8a44f9ebaea dbus-daemon[693]: [system] Activating via systemd: service name='com.axis.Larod1' unit='dbus-com.axis.Larod1.service' requested by ':1.636' (uid=0 pid=6284 comm="/tmp/oxblue_axis_ai")
Feb 12 10:41:23 axis-b8a44f9ebaea systemd[1]: Starting Machine learning service...
Feb 12 10:41:23 axis-b8a44f9ebaea dbus-daemon[693]: [system] Successfully activated service 'com.axis.Larod1'
Feb 12 10:41:23 axis-b8a44f9ebaea systemd[1]: Started Machine learning service.
Feb 12 10:41:23 axis-b8a44f9ebaea larod[6285]: Created a new session ID: 0, client: :1.636
Feb 12 10:41:23 axis-b8a44f9ebaea larod[6285]: Service started
Feb 12 10:41:25 axis-b8a44f9ebaea kernel: Adding 32764k swap on /mnt/flash/var/lib/larod-swap/swap.  Priority:0 extents:1 across:32764k SS
Feb 12 10:41:25 axis-b8a44f9ebaea systemd[1]: dev-zram0.swap: Deactivated successfully.
Feb 12 10:41:47 axis-b8a44f9ebaea systemd[1]: mnt-flash-var-lib-larod\x2dswap-swap.swap: Deactivated successfully.
Feb 12 10:41:50 axis-b8a44f9ebaea kernel: [galcore]: GPU[1] hang, automatic recovery.
Feb 12 10:41:50 axis-b8a44f9ebaea kernel: [galcore]: recovery done
Feb 12 10:41:50 axis-b8a44f9ebaea sh[6285]: ERROR: Failed to run graph
Feb 12 10:41:50 axis-b8a44f9ebaea larod[6285]: Node number 36 (Vx Delegate) failed to invoke.
Feb 12 10:41:50 axis-b8a44f9ebaea larod[6285]: Session 0: Could not load model: Unable to run warmup job: Failure when invoking interpreter
Feb 12 10:41:50 axis-b8a44f9ebaea larod[6285]: Could not send delete model result: Session has expired
Feb 12 10:41:50 axis-b8a44f9ebaea larod[6285]: Session 0 killed since client's (:1.636) connection has been lost

I did not get the liblarod version from 11.11.124.

I then downgraded to Axis OS 10.12.270 and everything looks fine

root@axis-b8a44f9ebaea:~# journalctl -u larod
-- Journal begins at Wed 2025-02-12 10:53:12 EST, ends at Wed 2025-02-12 11:17:03 EST. --
Feb 12 10:53:40 axis-b8a44f9ebaea systemd[1]: Starting Machine learning service...
Feb 12 10:53:41 axis-b8a44f9ebaea larod[1205]: Created NBG cache directory at "/var/lib/larod/nbg-cache"
Feb 12 10:53:41 axis-b8a44f9ebaea larod[1205]: Service started
Feb 12 10:53:41 axis-b8a44f9ebaea systemd[1]: Started Machine learning service.
Feb 12 10:57:04 axis-b8a44f9ebaea larod[1205]: Created a new session ID: 0, client: :1.517
Feb 12 10:57:05 axis-b8a44f9ebaea larod[1205]: W [HandleLayoutInfer:257]Op 18: default layout inference pass.
Feb 12 10:57:14 axis-b8a44f9ebaea larod[1205]: W [HandleLayoutInfer:257]Op 18: default layout inference pass.
Feb 12 10:57:16 axis-b8a44f9ebaea larod[1205]: Could not send delete model result: Session has expired
Feb 12 10:57:16 axis-b8a44f9ebaea larod[1205]: Session 0 killed since client's (:1.517) connection has been lost
Feb 12 10:57:16 axis-b8a44f9ebaea larod[1205]: Could not send delete model result: Session has expired
root@axis-b8a44f9ebaea:~# ls /usr/lib | grep larod
liblarod.so.1
liblarod.so.3.1.90

There are a few discussions on Tensorflow's Github page that seem partially relevant:

• TF ConvertedModel: Invoke fails with "Node number X (CONCATENATION) failed to prepare" error #46315
• TFLite GPU execution failed #35632
• [RNN] LSTM with TimeDistributed layer converts successfully but fails when invoking #46809

There is a similar discussion elsewhere on AxisCommunications Github is it possible to run a tflite model using artpec 7 gpu? #673

I would obviously like this to work on newer Axis OS versions. Could you please assist me in understanding what is failing here?

[github-actions[bot]]

This automatically generated reply acts as a friendly reminder.

Answers to your questions will most often come from the community, from developers like yourself. You will, from time to time, find that Axis employees answers some of the questions, but this is not a guarantee. Think of the discussion forum as a complement to other support channels, not a replacement to any of them.

If your question remains unanswered for a period of time, please revisit it to see whether it can be improved by following the guidelines listed in Axis support guidelines.

[vivekatoffice]

Hi @JsGjKJzi ,
If you can use the following details to test the model it will give more data to troubleshoot for Axis Developer Community members:

Test your model
computer-vision-on-device/optimization-tips#artpec-8

You can also upload the camera server report for complete logs:
How to download a server report from your Axis device - AXIS OS version 10.9 or later

AXIS Server Report Viewer
AXIS Server Report Viewer provides you with a graphical user interface tool to help you analyze the information provided in server reports from Axis devices.