LocalAI version:
v4.1.3 (fdc9f7b)
Environment, CPU architecture, OS, and Version:
uname -a
Linux localai 5.10.216-tegra #1 SMP PREEMPT Mon Jan 26 23:28:09 PST 2026 aarch64 aarch64 aarch64 GNU/Linux
localai@localai:~$ deviceQuery
deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Xavier"
CUDA Driver Version / Runtime Version 12.2 / 11.4
CUDA Capability Major/Minor version number: 7.2
Total amount of global memory: 30991 MBytes (32496128000 bytes)
(008) Multiprocessors, (064) CUDA Cores/MP: 512 CUDA Cores
GPU Max Clock rate: 1377 MHz (1.38 GHz)
Memory Clock rate: 1377 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 98304 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.2, CUDA Runtime Version = 11.4, NumDevs = 1
Result = PASS
Describe the bug
LocalAi does not work on Jetson AGX Xavier with CUDA upgraded to 12.2 (to statisfy the CUDA-12 requirement) as described in: https://forums.developer.nvidia.com/t/cuda-12-2-update-for-jetpack-5-1-2/316621, due to incompatible PTX. From what I understand this is due to we can upgrade the CUDA Driver Version, but not the CUDA Runtime Version.
To Reproduce
Try to use any model on AGX Xavier.
Expected behavior
Should work if the device is listed as compatible.
Logs
22:06:18.474
stderr
/LocalAI/backend/cpp/llama-cpp-fallback-build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
22:06:18.474
stderr
ggml_cuda_compute_forward: MUL_MAT failed
22:06:18.474
stderr
CUDA error: the provided PTX was compiled with an unsupported toolchain.
22:06:18.474
stderr
current device: 0, in function ggml_cuda_compute_forward at /LocalAI/backend/cpp/llama-cpp-fallback-build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2962
Additional context
Most probably Jetson AGX Xavier is just not supported at all, despite the CUDA upgrade attempt. Maybe it should just be removed from the supported platform list on https://localai.io/model-compatibility/index.print
There below Specialized Hardware it is listed:
NVIDIA Jetson (L4T CUDA 12): ARM64 support for embedded AI (AGX Orin, Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)
LocalAI version:
v4.1.3 (fdc9f7b)
Environment, CPU architecture, OS, and Version:
uname -a
Linux localai 5.10.216-tegra #1 SMP PREEMPT Mon Jan 26 23:28:09 PST 2026 aarch64 aarch64 aarch64 GNU/Linux
localai@localai:~$ deviceQuery
deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "Xavier"
CUDA Driver Version / Runtime Version 12.2 / 11.4
CUDA Capability Major/Minor version number: 7.2
Total amount of global memory: 30991 MBytes (32496128000 bytes)
(008) Multiprocessors, (064) CUDA Cores/MP: 512 CUDA Cores
GPU Max Clock rate: 1377 MHz (1.38 GHz)
Memory Clock rate: 1377 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 524288 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total shared memory per multiprocessor: 98304 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: Yes
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device supports Managed Memory: Yes
Device supports Compute Preemption: Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 12.2, CUDA Runtime Version = 11.4, NumDevs = 1
Result = PASS
Describe the bug
LocalAi does not work on Jetson AGX Xavier with CUDA upgraded to 12.2 (to statisfy the CUDA-12 requirement) as described in: https://forums.developer.nvidia.com/t/cuda-12-2-update-for-jetpack-5-1-2/316621, due to incompatible PTX. From what I understand this is due to we can upgrade the CUDA Driver Version, but not the CUDA Runtime Version.
To Reproduce
Try to use any model on AGX Xavier.
Expected behavior
Should work if the device is listed as compatible.
Logs
22:06:18.474
stderr
/LocalAI/backend/cpp/llama-cpp-fallback-build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:97: CUDA error
22:06:18.474
stderr
ggml_cuda_compute_forward: MUL_MAT failed
22:06:18.474
stderr
CUDA error: the provided PTX was compiled with an unsupported toolchain.
22:06:18.474
stderr
current device: 0, in function ggml_cuda_compute_forward at /LocalAI/backend/cpp/llama-cpp-fallback-build/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2962
Additional context
Most probably Jetson AGX Xavier is just not supported at all, despite the CUDA upgrade attempt. Maybe it should just be removed from the supported platform list on https://localai.io/model-compatibility/index.print
There below Specialized Hardware it is listed:
NVIDIA Jetson (L4T CUDA 12): ARM64 support for embedded AI (AGX Orin, Jetson Nano, Jetson Xavier NX, Jetson AGX Xavier)