Failed to use Tensorrt as execution provider on Jetson with built onnxruntime from source

[jose-rincon]

Hello, we are trying to use your ort rust bindings on a jetson Xavier (Jetpack 5.5.1 with cuda11.4). We built the shared libraries as well as the python wheel from source.

We can run our models using the python wheel with tensorrt as execution provider and we get a performance of 21 fps in the model. However when using the ort-rust binding we can only use CUDA as the execution provider and our model's fps drops to 5 fps. We tried two strategies.

1. First strategy

• Environment variables:

export ORT_STRATEGY=load-dynamic
export ORT_DYLIB_PATH=/data/onnxruntime/build/Linux/Release/libonnxruntime.so.1.15.0

We also added "load-dynamic" as a feature for ort-rust in Cargo.toml

• Warnings

2023-10-05T13:10:37.585595Z  WARN ort: ort 1.14 may have compatibility issues with the ONNX Runtime binary found at `/data/onnxruntime/build/Linux/Release/libonnxruntime.so.1.15.0`; expected GetVersionString to return '1.14.x', but got '1.15.0'
[src/main.rs:19] tensorrt.is_available() = true
[src/main.rs:22] cuda.is_available() = true
2023-10-05T13:10:37.608202Z  WARN apply_execution_providers: ort::execution_providers: An error occurred when attempting to register `TensorrtExecutionProvider`: key/value cannot be empty
2023-10-05T13:10:38.018700Z  INFO apply_execution_providers: ort::execution_providers: Successfully registered `CUDAExecutionProvider`

So even though tensorrt.is_available() = true, we failed at attempting to register Tensorrt as execution provider.

We also tried to use the system strategy. We installed the dynamic libraries with make in /usr/local/lib. However, buikding onnxruntime does not produce the onnxruntime.a static library
2. Second strategy

• Environment variables

export ORT_STRATEGY=system
export ORT_LIB_LOCATION /usr/local/lib

• Warnings/errors

error: could not find native static library `onnx`, perhaps an -L flag is missing?
The following warnings were emitted during compilation:

warning: Provider not available for this strategy and/or target: CUDA
warning: Provider not available for this strategy and/or target: TENSORRT

Do you guys have any suggestions as to how to make onnxruntime work with tensorrt execution provider?

[decahedron1]

The TensorRT issue was fixed recently. It should be working with an update to ort 1.15.5 or 1.16.0.