onnxruntime running in Fallback mode

[VMM-MMV]

I try to ocr some pdfs on the gpu, but I'm getting such warnings:

2024-12-18 14:17:49.573594425 [W:onnxruntime:Default, conv.cc:425 UpdateState] OP Conv(/feat_extractor/feat_extractor.0/Conv) running in Fallback mode. May be extremely slow.
2024-12-18 14:17:49.682653080 [W:onnxruntime:Default, conv.cc:425 UpdateState] OP Conv(/feat_extractor/feat_extractor.3/feat_extractor.3.0/fused_conv/Conv) running in Fallback mode. May be extremely slow.
2024-12-18 14:17:49.683018957 [W:onnxruntime:Default, conv.cc:425 UpdateState] OP Conv(/feat_extractor/feat_extractor.3/feat_extractor.3.1/fused_conv/Conv) running in Fallback mode. May be extremely slow.
2024-12-18 14:17:49.683212080 [W:onnxruntime:Default, conv.cc:425 UpdateState] OP Conv(/feat_extractor/feat_extractor.3/feat_extractor.3.2/fused_conv/Conv) running in Fallback mode. May be extremely slow.

I have tried to set the provider to only gpu

providers = ['CUDAExecutionProvider']
session = ort.InferenceSession(model_path, sess_options=options, providers=providers)

But I'm getting:

2024-12-18 14:17:49.087771928 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.

Here's the verbose log
https://pastebin.com/cBZMMWpB

Some additional info:

serveruser@server:~/groq$ nvcc --version
        nvcc: NVIDIA (R) Cuda compiler driver
        Copyright (c) 2005-2023 NVIDIA Corporation
        Built on Fri_Jan__6_16:45:21_PST_2023
        Cuda compilation tools, release 12.0, V12.0.140
        Build cuda_12.0.r12.0/compiler.32267302_0

serveruser@server:~/groq$ ldconfig -p | grep libcublasLt.so
        libcublasLt.so.12 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcublasLt.so.12
        libcublasLt.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcublasLt.so
        
serveruser@server:~/groq$ locate libcudnn.so
      /home/serveruser/anaconda3/envs/class/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn.so.9
      /home/serveruser/anaconda3/envs/classify/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn.so.9

[felixdittrich92]

Hi again @VMM-MMV :)

Yeah some nodes will be fallback to CPU but in the most cases this hasn't any impact in this case:
[CPUExecutionProvider]. Number of nodes: 18
[CUDAExecutionProvider]. Number of nodes: 90

For example some operations like unsqueeze, gather and so on will be executed on CPU because that's small computations where it would take more time to move to CUDA and detach to CPU afterwards instead of keeping on CPU

All compute intensive operations like convolutions are moved and computed on the GPU

The layers which are executed on CPU and which ones on GPU is also in your shared log file :)

So the CPU fallback is required for some operations, additional the default configuration has some additional parameters to get the best inference latency

Overview

Running on CPU:

2024-12-18 14:17:49.087206481 [V:onnxruntime:, session_state.cc:1148 VerifyEachNodeIsAssignedToAnEp] Node placements
2024-12-18 14:17:49.087213111 [V:onnxruntime:, session_state.cc:1154 VerifyEachNodeIsAssignedToAnEp]  Node(s) placed on [CPUExecutionProvider]. Number of nodes: 18
2024-12-18 14:17:49.087218550 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Slice (/neck/Slice)
2024-12-18 14:17:49.087224285 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Concat (/neck/Concat_1)
2024-12-18 14:17:49.087228906 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Slice (/neck/Slice_1)
2024-12-18 14:17:49.087233540 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Concat (/neck/Concat_3)
2024-12-18 14:17:49.087238726 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Gather (/neck/Gather_4)
2024-12-18 14:17:49.087243458 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Gather (/neck/Gather_5)
2024-12-18 14:17:49.087249633 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Unsqueeze (/neck/Unsqueeze_4)
2024-12-18 14:17:49.087254363 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Unsqueeze (/neck/Unsqueeze_5)
2024-12-18 14:17:49.087259111 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Concat (/neck/Concat_4)
2024-12-18 14:17:49.087263808 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Slice (/neck/Slice_2)
2024-12-18 14:17:49.087276802 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Concat (/neck/Concat_5)
2024-12-18 14:17:49.087282526 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Gather (/Gather)
2024-12-18 14:17:49.087287160 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Gather (/Gather_1)
2024-12-18 14:17:49.087291969 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Unsqueeze (/Unsqueeze)
2024-12-18 14:17:49.087297065 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Unsqueeze (/Unsqueeze_1)
2024-12-18 14:17:49.087302060 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Concat (/Concat)
2024-12-18 14:17:49.087306663 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Slice (/Slice)
2024-12-18 14:17:49.087311760 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Concat (/Concat_1)
2024-12-18 14:17:49.087316591 [V:onnxruntime:, session_state.cc:1154 VerifyEachNodeIsAssignedToAnEp]  Node(s) placed on

Running on GPU:

2024-12-18 14:17:49.087321542 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Conv (/feat_extractor/feat_extractor.0/Conv)
2024-12-18 14:17:49.087326554 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Relu (/feat_extractor/feat_extractor.2/Relu)
2024-12-18 14:17:49.087331345 [V:onnxruntime:, session_state.cc:1156 VerifyEachNodeIsAssignedToAnEp]   Conv (/feat_extractor/feat_extractor.3/feat_extractor.3.0/fused_conv/Conv)

....

[felixdittrich92]

If you process multiple files you should play a bit around with det_bs and reco_bs values depending on your hardware (especially GPU VRAM)

[felixdittrich92]

Another option would be (depending on your GPU architecture) to use TensorRT as execution provider I haven't had the chance yet to test it but this should boost the inference latency again multiple times

[VMM-MMV]

Thank You! That is quite weird though, because I switched from docTR to onnxTR, and I got a 15% decrease in processing. So probably I am doing something wrong.

[felixdittrich92]

Hi @VMM-MMV 👋,

Do you use the default EngineConfig or your own ?

[VMM-MMV]

The default