TensorRT trying to load Windows DSO on Ubuntu?

[thefoxfarmer]

I have installed the Comfy node and its requirements in the venv I'm using, Python 3.10 on Ubuntu 22. When I attempt to convert a model, it gets part of the way through before failing because it cannot load a DSO that, based on the name ending in "_win" and me having one by the same name except without "_win", I conclude is a Windows DSO.

[TRT] [E] IBuilder::buildSerializedNetwork: Error Code 6: API Usage Error (Unable to load library: libnvinfer_builder_resource_win.so.10.2.0: libnvinfer_builder_resource_win.so.10.2.0: cannot open shared object file: No such file or directory)

I tried to fake it out by making a symlink from libnvinfer_builder_resource_win.so.10.2.0 to libnvinfer_builder_resource.so.10.2.0 but it the symbols themselves are also suffixed.

[TRT] [E] IBuilder::buildSerializedNetwork: Error Code 3: API Usage Error (SymbolAddress for getNbCaskKLibs_INTERNAL_WIN could not be loaded, check function name against library symbol)

[thefoxfarmer]

Fwiw, here are my pip installed packages in the venv.

Package                   Version
------------------------- ------------
aiohttp                   3.9.5
aiosignal                 1.3.1
albucore                  0.0.12
albumentations            1.4.10
annotated-types           0.7.0
async-timeout             4.0.3
attrs                     23.2.0
certifi                   2024.7.4
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
cmake                     3.30.0
coloredlogs               15.0.1
contourpy                 1.2.1
cryptography              42.0.8
cstr                      0.1.0
cycler                    0.12.1
Cython                    3.0.10
Deprecated                1.2.14
dynamicprompts            0.30.2
easydict                  1.13
einops                    0.8.0
fairscale                 0.4.13
ffmpy                     0.3.0
filelock                  3.15.4
flatbuffers               24.3.25
fonttools                 4.53.0
frozenlist                1.4.1
fsspec                    2024.6.1
gitdb                     4.0.11
GitPython                 3.1.43
huggingface-hub           0.23.4
humanfriendly             10.0
idna                      3.7
imageio                   2.34.2
img2texture               1.0.6
insightface               0.7.3
Jinja2                    3.1.4
joblib                    1.4.2
jsonschema                4.22.0
jsonschema-specifications 2023.12.1
kiwisolver                1.4.5
kornia                    0.7.3
kornia_rs                 0.1.4
lazy_loader               0.4
llvmlite                  0.43.0
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.9.1
matrix-client             0.4.0
mdurl                     0.1.2
mpmath                    1.3.0
multidict                 6.0.5
natsort                   8.4.0
networkx                  3.3
numba                     0.60.0
numpy                     1.26.4
nvidia-cublas-cu12        12.1.3.1
nvidia-cuda-cupti-cu12    12.1.105
nvidia-cuda-nvrtc-cu12    12.1.105
nvidia-cuda-runtime-cu12  12.1.105
nvidia-cudnn-cu12         8.9.2.26
nvidia-cufft-cu12         11.0.2.54
nvidia-curand-cu12        10.3.2.106
nvidia-cusolver-cu12      11.4.5.107
nvidia-cusparse-cu12      12.1.0.106
nvidia-nccl-cu12          2.20.5
nvidia-nvjitlink-cu12     12.5.82
nvidia-nvtx-cu12          12.1.105
onnx                      1.16.1
onnxruntime               1.18.1
opencv-python             4.10.0.84
opencv-python-headless    4.10.0.84
packaging                 24.1
pandas                    2.2.2
piexif                    1.1.3
pilgram                   1.2.1
Pillow                    9.5.0
pip                       22.0.2
platformdirs              4.2.2
pooch                     1.8.2
prettytable               3.10.0
protobuf                  5.27.2
psutil                    6.0.0
py-cpuinfo                9.0.0
pycparser                 2.22
pydantic                  2.8.2
pydantic_core             2.20.1
PyGithub                  2.3.0
Pygments                  2.18.0
PyJWT                     2.8.0
PyMatting                 1.1.12
PyNaCl                    1.5.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
pytz                      2024.1
PyYAML                    6.0.1
referencing               0.35.1
regex                     2024.5.15
rembg                     2.0.57
requests                  2.32.3
rich                      13.7.1
rpds-py                   0.18.1
safetensors               0.4.3
scikit-image              0.24.0
scikit-learn              1.5.1
scipy                     1.14.0
seaborn                   0.13.2
segment-anything          1.0
setuptools                59.6.0
shellingham               1.5.4
six                       1.16.0
smmap                     5.0.1
soundfile                 0.12.1
spandrel                  0.3.4
sympy                     1.12.1
tensorrt                  10.2.0
tensorrt-cu12             10.2.0
tensorrt-cu12-bindings    10.2.0
tensorrt-cu12-libs        10.2.0
threadpoolctl             3.5.0
tifffile                  2024.7.2
timm                      1.0.7
tokenizers                0.19.1
tomli                     2.0.1
torch                     2.3.1+cu121
torchaudio                2.3.1+cu121
torchsde                  0.2.6
torchvision               0.18.1+cu121
tqdm                      4.66.4
trampoline                0.1.2
transformers              4.42.3
triton                    2.3.1
typer                     0.12.3
typing_extensions         4.12.2
tzdata                    2024.1
ultralytics               8.2.48
ultralytics-thop          2.0.0
urllib3                   1.26.19
wcwidth                   0.2.13
wrapt                     1.16.0
yarl                      1.9.4                                                                                                                                                                                                          ```

[thefoxfarmer]

Using the backtrace, I found that this error is coming from TRT_MODEL_CONVERSION_BASE._convert. It creates a tensorrt.Builder object as such:

        # TRT conversion starts here
        logger = trt.Logger(trt.Logger.INFO)
        builder = trt.Builder(logger)

I changed INFO to VERBOSE and then saw this message just before the failure occurs:

[07/04/2024-14:53:27] [TRT] [V] Trying to load shared library libnvinfer_builder_resource_win.so.10.2.0

Grepping my entire venv for the string Trying to load shared library turns up only one file: lib/python3.10/site-packages/tensorrt_libs/libnvinfer.so.10. That file is also the only file in the venv that contains the string libnvinfer_builder_resource_win.so. It also includes the string libnvinfer_builder_resource.so. So...my best guess so far is that somehow this TensorRT DSO is mistaken about which platform it is running on.

[thefoxfarmer]

I inserted os.kill(os.getpid(), signal.SIGSTOP) after the creation of the trt.Builder so that I could run lsof on it at that point.

pt_main_t 41348    ff mem       REG              259,2 1379530192  99235669 /home/ff/ComfyUI/venv/lib/python3.10/site-packages/tensorrt_libs/libnvinfer_builder_resource.so.10.2.0

So the non-_win form of this library is already open when (or at least very shortly before) it tries to open the one whose name ends in _win.

[thefoxfarmer]

This is happening inside tensorrt.Builder.build_serialized_network, which is in the compiled tensorrt.so file. Clearly not a Comfy-related issue. Furthermore, others—at least one non-Comfy user—are reporting this.
NVIDIA/TensorRT#3983

[thefoxfarmer]

This appears to be a problem with tensorrt 10.2.0. Downgrading to tensorrt-cu12==10.1.0 tensorrt-cu12-bindings==10.1.0 tensorrt-cu12-libs==10.1.0 resolved the issue for me.