Skip to content

fix: support TensorRT older than 10.15 - #4468

Open
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:trt-version-guard
Open

fix: support TensorRT older than 10.15#4468
shoumikhin wants to merge 2 commits into
pytorch:mainfrom
shoumikhin:trt-version-guard

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

The problem

Two places in the runtime call TensorRT APIs that only exist in TensorRT 10.15 and newer,
without checking first. pyproject.toml declares a bare tensorrt dependency with no
minimum version, so an older but otherwise usable TensorRT fails in confusing ways instead
of reporting an unmet requirement.

This matters for platforms pinned to an older TensorRT. Some embedded and Jetson-class
configurations ship 10.13 and have no newer build available.

1. The C++ runtime does not compile.

core/runtime/TRTEngine.cpp:298: error: 'class nvinfer1::ICudaEngine'
    has no member named 'getAliasedInputTensor'

2. Exporting a model with a KV cache raises at export time.

AttributeError: module 'tensorrt' has no attribute 'KVCacheMode'

The second is the worse of the two. A compile error at least says plainly that the
toolchain is too old. This one appears in the middle of an export and reads like a bug in
the user's own model.

Why they are reachable

ICudaEngine::getAliasedInputTensor, INetworkDefinition::add_kv_cache_update and
KVCacheMode all arrived in TensorRT 10.15, together with the IKVCacheUpdateLayer that
produces the aliasing they describe.

For the export path, the validator that decides whether the KV fast path applies checks
only graph shape (rank, static dimensions, batch size, write dimension, single position).
Nothing in it consults the TensorRT version, so a qualifying graph reaches the emitter on
any version and then calls a name that is not there.

The fix

C++ side. One direct version check at the point of use, matching how the ScatterAdd
plugin include is gated in core/plugins/register_plugins.cpp:

  this->aliased_io = aliased_io;
// ICudaEngine::getAliasedInputTensor is not available before TRT 10.15 (e.g. Jetpack
// L4T builds), where an engine cannot report its aliasing and the build-time map is
// the only source of truth.
#if NV_TENSORRT_MAJOR > 10 || (NV_TENSORRT_MAJOR == 10 && NV_TENSORRT_MINOR >= 15)
  for (const auto& out_name : this->out_binding_names) {
    ...
  }
#endif

The loop only reconciles the build-time alias map against what the engine reports. On a
TensorRT that cannot report aliasing there is nothing to reconcile against, so the
build-time map is already the only source of truth.

Python side. emit_kv_cache_update_layer already returns None to mean "fast path
unavailable", and its callers fall back to the general scatter, so the guard reuses that
existing contract rather than adding a new one:

    if not hasattr(ctx.net, "add_kv_cache_update") or not hasattr(trt, "KVCacheMode"):
        logger.debug(
            "KV cache update: skipped — this TensorRT build has no KV-cache update layer"
        )
        return None

An older TensorRT gets the slower general scatter, which is correct, instead of an
exception.

Behavior on TensorRT 10.15 and newer is unchanged in both cases. The whole change is 14
added lines across two files, with no deletions.

Testing

Compiled the guarded C++ region against every TensorRT version available to me:

TensorRT 10.3   exit=0
TensorRT 10.13  exit=0     (fails without this change, with the error above)
TensorRT 10.15  exit=0
TensorRT 10.16  exit=0
TensorRT 11.1   exit=0

The control matters: with the check removed, 10.13 reproduces the original
has no member named 'getAliasedInputTensor' error, which confirms the gate is what fixes
the build rather than an unrelated include change. Also confirmed on an aarch64
Jetson-class device with system TensorRT 10.13.3.9, where the symbol is absent from the
installed headers.

For the Python guard, exercised the emitter both ways:

current TensorRT (11.1), layer available -> add_kv_cache_update is still called
simulated older build, layer absent      -> returns None, caller falls back

Existing tests: tests/py/dynamo/lowering/test_buffer_lifting.py passes, and
tests/py/dynamo/executorch/ gives the same result with and without this change (the
failures there reproduce on the unmodified branch, so they are pre-existing and unrelated).

A note on the declared dependency

If the project intends to require TensorRT 10.15 or newer regardless, declaring that floor
in pyproject.toml would turn both of these into an explicit dependency error. That felt
like a separate decision, so this change only makes the code work on what it can actually
support. Happy to add the floor instead if that is the preference.

@meta-cla meta-cla Bot added the cla signed label Aug 8, 2026
@github-actions github-actions Bot added component: core Issues re: The core compiler component: runtime labels Aug 8, 2026
@github-actions
github-actions Bot requested a review from narendasan August 8, 2026 21:45
Building Torch-TensorRT against TensorRT 10.13 or older fails to compile:

  core/runtime/TRTEngine.cpp:298: error: 'class nvinfer1::ICudaEngine'
      has no member named 'getAliasedInputTensor'

That API was added in TensorRT 10.15, together with the IKVCacheUpdateLayer
that produces the aliasing it reports. The runtime calls it unconditionally,
while the package declares a bare tensorrt dependency with no minimum version,
so an older TensorRT produces a compile error rather than a clear message.

The call only reconciles the build-time aliased I/O map against what the engine
reports. On older TensorRT there is nothing to reconcile against, so the
build-time map stands on its own and the reconciliation is skipped.

Gate it with a direct version check at the point of use, matching how the
ScatterAdd plugin include is gated in core/plugins/register_plugins.cpp.
@github-actions github-actions Bot added component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Aug 8, 2026
@shoumikhin shoumikhin changed the title fix: build against TensorRT older than 10.15 fix: support TensorRT older than 10.15 Aug 8, 2026
Exporting a model with a KV cache on TensorRT 10.13 or older fails with:

  AttributeError: module 'tensorrt' has no attribute 'KVCacheMode'

add_kv_cache_update and KVCacheMode were added in TensorRT 10.15. The KV fast
path calls both unconditionally, and the validator that gates it checks only
graph shape, so a qualifying graph reaches the call on any TensorRT version.

This one is worse than a compile error: it surfaces during export and reads like
a bug in the user's model rather than a version gap.

emit_kv_cache_update_layer already returns None to mean "fast path unavailable",
and its callers fall back to the general scatter, so check for the API and take
that existing path instead of raising.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: runtime

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant