fix: support TensorRT older than 10.15 - #4468
Open
shoumikhin wants to merge 2 commits into
Open
Conversation
Building Torch-TensorRT against TensorRT 10.13 or older fails to compile:
core/runtime/TRTEngine.cpp:298: error: 'class nvinfer1::ICudaEngine'
has no member named 'getAliasedInputTensor'
That API was added in TensorRT 10.15, together with the IKVCacheUpdateLayer
that produces the aliasing it reports. The runtime calls it unconditionally,
while the package declares a bare tensorrt dependency with no minimum version,
so an older TensorRT produces a compile error rather than a clear message.
The call only reconciles the build-time aliased I/O map against what the engine
reports. On older TensorRT there is nothing to reconcile against, so the
build-time map stands on its own and the reconciliation is skipped.
Gate it with a direct version check at the point of use, matching how the
ScatterAdd plugin include is gated in core/plugins/register_plugins.cpp.
shoumikhin
force-pushed
the
trt-version-guard
branch
from
August 8, 2026 21:52
9c1e299 to
80f2e04
Compare
Exporting a model with a KV cache on TensorRT 10.13 or older fails with: AttributeError: module 'tensorrt' has no attribute 'KVCacheMode' add_kv_cache_update and KVCacheMode were added in TensorRT 10.15. The KV fast path calls both unconditionally, and the validator that gates it checks only graph shape, so a qualifying graph reaches the call on any TensorRT version. This one is worse than a compile error: it surfaces during export and reads like a bug in the user's model rather than a version gap. emit_kv_cache_update_layer already returns None to mean "fast path unavailable", and its callers fall back to the general scatter, so check for the API and take that existing path instead of raising.
shoumikhin
force-pushed
the
trt-version-guard
branch
from
August 9, 2026 23:37
debf56a to
9bce699
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Two places in the runtime call TensorRT APIs that only exist in TensorRT 10.15 and newer,
without checking first.
pyproject.tomldeclares a baretensorrtdependency with nominimum version, so an older but otherwise usable TensorRT fails in confusing ways instead
of reporting an unmet requirement.
This matters for platforms pinned to an older TensorRT. Some embedded and Jetson-class
configurations ship 10.13 and have no newer build available.
1. The C++ runtime does not compile.
2. Exporting a model with a KV cache raises at export time.
The second is the worse of the two. A compile error at least says plainly that the
toolchain is too old. This one appears in the middle of an export and reads like a bug in
the user's own model.
Why they are reachable
ICudaEngine::getAliasedInputTensor,INetworkDefinition::add_kv_cache_updateandKVCacheModeall arrived in TensorRT 10.15, together with theIKVCacheUpdateLayerthatproduces the aliasing they describe.
For the export path, the validator that decides whether the KV fast path applies checks
only graph shape (rank, static dimensions, batch size, write dimension, single position).
Nothing in it consults the TensorRT version, so a qualifying graph reaches the emitter on
any version and then calls a name that is not there.
The fix
C++ side. One direct version check at the point of use, matching how the ScatterAdd
plugin include is gated in
core/plugins/register_plugins.cpp:The loop only reconciles the build-time alias map against what the engine reports. On a
TensorRT that cannot report aliasing there is nothing to reconcile against, so the
build-time map is already the only source of truth.
Python side.
emit_kv_cache_update_layeralready returnsNoneto mean "fast pathunavailable", and its callers fall back to the general scatter, so the guard reuses that
existing contract rather than adding a new one:
An older TensorRT gets the slower general scatter, which is correct, instead of an
exception.
Behavior on TensorRT 10.15 and newer is unchanged in both cases. The whole change is 14
added lines across two files, with no deletions.
Testing
Compiled the guarded C++ region against every TensorRT version available to me:
The control matters: with the check removed, 10.13 reproduces the original
has no member named 'getAliasedInputTensor'error, which confirms the gate is what fixesthe build rather than an unrelated include change. Also confirmed on an aarch64
Jetson-class device with system TensorRT 10.13.3.9, where the symbol is absent from the
installed headers.
For the Python guard, exercised the emitter both ways:
Existing tests:
tests/py/dynamo/lowering/test_buffer_lifting.pypasses, andtests/py/dynamo/executorch/gives the same result with and without this change (thefailures there reproduce on the unmodified branch, so they are pre-existing and unrelated).
A note on the declared dependency
If the project intends to require TensorRT 10.15 or newer regardless, declaring that floor
in
pyproject.tomlwould turn both of these into an explicit dependency error. That feltlike a separate decision, so this change only makes the code work on what it can actually
support. Happy to add the floor instead if that is the preference.