fix(executorch): do not resize a 0-d delegate output - #4469
Open
shoumikhin wants to merge 1 commit into
Open
Conversation
A model whose graph passes a scalar from a TensorRT partition to another backend fails at run time: Attempted to change the tensor rank which is immutable: old=0, new=1 TensorRTBackend::execute: resize_tensor failed for output 'output15' A 0-d tensor has an immutable rank of zero, while TensorRT reports a scalar as a 1-element one-dimensional shape. The output resize is unconditional, so the two descriptions disagree and the resize is rejected even though the buffer is already the right size. Skip the resize when the output is 0-d and the engine reports a single element. Any other shape mismatch still resizes and still reports a failure. This comes up with a length-aware attention kernel, where a scalar sequence length crosses the partition boundary into a kernel that consumes it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Running a model where a TensorRT partition hands a scalar to another backend fails at run
time:
A 0-d tensor (a single value with no dimensions) has a rank that cannot change. TensorRT
describes the same scalar as a 1-element one-dimensional shape. The delegate resizes every
output unconditionally, so those two descriptions disagree and the resize is rejected, even
though the buffer is already exactly the right size.
Concretely:
Where it shows up
Any graph that passes a scalar out of a TensorRT partition. A practical example is a
length-aware attention kernel: the sequence length is computed as a scalar and consumed by
a kernel that runs on a different backend, so the scalar has to cross the partition
boundary. Nothing is wrong with the value or its storage, only with the attempt to restate
its rank.
The fix
Skip the resize when the output is 0-d and the engine reports a single element, since there
is nothing to change:
The condition is deliberately narrow. Only a 0-d output paired with a 1-element report
skips the resize, so any real shape mismatch still resizes and still reports a failure. No
behavior changes for outputs with one or more dimensions.
Testing
Built the delegate and ran a program where a scalar crosses from a TensorRT partition into
a kernel on another backend. Before the change the run fails with the error above at the
first delegate call. After it, the program loads, runs, and generates, and its logits match
the eager model on the same inputs.
Measured on a reduced-size model with randomly initialised weights, which is what exercises
this code path: the check is on rank and shape handling, not on model quality.
Also compiled the delegate against TensorRT 11.1 to confirm the change is warning-free.