Skip to content

fix(executorch): do not resize a 0-d delegate output - #4469

Open
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:trt-rank0-output
Open

fix(executorch): do not resize a 0-d delegate output#4469
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:trt-rank0-output

Conversation

@shoumikhin

@shoumikhin shoumikhin commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

The problem

Running a model where a TensorRT partition hands a scalar to another backend fails at run
time:

Attempted to change the tensor rank which is immutable: old=0, new=1
TensorRTBackend::execute: resize_tensor failed for output 'output15'
CALL_DELEGATE execute failed at instruction 2: 0x10

A 0-d tensor (a single value with no dimensions) has a rank that cannot change. TensorRT
describes the same scalar as a 1-element one-dimensional shape. The delegate resizes every
output unconditionally, so those two descriptions disagree and the resize is rejected, even
though the buffer is already exactly the right size.

Concretely:

what the graph declares : rank 0, 1 element
what TensorRT reports   : rank 1, shape [1], 1 element
resize_tensor(rank 0 -> rank 1)  ->  rejected, rank is immutable

Where it shows up

Any graph that passes a scalar out of a TensorRT partition. A practical example is a
length-aware attention kernel: the sequence length is computed as a scalar and consumed by
a kernel that runs on a different backend, so the scalar has to cross the partition
boundary. Nothing is wrong with the value or its storage, only with the attempt to restate
its rank.

The fix

Skip the resize when the output is 0-d and the engine reports a single element, since there
is nothing to change:

    const bool scalar_output =
        et_out.dim() == 0 && actual_dims.nbDims == 1 && actual_dims.d[0] == 1;
    if (!scalar_output) {
      Error resize_err = executorch::runtime::resize_tensor(...);
      if (resize_err != Error::Ok) { ... return resize_err; }
    }

The condition is deliberately narrow. Only a 0-d output paired with a 1-element report
skips the resize, so any real shape mismatch still resizes and still reports a failure. No
behavior changes for outputs with one or more dimensions.

Testing

Built the delegate and ran a program where a scalar crosses from a TensorRT partition into
a kernel on another backend. Before the change the run fails with the error above at the
first delegate call. After it, the program loads, runs, and generates, and its logits match
the eager model on the same inputs.

Measured on a reduced-size model with randomly initialised weights, which is what exercises
this code path: the check is on rank and shape handling, not on model quality.

Also compiled the delegate against TensorRT 11.1 to confirm the change is warning-free.

A model whose graph passes a scalar from a TensorRT partition to another backend
fails at run time:

  Attempted to change the tensor rank which is immutable: old=0, new=1
  TensorRTBackend::execute: resize_tensor failed for output 'output15'

A 0-d tensor has an immutable rank of zero, while TensorRT reports a scalar as a
1-element one-dimensional shape. The output resize is unconditional, so the two
descriptions disagree and the resize is rejected even though the buffer is
already the right size.

Skip the resize when the output is 0-d and the engine reports a single element.
Any other shape mismatch still resizes and still reports a failure.

This comes up with a length-aware attention kernel, where a scalar sequence
length crosses the partition boundary into a kernel that consumes it.
@meta-cla meta-cla Bot added the cla signed label Aug 8, 2026
@github-actions github-actions Bot added the component: api [C++] Issues re: C++ API label Aug 8, 2026
@github-actions
github-actions Bot requested a review from narendasan August 8, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant