🐛 [Bug] Incorrect output when using grid-sample operator with use_explicit_typing since tensorrt 10.14.1.48

[TNTwise]

Bug Description

After updating to tensorrt 10.14.1.48, Grid sample (forced to float32) building on a float16 model has an incorrect output when building a mixed precision engine. happens on any torch-tensorrt version. Just occurs when switching tensorrt versions

To Reproduce

Steps to reproduce the behavior:

1. enable use_explicit_typing, if not already enabled.
2. force grid_sample to run in fp32, while the rest of the network runs in fp16

my model build config

model_trt = torch_tensorrt.dynamo.compile(
                exported_program,
                tuple(inputs),
                device=device,
                enabled_precisions=(torch.float,),
                use_explicit_typing=True,
                num_avg_timing_iters=4,
                workspace_size=0,
                min_block_size=1,
)

This line of code causes the issue:
F.grid_sample( input=tenInput.to(torch.float), grid=g.to(torch.float), mode='bilinear', padding_mode=pd, align_corners=True, ).to(torch.float16)

Expected behavior

(Inferencing video with RIFE) - Correct warping with the frame (tensorrt 13 and below)

Logs when building engine:

2026-02-07 15:00:55,522 INFO torch_tensorrt.dynamo._compiler: Compilation Settings: CompilationSettings(enabled_precisions={<dtype.f32: 7>}, workspace_size=0, min_block_size=1, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, assume_dynamic_shape_support=False, sparse_weights=False, engine_capability=<EngineCapability.STANDARD: 1>, num_avg_timing_iters=4, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, timing_cache_path='/tmp/torch_tensorrt_engine_cache/timing_cache.bin', lazy_engine_init=False, cache_built_engines=False, reuse_cached_engines=False, use_explicit_typing=True, use_fp32_acc=False, refit_identical_engine_weights=False, strip_engine_weights=False, immutable_weights=True, enable_weight_streaming=False, enable_cross_compile_for_windows=False, tiling_optimization_level='none', l2_limit_for_tiling=-1, use_distributed_mode_trace=False, offload_module_to_cpu=False, enable_autocast=False, autocast_low_precision_type=None, autocast_excluded_nodes=set(), autocast_excluded_ops=set(), autocast_max_output_threshold=512, autocast_max_depth_of_reduction=None, autocast_calibration_dataloader=None, enable_resource_partitioning=False, cpu_memory_budget=None)

2026-02-07 15:00:56,526 INFO torch_tensorrt.dynamo._compiler: Partitioning the graph via the fast partitioner
2026-02-07 15:00:56,676 INFO torch_tensorrt [TensorRT Conversion Context]: [MemUsageChange] Init CUDA: CPU -2, GPU +0, now: CPU 406, GPU 1933 (MiB)
2026-02-07 15:00:56,811 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node img0 [img0] (Inputs: () | Outputs: (img0: (1, 3, 1088, 1920)@torch.float16))
2026-02-07 15:00:56,812 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node /clamp [aten.clamp.default] (Inputs: (img0: (1, 3, 1088, 1920)@torch.float16, 0.0, 1.0) | Outputs: (clamp: (1, 3, 1088, 1920)@torch.float16))
2026-02-07 15:00:56,812 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node img1 [img1] (Inputs: () | Outputs: (img1: (1, 3, 1088, 1920)@torch.float16))
2026-02-07 15:00:56,813 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node /clamp_1 [aten.clamp.default] (Inputs: (img1: (1, 3, 1088, 1920)@torch.float16, 0.0, 1.0) | Outputs: (clamp_1: (1, 3, 1088, 1920)@torch.float16))
2026-02-07 15:00:56,813 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node f0 [f0] (Inputs: () | Outputs: (f0: (1, 4, 1088, 1920)@torch.float16))
2026-02-07 15:00:56,813 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node f1 [f1] (Inputs: () | Outputs: (f1: (1, 4, 1088, 1920)@torch.float16))
2026-02-07 15:00:56,813 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Converted node timestep [timestep] (Inputs: () | Outputs: (timestep: (1, 1, 1088, 1920)@torch.float16))
...

...
Start building engine.
2026-02-07 15:01:01,186 INFO torch_tensorrt [TensorRT Conversion Context]: [MemUsageChange] Init builder kernel library: CPU -391, GPU +0, now: CPU 4026, GPU 2432 (MiB)
2026-02-07 15:01:01,187 INFO torch_tensorrt [TensorRT Conversion Context]: Global timing cache in use. Profiling results in this builder pass will be stored.
2026-02-07 15:01:01,188 INFO torch_tensorrt [TensorRT Conversion Context]: Compiler backend is used during engine build.
2026-02-07 15:01:01,802 INFO torch_tensorrt [TensorRT Conversion Context]: Detected 1 inputs and 1 output network tensors.
2026-02-07 15:01:01,808 INFO torch_tensorrt [TensorRT Conversion Context]: Total Host Persistent Memory: 80 bytes
2026-02-07 15:01:01,808 INFO torch_tensorrt [TensorRT Conversion Context]: Total Device Persistent Memory: 0 bytes
2026-02-07 15:01:01,808 INFO torch_tensorrt [TensorRT Conversion Context]: Max Scratch Memory: 50137088 bytes
2026-02-07 15:01:01,808 INFO torch_tensorrt [TensorRT Conversion Context]: [BlockAssignment] Started assigning block shifts. This will take 1 steps to complete.
2026-02-07 15:01:01,808 INFO torch_tensorrt [TensorRT Conversion Context]: [BlockAssignment] Algorithm ShiftNTopDown took 0.01567ms to assign 1 blocks to 1 nodes requiring 50137088 bytes.
2026-02-07 15:01:01,808 INFO torch_tensorrt [TensorRT Conversion Context]: Total Activation Memory: 50137088 bytes
2026-02-07 15:01:01,809 INFO torch_tensorrt [TensorRT Conversion Context]: Total Weights Memory: 12561536 bytes
2026-02-07 15:01:01,809 INFO torch_tensorrt [TensorRT Conversion Context]: Compiler backend is used during engine execution.
2026-02-07 15:01:01,809 INFO torch_tensorrt [TensorRT Conversion Context]: Engine generation completed in 0.622215 seconds.
2026-02-07 15:01:01,809 INFO torch_tensorrt [TensorRT Conversion Context]: [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 577 MiB
2026-02-07 15:01:01,809 INFO torch_tensorrt.dynamo.conversion._TRTInterpreter: Build TRT engine elapsed time: 0:00:00.807143
2026-02-07 15:01:01,927 INFO torch_tensorrt [TensorRT Conversion Context]: Serialized 6196 bytes of code generator cache.
2026-02-07 15:01:02,044 INFO torch_tensorrt [TensorRT Conversion Context]: Serialized 9769322 bytes of compilation cache.
2026-02-07 15:01:02,044 INFO torch_tensorrt [TensorRT Conversion Context]: Serialized 0 timing cache entries

Environment

• Torch-TensorRT Version (e.g. 2.10.0):
• PyTorch Version (e.g. 2.10.0):
• CPU Architecture: x86_64
• OS (e.g., Linux): Arch Linux
• How you installed PyTorch (conda, pip, libtorch, source): pip
• Build command you used (if compiling from source):
• Are you using local sources or building from archives:
• Python version: 3.12.9
• CUDA version: 13.0
• GPU models and configuration: RTX 5090
• Any other relevant information:

Additional context

This issue doesnt happen when making enabled_precisions strictly float32 or float16. only when using explicit_typing.
No error happens, just the change in the output.

Thank you
I apologize if i am missing any information

[narendasan]

We will take a look, if its possible to construct a minimal reproducer, that would help a lot

[TNTwise]

Thank you for the quick response. Sorry I'm late to reply. After testing some things, I have narrowed it down to the rife architecture along with grid_sample.

• If I force grid_sample to run on pytorch, with the rest of RIFE running on tensorrt, the model outputs correctly (use_explicit_typing=True)
• If I use grid_sample standalone, making an engine just off the warp layer in RIFE, the output is correct for both tensorrt and pytorch. (use_explicit_typing=True)
• If I put the whole RIFE model on tensorrt, It fails to output correctly. (use_explicit_typing=True)

Rife works fine using full float32 and float16, but grid_sample wont output correctly without fp32.
I apologize if this is not the correct place to put this issue, as I dont know where it is originating from. From as much as I know, It seems like a deeper tensorrt optimization issue.

[narendasan]

yeah some sort of script here would help a lot, we can forward it along to relevant teams. Have you tried explicit casting grid sample operands down to fp32 and cast back to the main operating type after? does that avoid the behavior?

[TNTwise]

Thank you for the response.
Here is a script i have been using for specific testing:
https://github.com/TNTwise/REAL-Video-Enhancer/blob/c140368ca009e7e531f721226fa8e10fef96cf28/backend/tests/unit_tests/gridsample_test.py
Forcing grid_sample to fp32 while the model itself runs in fp16 is the root cause of the issue, running grid_sample on the same precision as the model results in the same result as pytorch.
I apologize for the length of the script, as the issue only happens with the RIFE architecture, not just grid_sample, I do not understand how to simplify it from there.

[asfiyab-nvidia]

@TNTwise are you able to share the weights? With the script you shared, the weights are initialized randomly every time and the output is a garbage image.
If you're unable to share the weights, I can initialize a single model with random weights and compare the output tensors

[TNTwise]

I apologize for not sharing that sooner, that would make sense.
They need to be loaded like such:

state_dict = {
k.replace("module.", ""): v
for k, v in state_dict.items()
if "module." in k
}

flownet = IFNet()
flownet.load_state_dict(state_dict=state_dict, strict=False)
https://github.com/TNTwise/real-video-enhancer-models/releases/download/models/rife4.6.pkl

[asfiyab-nvidia]

@TNTwise Thank you for sharing the weights. Unfortunately, I'm unable to reproduce the bug exactly as you've described. Few points:

1. I'm observing that all output images are different, and the output tensors for each case are also different (PyTorch output doesn't match TensorRT output for the correct pass where use_explicit_typing=False)
2. Can you please clarify the intention? You mentioned wanting to force gridsample to run in FP32 but it runs in FP16 for one of the cases (trt broken). My understanding is that you would like to run the model in FP16 but only run the gridsample layer in FP32. Is that correct?

Here's my updated script with the loaded weights

import torch
import torch.nn.functional as F
import importlib
from pathlib import Path
from PIL import Image
import torch.nn as nn
from torch.nn.functional import interpolate

device = torch.device('cuda:0')
torch_tensorrt = importlib.import_module('torch_tensorrt') if importlib.util.find_spec('torch_tensorrt') else None

def warp(tenInput, tenFlow, tenFlow_div, backwarp_tenGrid):
    dtype = tenInput.dtype
    tenInput = tenInput.to(torch.float)
    tenFlow = tenFlow.to(torch.float)

    tenFlow = torch.cat(
        [tenFlow[:, 0:1] / tenFlow_div[0], tenFlow[:, 1:2] / tenFlow_div[1]], 1
    )
    g = (backwarp_tenGrid + tenFlow).permute(0, 2, 3, 1)
    pd = 'border'
    print("grid sample dtype", dtype)
    return F.grid_sample( # here is the issue
        input=tenInput,
        grid=g,
        mode='bilinear',
        padding_mode=pd,
        align_corners=True,
    ).to(torch.float32)
    
def conv(in_planes, out_planes, kernel_size=3, stride=1, padding=1, dilation=1):
    return nn.Sequential(
        nn.Conv2d(
            in_planes,
            out_planes,
            kernel_size=kernel_size,
            stride=stride,
            padding=padding,
            dilation=dilation,
            bias=True,
        ),
        nn.LeakyReLU(0.2, True),
    )


class ResConv(nn.Module):
    def __init__(self, c, dilation=1):
        super().__init__()
        self.conv = nn.Conv2d(c, c, 3, 1, dilation, dilation=dilation, groups=1)
        self.beta = nn.Parameter(torch.ones((1, c, 1, 1)), requires_grad=True)
        self.relu = nn.LeakyReLU(0.2, True)

    def forward(self, x):
        return self.relu(self.conv(x) * self.beta + x)


class IFBlock(nn.Module):
    def __init__(self, in_planes, c=64):
        super().__init__()
        self.conv0 = nn.Sequential(
            conv(in_planes, c // 2, 3, 2, 1),
            conv(c // 2, c, 3, 2, 1),
        )
        self.convblock = nn.Sequential(
            ResConv(c),
            ResConv(c),
            ResConv(c),
            ResConv(c),
            ResConv(c),
            ResConv(c),
            ResConv(c),
            ResConv(c),
        )
        self.lastconv = nn.Sequential(
            nn.ConvTranspose2d(c, 4 * 6, 4, 2, 1), nn.PixelShuffle(2)
        )

    def forward(self, x, flow=None, scale=1):
        x = interpolate(x, scale_factor=1.0 / scale, mode='bilinear')
        if flow is not None:
            flow = (
                interpolate(flow, scale_factor=1.0 / scale, mode='bilinear')
                * 1.0
                / scale
            )
            x = torch.cat((x, flow), 1)
        feat = self.conv0(x)
        feat = self.convblock(feat)
        tmp = self.lastconv(feat)
        tmp = interpolate(tmp, scale_factor=scale, mode='bilinear')
        flow = tmp[:, :4] * scale
        mask = tmp[:, 4:5]
        return flow, mask


class IFNet(nn.Module):
    def __init__(
        self,
        scale=1.0,
        ensemble=False,
        dtype=torch.float32,
        device: torch.device = torch.device('cuda'),
        width=1920,
        height=1080,
    ):
        super().__init__()
        self.block0 = IFBlock(7, c=192)
        self.block1 = IFBlock(8 + 4, c=128)
        self.block2 = IFBlock(8 + 4, c=96)
        self.block3 = IFBlock(8 + 4, c=64)
        scale = scale
        self.scale_list = [8 / scale, 4 / scale, 2 / scale, 1 / scale]
        self.ensemble = ensemble
        self.dtype = dtype
        self.device = device
        self.width = width
        self.height = height
        self.block = [self.block0, self.block1, self.block2, self.block3]

        self.warp = warp

    def forward(
        self, img0, img1, timestep, tenFlow_div, backwarp_tenGrid, scale=None
    ):
        img0 = img0.clamp(0.0, 1.0)
        img1 = img1.clamp(0.0, 1.0)
        if scale is not None:
            self.scale_list = [8 / scale, 4 / scale, 2 / scale, 1 / scale]
        warped_img0 = img0
        warped_img1 = img1
        flow = None
        mask = None

        for i in range(4):
            if flow is None:
                flow, mask = self.block[i](
                    torch.cat((img0, img1, timestep), 1),
                    None,
                    scale=self.scale_list[i],
                )
                if self.ensemble:
                    f1, m1 = self.block[i](
                        torch.cat((img1, img0, 1 - timestep), 1),
                        None,
                        scale=self.scale_list[i],
                    )
                    flow = (flow + torch.cat((f1[:, 2:4], f1[:, :2]), 1)) / 2
                    mask = (mask + (-m1)) / 2
            else:
                f0, m0 = self.block[i](
                    torch.cat((warped_img0, warped_img1, timestep, mask), 1),
                    flow,
                    scale=self.scale_list[i],
                )
                if self.ensemble:
                    f1, m1 = self.block[i](
                        torch.cat(
                            (
                                warped_img1,
                                warped_img0,
                                1 - timestep,
                                -mask,
                            ),
                            1,
                        ),
                        torch.cat((flow[:, 2:4], flow[:, :2]), 1),
                        scale=self.scale_list[i],
                    )
                    f0 = (f0 + torch.cat((f1[:, 2:4], f1[:, :2]), 1)) / 2
                    m0 = (m0 + (-m1)) / 2
                flow = flow + f0
                mask = mask + m0
            latest_mask = mask
            warped_img0 = self.warp(
                img0, flow[:, :2], tenFlow_div, backwarp_tenGrid
            )
            warped_img1 = self.warp(
                img1, flow[:, 2:4], tenFlow_div, backwarp_tenGrid
            )

        temp = torch.sigmoid(latest_mask)
        frame = warped_img0 * temp + warped_img1 * (1 - temp)
        return frame


def _build_inputs(device, dtype=torch.float16, height=1088, width=1920, seed=1234):
    generator = torch.Generator(device=device)
    generator.manual_seed(seed)
    img0 = torch.rand(1, 3, height, width, device=device, dtype=dtype, generator=generator)
    img1 = torch.rand(1, 3, height, width, device=device, dtype=dtype, generator=generator)
    tenFlow = torch.rand(1, 2, height, width, device=device, dtype=dtype, generator=generator)
    tenFlow_div = torch.tensor([1.0, 1.0], device=device, dtype=dtype)
    backwarp_tenGrid = torch.rand(1, 2, height, width, device=device, dtype=dtype, generator=generator)
    timestep = timestep_tens = torch.full(
                (1, 1, height, width),
                0.5,
            ).cuda().to(dtype)
    return img0, img1, tenFlow_div, backwarp_tenGrid, timestep


def save_output_image(output: torch.Tensor, path: str):
    image_tensor = output.detach().to(torch.float32).cpu()[0]
    image_tensor = image_tensor.clamp(0.0, 1.0)

    image_array = (
        image_tensor[:3].permute(1, 2, 0) * 255.0
    ).to(torch.uint8).numpy()
    image = Image.fromarray(image_array, mode='RGB')

    output_path = Path(path)
    output_path.parent.mkdir(parents=True, exist_ok=True)
    image.save(output_path)


def load_rife_weights(model: IFNet, pkl_path: str) -> IFNet:
    state_dict = torch.load(pkl_path, map_location='cpu')
    # Strip 'module.' prefix
    state_dict = {k.replace("module.", ""): v for k, v in state_dict.items()if "module." in k}
    model.load_state_dict(state_dict)
    return model


def test_warp(model, img0, img1, timestep, tenFlow_div, backwarp_tenGrid, dtype=torch.float32):
    img0 = img0.to(device, dtype)
    img1 = img1.to(device, dtype)
    timestep = timestep.to(device, dtype)
    tenFlow_div = tenFlow_div.to(device, dtype)
    backwarp_tenGrid =backwarp_tenGrid.to(device, dtype)
    model = model.to(device, dtype)

    output = model(img0, img1, timestep, tenFlow_div, backwarp_tenGrid)
    assert output.shape == img0.shape
    return output


def test_warp_trt(model, img0, img1, timestep, tenFlow_div, backwarp_tenGrid, use_explicit_typing=True, dtype=torch.float32):
    img0 = img0.to(device, dtype)
    img1 = img1.to(device, dtype)
    timestep = timestep.to(device, dtype)
    tenFlow_div = tenFlow_div.to(device, dtype)
    backwarp_tenGrid =backwarp_tenGrid.to(device, dtype)
    model = model.to(device, dtype)

    example_inputs = (img0, img1, timestep, tenFlow_div, backwarp_tenGrid)

    exported = torch.export.export(model, example_inputs)
    model_trt = torch_tensorrt.dynamo.compile(
        exported,
        tuple(example_inputs),
        device=torch.device('cuda:0'),
        enabled_precisions=(torch.float,) if use_explicit_typing else (dtype,),
        use_explicit_typing=use_explicit_typing,
        num_avg_timing_iters=4,
        workspace_size=0,
        min_block_size=1,
    )
    output = model_trt(img0, img1, timestep, tenFlow_div, backwarp_tenGrid)
    assert output.shape == img0.shape
    return output


if __name__ == "__main__":
    img0, img1, tenFlow_div, backwarp_tenGrid, timestep = _build_inputs(
        device,
        dtype=torch.float32,
        seed=1234,
    )
    model = load_rife_weights(IFNet().eval(), 'rife4.6.pkl')

    print("testing pytorch")
    output = test_warp(model, img0, img1, timestep, tenFlow_div, backwarp_tenGrid)

    save_output_image(output, 'backend/tests/unit_tests/output_pytorch.png')

    print("testing trt broken")
    output_trt_broken = test_warp_trt(model, img0, img1, timestep, tenFlow_div, backwarp_tenGrid, use_explicit_typing=True, dtype=torch.float16)
    save_output_image(output_trt_broken, 'backend/tests/unit_tests/output_trt_broken.png')

    print("testing trt correct")
    output_trt_correct = test_warp_trt(model, img0, img1, timestep, tenFlow_div, backwarp_tenGrid, use_explicit_typing=False, dtype=torch.float32)
    save_output_image(output_trt_correct, 'backend/tests/unit_tests/output_trt_correct.png')

    print(output)
    print(output_trt_broken)
    print(output_trt_correct)

[TNTwise]

That is indeed my intention. I want to run the model in fp16 with grid_sample in fp32. With randomly initialized tensors, it is hard to see. I have initialized one red and one yellow tensor for visualization purposes:
https://github.com/TNTwise/REAL-Video-Enhancer/blob/d4f6289f0c133df59f5c3bd16ca7a9b1a340c948/backend/tests/unit_tests/gridsample_test.py Thank you for your replies and time, I apologize for being unclear.

[asfiyab-nvidia]

Thanks for updating the script. I re-ran the script and see that all images look identical, attaching the Pytorch output for reference

I also printed the torch_tensorrt.dynamo.utils.cosine_similarity for every pair and they're well above the threshold of (0.99). I'm using TensorRT 10.14.1.48 FYI. Can you please check the cosine similarity metric on your end too.

# cosine similarity
0.9999966621398926 # PyTorch vs TRT (explicit typing set to True)
1.0000001192092896 # PyTorch vs TRT (explicit typing set to False)
0.9999968409538269 # TRT 

[TNTwise]

This is the output for trt_broken on my system

[asfiyab-nvidia]

Attaching trt broken generated in my env:

I used the 26.01 container from https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch/tags?version=26.01-py3 to reproduce my results. My guess is there may be an env mismatch causing the differences in the results we see. Would you like to test the script on the linked container to confirm?

[TNTwise]

Using that environment in docker, it is still broken:
Cosine similarity (pytorch vs trt_broken): [0.5109229683876038]
Cosine similarity (pytorch vs trt_correct): [1.004262089729309]
Cosine similarity (trt_broken vs trt_correct): [0.5109206438064575]
It is broken across linux and windows too (not containerized).

[asfiyab-nvidia]

Hmm, a few questions

1. Are you running the exact script from here unchanged with the weights you provided earlier?
2. Are you running any other package installations or upgrades when inside the 26.01 PyTorch container?

[TNTwise]

I am running the container without modifications, with the same script and weights.
this is how im entering docker
sudo docker run --gpus all -it --rm \ -v $(pwd):/workspace \ nvcr.io/nvidia/pytorch:26.01-py3
nvidia smi from within docker:

| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:01:00.0  On |                  N/A |
|  0%   48C    P5             67W /  450W |     807MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+```