[BUG] Unchecked CUDA calls in noexcept functions cause sticky errors to leak into unrelated downstream paths

[lennartvoelz]

Describe the Bug

Several CUDA API calls are made without any error checking. I first observed this when encountering the bug described in issue #1237, but there are more instances around the codebase.
The standard error macro MATX_CUDA_CHECK is not suited for some of the instances, as they are declared noexcept, so a throw would terminate.
When one of these calls fails, nothing observes it at the call site and they will only surface later on an unrelated call.

To Reproduce

Check issue #1237 for one reference example.

Expected Behavior

A failing CUDA call should be logged at minimum but never be dropped silently.

Code Snippets

// include/matx/core/tensor.h:743-757
  __MATX_INLINE__ void PrefetchDevice(cudaStream_t const stream)
  const noexcept
  {
    MATX_NVTX_START("", matx::MATX_NVTX_LOG_API)

    int dev;
    cudaGetDevice(&dev);                 // <- unchecked
  #if CUDART_VERSION <= 12000
    cudaMemPrefetchAsync(this->Data(), this->desc_.TotalSize() * sizeof(T), dev, stream); // <- unchecked
  #else
    cudaMemLocation loc;
    loc.id = dev;
    loc.type = cudaMemLocationTypeDevice;
    cudaMemPrefetchAsync(this->Data(), this->desc_.TotalSize() * sizeof(T), loc, 0, stream); // <- unchecked
  #endif
  }

System Details

OS: Ubuntu 24.04.4 (JetPack 7)
CUDA version: 13.2
g++ version: 13.3.0
System: Jetson Orin AGX (sm_87)

Additional Context

I'll add a non-throwing macro that logs and clears the sticky error, which can then be used in noexcept/destructor contexts.
This would fix issue #1237, but as mentioned in the linked issue, it would still involve invalid API usage, as cudaMemPrefetchAsync is only defined for managed memory.

[lennartvoelz]

Working on it