Skip to content

[Bug] CUDA extension failures terminate the Python process #9

Description

@morluto

Problem

The CUDA extension's CUCHECK and CUDACHECK macros print an error and call
exit(EXIT_FAILURE). Any checked CUDA driver or runtime failure therefore
terminates the Python process from inside the library.

Python caller --> pybind binding --> CUDA error --> exit()
                                              |
                                              `--> process ends

The caller cannot catch the error, attach operation context, clean up its own
state, or coordinate an orderly distributed shutdown.

Reproduction

Pass nvl_multicast_import a valid POSIX file descriptor for /dev/null. The
descriptor is valid at the OS boundary but is not a CUDA-exported shareable
allocation, so the CUDA import call returns an error.

On an H100 NVL with origin/master, the child process printed the old macro's
CUDA error and exited with status 1. The parent regression failed because no
Python exception was catchable:

assert result.returncode == 0
actual returncode: 1
stderr: Failed: CUDA error ... 'invalid device ordinal'

This is a defined error path: it does not fabricate or reinterpret an opaque
CUDA allocation handle.

Expected behavior

  • Checked CUDA calls raise a Python-visible RuntimeError through pybind.
  • The message identifies the failed expression, symbolic error name, numeric
    code, and CUDA description when available.
  • Partially acquired VMM resources are released while the C++ stack unwinds.
  • Successful allocation, mapping, and multicast ownership behavior remains
    unchanged.
Python caller --> pybind binding --> CUDA error --> RuntimeError
                                              |
                                              `--> scoped VMM cleanup

Some asynchronous CUDA failures can leave a CUDA context unusable, and the
application may still decide to terminate. That recovery policy should remain
with the caller rather than being enforced by a library macro for every CUDA
error.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions