Skip to content

[BUG]: Race condition when closing the cufile driver #1022

@cpcloud

Description

@cpcloud

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.bindings

Describe the bug

Simply calling cufile.driver_open() -> cufile.driver_close() across threads sometimes raises:

cuda.bindings.cufile.cuFileError: DRIVER_NOT_INITIALIZED (5001)

How to Reproduce

This is pretty hard to reproduce, but I was able to run this to trigger it:

pytest cuda_bindings/tests/test_cufile.py::test_driver_open --threads 5 --iterations 10

Expected behavior

I think I would not expect an exception to be raised.

Operating System

Linux

nvidia-smi output

❯ nvidia-smi
Thu Sep 25 14:18:21 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.09              Driver Version: 580.82.09      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 2080 ...    Off |   00000000:0A:00.0  On |                  N/A |
| 36%   47C    P0             63W /  250W |     510MiB /   8192MiB |     23%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            2093      G   ...s2i-xorg-server-21.1.18/bin/X        284MiB |
|    0   N/A  N/A            3248      G   ...lacritty-0.15.1/bin/alacritty         12MiB |
|    0   N/A  N/A          738349      G   ...ox-143.0/bin/.firefox-wrapped        192MiB |
+-----------------------------------------------------------------------------------------+

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions