Attempting to run cuBLAS, but there was no current CUDA context!

[alonadi]

UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:135.)
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass.

when I run nvidia-smi, I see that there is not gpu uses.

[jacobnrohan]

I'm getting a similar warning, but after some investigating I can confirm it is running on GPU, despite the error. It may be that this error message is unrelated to the lack of GPU utilization. @alonadi - which version of CUDA do you have? did you install PyTorch and Torchvision accordingly? Here are wheels for pytorch and torchvision for various version of python/CUDA.

Here's what I got:

$CONDA/envs/apeStyleGAN/lib/python3.11/site-packages/torch/autograd/graph.py:768: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:135.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

The default network is training at 1.7 second per iteration. nvidia-smi shows ~4GB memory utilization:

|    0   N/A  N/A    426695      C   ...$CONDA/envs/$MYENV/bin/python     4014MiB |

Conda environment.

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
aim                       3.23.0                   pypi_0    pypi
aim-ui                    3.23.0                   pypi_0    pypi
aimrecords                0.0.7                    pypi_0    pypi
aimrocks                  0.5.2                    pypi_0    pypi
aiofiles                  24.1.0                   pypi_0    pypi
alembic                   1.13.2                   pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.4.0                    pypi_0    pypi
base58                    2.0.1                    pypi_0    pypi
boto3                     1.34.146                 pypi_0    pypi
botocore                  1.34.146                 pypi_0    pypi
bzip2                     1.0.8                h5eee18b_6  
ca-certificates           2024.7.2             h06a4308_0  
cachetools                5.4.0                    pypi_0    pypi
certifi                   2024.7.4                 pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
contrastive-learner       0.1.1                    pypi_0    pypi
cryptography              43.0.0                   pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
dnspython                 2.6.1                    pypi_0    pypi
einops                    0.8.0                    pypi_0    pypi
email-validator           2.2.0                    pypi_0    pypi
fastapi                   0.111.1                  pypi_0    pypi
fastapi-cli               0.0.4                    pypi_0    pypi
filelock                  3.15.4                   pypi_0    pypi
fire                      0.6.0                    pypi_0    pypi
fsspec                    2024.6.1                 pypi_0    pypi
greenlet                  3.0.3                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
httpcore                  1.0.5                    pypi_0    pypi
httptools                 0.6.1                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
idna                      3.7                      pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
kornia                    0.7.3                    pypi_0    pypi
kornia-rs                 0.1.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_1  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
libuuid                   1.41.5               h5eee18b_0  
mako                      1.3.5                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.3                      pypi_0    pypi
numpy                     2.0.1                    pypi_0    pypi
nvidia-cublas-cu11        11.11.3.6                pypi_0    pypi
nvidia-cuda-cupti-cu11    11.8.87                  pypi_0    pypi
nvidia-cuda-nvrtc-cu11    11.8.89                  pypi_0    pypi
nvidia-cuda-runtime-cu11  11.8.89                  pypi_0    pypi
nvidia-cudnn-cu11         9.1.0.70                 pypi_0    pypi
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
nvidia-curand-cu11        10.3.0.86                pypi_0    pypi
nvidia-cusolver-cu11      11.4.1.48                pypi_0    pypi
nvidia-cusparse-cu11      11.7.5.86                pypi_0    pypi
nvidia-nccl-cu11          2.20.5                   pypi_0    pypi
nvidia-nvtx-cu11          11.8.86                  pypi_0    pypi
openssl                   1.1.1w               h7f8727e_0  
packaging                 24.1                     pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pip                       24.0            py311h06a4308_0  
psutil                    6.0.0                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pycparser                 2.22                     pypi_0    pypi
pydantic                  2.8.2                    pypi_0    pypi
pydantic-core             2.20.1                   pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
python                    3.11.0               h7a1cb2a_3  
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             1.0.1                    pypi_0    pypi
python-multipart          0.0.9                    pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.32.3                   pypi_0    pypi
restrictedpython          7.1                      pypi_0    pypi
retry                     0.9.2                    pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
s3transfer                0.10.2                   pypi_0    pypi
setuptools                69.5.1          py311h06a4308_0  
shellingham               1.5.4                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
sqlalchemy                2.0.31                   pypi_0    pypi
sqlite                    3.45.3               h5eee18b_0  
starlette                 0.37.2                   pypi_0    pypi
stylegan2-pytorch         1.8.10                   pypi_0    pypi
sympy                     1.13.1                   pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
tk                        8.6.14               h39e8969_0  
torch                     2.4.0+cu118              pypi_0    pypi
torchvision               0.19.0+cu118             pypi_0    pypi
tqdm                      4.66.4                   pypi_0    pypi
triton                    3.0.0                    pypi_0    pypi
typer                     0.12.3                   pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2024a                h04d1e81_0  
urllib3                   2.2.2                    pypi_0    pypi
uvicorn                   0.30.3                   pypi_0    pypi
uvloop                    0.19.0                   pypi_0    pypi
vector-quantize-pytorch   0.1.0                    pypi_0    pypi
watchfiles                0.22.0                   pypi_0    pypi
websockets                12.0                     pypi_0    pypi
wheel                     0.43.0          py311h06a4308_0  
xz                        5.4.6                h5eee18b_1  
zlib                      1.2.13               h5eee18b_1  

CUDA details:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0