Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Experimental] Adopt numba/cuda/cuda_paths.py #447

Draft
wants to merge 33 commits into
base: main
Choose a base branch
from

Conversation

rwgk
Copy link
Collaborator

@rwgk rwgk commented Feb 12, 2025

These commits expand the experiment to adopt the entire numba/cuda/cuda_paths.py

  • commit d31920c — Copy from NVIDIA/numba-cuda#155 as-is (as of Tue Mar 18 09:29:19 2025 -0700)
  • commit ed0ebb3 — ruff format, no manual changes
  • commit 0c5aca5 — Minimal changes to replace external dependencies.

Example:

$ python tests/show_ecosystem_cuda_paths.py
nvvm: _env_path_tuple(by='CUDA_HOME', info='/usr/local/cuda/nvvm/lib64/libnvvm.so.4.0.0')
libdevice: _env_path_tuple(by='CUDA_HOME', info='/usr/local/cuda/nvvm/libdevice/libdevice.10.bc')
cudalib_dir: _env_path_tuple(by='CUDA_HOME', info='/usr/local/cuda/lib64')
static_cudalib_dir: _env_path_tuple(by='CUDA_HOME', info='/usr/local/cuda/lib64')
include_dir: _env_path_tuple(by='CUDA_INCLUDE_PATH Config Entry', info='/usr/local/cuda/include')

First step was:

Experiment related to #441, triggered by this comment (by @kkraus14).

Context: Potentially use this code from cuda_bindings/cuda/bindings/_internal/nvvm_linux.pyx

This PR: Stripped-down (and ruff'ed) copies of:

Tested interactively with:

import cuda_paths
nvvm_path = cuda_paths.get_nvvm_path()
print(f"{nvvm_path=}")

Output:

nvvm_path=_env_path_tuple(by='System', info='/usr/local/cuda/nvvm/lib64/libnvvm.so.4.0.0')

Advantage of this approach: Battle-tested and time-tested.

Disadvantages: TBD @leofang

Copy link
Contributor

copy-pr-bot bot commented Feb 12, 2025

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk rwgk changed the title [Experimental] Clone numba get_nvvm_path() [Experimental] Adopt numba/cuda/cuda_paths.py Mar 19, 2025
@rwgk
Copy link
Collaborator Author

rwgk commented Mar 22, 2025

/ok to test

Copy link

```
______________________ ERROR collecting test_nvjitlink.py ______________________
tests/test_nvjitlink.py:62: in <module>
    not check_nvjitlink_usable(), reason="nvJitLink not usable, maybe not installed or too old (<12.3)"
tests/test_nvjitlink.py:58: in check_nvjitlink_usable
    return inner_nvjitlink._inspect_function_pointer("__nvJitLinkVersion") != 0
cuda/bindings/_internal/nvjitlink.pyx:257: in cuda.bindings._internal.nvjitlink._inspect_function_pointer
    ???
cuda/bindings/_internal/nvjitlink.pyx:260: in cuda.bindings._internal.nvjitlink._inspect_function_pointer
    ???
cuda/bindings/_internal/nvjitlink.pyx:208: in cuda.bindings._internal.nvjitlink._inspect_function_pointers
    ???
cuda/bindings/_internal/nvjitlink.pyx:102: in cuda.bindings._internal.nvjitlink._check_or_init_nvjitlink
    ???
cuda/bindings/_internal/nvjitlink.pyx:59: in cuda.bindings._internal.nvjitlink.load_library
    ???
/opt/hostedtoolcache/Python/3.13.2/x64/lib/python3.13/site-packages/cuda/bindings/path_finder.py:312: in get_cuda_paths
    "nvvm": _get_nvvm_path(),
/opt/hostedtoolcache/Python/3.13.2/x64/lib/python3.13/site-packages/cuda/bindings/path_finder.py:285: in _get_nvvm_path
    by, path = _get_nvvm_path_decision()
/opt/hostedtoolcache/Python/3.13.2/x64/lib/python3.13/site-packages/cuda/bindings/path_finder.py:96: in _get_nvvm_path_decision
    if os.path.exists(nvvm_ctk_dir):
<frozen genericpath>:19: in exists
    ???
E   TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
```
@rwgk
Copy link
Collaborator Author

rwgk commented Mar 22, 2025

/ok to test

```
______________________ ERROR collecting test_nvjitlink.py ______________________
tests/test_nvjitlink.py:62: in <module>
    not check_nvjitlink_usable(), reason="nvJitLink not usable, maybe not installed or too old (<12.3)"
tests/test_nvjitlink.py:58: in check_nvjitlink_usable
    return inner_nvjitlink._inspect_function_pointer("__nvJitLinkVersion") != 0
cuda/bindings/_internal/nvjitlink.pyx:257: in cuda.bindings._internal.nvjitlink._inspect_function_pointer
    ???
cuda/bindings/_internal/nvjitlink.pyx:260: in cuda.bindings._internal.nvjitlink._inspect_function_pointer
    ???
cuda/bindings/_internal/nvjitlink.pyx:208: in cuda.bindings._internal.nvjitlink._inspect_function_pointers
    ???
cuda/bindings/_internal/nvjitlink.pyx:102: in cuda.bindings._internal.nvjitlink._check_or_init_nvjitlink
    ???
cuda/bindings/_internal/nvjitlink.pyx:59: in cuda.bindings._internal.nvjitlink.load_library
    ???
/opt/hostedtoolcache/Python/3.13.2/x64/lib/python3.13/site-packages/cuda/bindings/path_finder.py:313: in get_cuda_paths
    "libdevice": _get_libdevice_paths(),
/opt/hostedtoolcache/Python/3.13.2/x64/lib/python3.13/site-packages/cuda/bindings/path_finder.py:126: in _get_libdevice_paths
    by, libdir = _get_libdevice_path_decision()
/opt/hostedtoolcache/Python/3.13.2/x64/lib/python3.13/site-packages/cuda/bindings/path_finder.py:73: in _get_libdevice_path_decision
    if os.path.exists(libdevice_ctk_dir):
<frozen genericpath>:19: in exists
    ???
E   TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
```
@rwgk
Copy link
Collaborator Author

rwgk commented Mar 22, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 25, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 25, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 25, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 25, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 25, 2025

https://github.com/NVIDIA/cuda-python/actions/runs/14070306494/job/39403538712?pr=447

logs_36215923480.zip

$ grep 'RuntimeError: ' *.txt | sed 's/.* RuntimeError: //' | sort | uniq -c
     20 Failure obtaining paths_cudalib_dir.info
     40 Unable to load libnvJitLink.so: No such file: /__w/cuda-python/cuda-python/cuda_toolkit/lib64/libnvJitLink.so

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 25, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 26, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 26, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 26, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 27, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 27, 2025

/ok to test

@rwgk
Copy link
Collaborator Author

rwgk commented Mar 27, 2025

/ok to test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant