Run the Python test payloads in a minimal sibling container - #11018
Run the Python test payloads in a minimal sibling container#11018shwina wants to merge 9 commits into
Conversation
cuda.compute is meant to work with nothing installed beyond its declared pip dependencies -- no host compiler, no system CUDA toolkit. Every CI job runs in the CCCL devcontainer, which supplies both, so a passing test cannot distinguish "we depend only on our wheels" from "we found gcc and /usr/local/cuda lying around". Each Python test lane is now two scripts: the entry point provisions the wheel (which needs gh) and then hands the payload to a sibling container holding nothing but Python, launched through the host's docker daemon -- the same docker-outside-of-docker arrangement build_cuda_cccl_python.sh already uses. sysctk lanes stay in the devcontainer, since testing against a system-provided toolkit is their entire purpose. This surfaced two real dependency gaps that the devcontainer had been hiding, both fixed here: - The v2 (HostJIT) bindings need libcudart at import: libnvcc.so and libcccl.c.parallel.v2.so both carry a DT_NEEDED on it, auditwheel excludes it from the wheel, and _bindings.py preloaded only nvrtc and nvJitLink. A plain `pip install cuda-cccl[cu13]` on a driver-only machine got bindings that failed to import. - The examples use cp.random, and CuPy loads curand lazily via cuda-pathfinder. Nothing installed it, so request CuPy's own `ctk` extra on the pip-toolkit lanes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 760e392 |
NVIDIA_VISIBLE_DEVICES cannot be used to decide what to hand the sibling: the devcontainer image sets it to "void" -- the nvidia-container-toolkit spelling of "no GPU" -- and that wins over the value workflow-run-job-linux passes in, so it reads "void" inside the devcontainer even on GPU jobs where the runner set it to "all". The previous guard therefore treated every GPU lane as misconfigured. Ask the driver which GPUs are actually reachable and name them explicitly. `--gpus all` would not do: it reaches every GPU on the host, including any assigned to another job on a shared runner. Docker splits the --gpus value on commas unless the device list is quoted within the argument, which the multi-GPU case needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 96bba10 |
This comment has been minimized.
This comment has been minimized.
Mirrors the Linux split: ci/windows/test_cuda_compute_python.ps1 provisions the wheel (which needs gh) and hands the payload to a sibling container launched through the host's Docker daemon, the arrangement Invoke-Cuda13NestedBuild already uses for CUDA 13 wheel builds. Two things differ from Linux because Windows requires it. GPUs are exposed as a whole device class rather than individually, and only under process isolation. And the sibling image must match the host kernel: the Windows devcontainer images report os.version 10.0.20348, so the default is servercore:ltsc2022. Neither platform's image ships Python -- Get-Python installs it with uv, exactly as it already does in the devcontainer. run_compute_tests.ps1 deliberately does not import build_common.psm1: that module resolves cl.exe at import time, which by design does not exist in the minimal image. Get-CudaVersion and Get-CudaMajor learn to fall back to CCCL_CUDA_VERSION, since there is no nvcc in there either; the entry point resolves it outside and passes it in. The matrix override is set to one Linux and one Windows lane for bring-up, and must be reset to empty before merge. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test c1b5dcd |
This comment has been minimized.
This comment has been minimized.
The GPU probe crashed the helper: nvidia-smi is not on PATH in the Windows devcontainer, and with $ErrorActionPreference = "Stop" a missing command throws rather than setting $LASTEXITCODE. There is also nothing to probe -- the CI action passes its GPU decision to the job container as a docker flag, not as an environment variable, so it cannot be read from inside. Request the device class unconditionally instead. Every lane reaching this helper is a GPU lane, the devcontainer around us was started with the same flag, and docker fails loudly if the device is absent. CCCL_MINIMAL_CONTAINER_NO_GPU=1 opts out for a GPU-less local run. Also force TLS 1.2 before fetching the uv installer: Windows PowerShell 5.1 in a bare servercore image can still negotiate TLS 1.0, which astral.sh refuses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 03e2294 |
😬 CI Workflow Results🟥 Finished in 1h 00m: Pass: 75%/4 | Total: 1h 28m | Max: 29m 13sSee results here. AI failure analysis1. Windows minimal container cannot load native Python extensions · 1 jobExplanation: The PR moves this test into a bare Windows Server Core container, where both CCCL's `_bindings_impl` and Numba's unrelated `_typeconv` native extension fail to resolve a DLL. This strongly indicates a missing common native runtime, likely an MSVC redistributable component, but the log does not identify the exact dependency. Evidence: Copy this prompt into a coding agentJobs: 2. RMM device_scalar rejects temporary initialization in cuML and cuGraph · 2 jobsExplanation: RMM 26.10 deletes construction from an rvalue scalar, while current cuML and cuGraph sources initialize `device_scalar` objects with temporary values such as `0`, `1.0`, and `size_t{0}`. Both RAPIDS jobs therefore hit the same source-compatibility break; the supplied PR diff does not modify these RAPIDS components. Evidence: Copy this prompt into a coding agentJobs: |
The Windows lane failed every test: numba's _typeconv and our own cccl.c.parallel.dll both died with "DLL load failed ... The specified module could not be found". mcr.microsoft.com/windows/servercore ships none of msvcp140.dll, vcruntime140.dll or vcruntime140_1.dll, and both binaries import them. That is the one place the Linux/Windows symmetry does not hold: python:3.14-slim still ships glibc and libstdc++, because every C/C++ Python extension links against them, whereas the Windows equivalents come from the MSVC redistributable rather than from Windows itself. It is not an undeclared dependency a wheel could carry either -- numba has the same requirement, and python.org's own installer bundles the redistributable. A bootstrap now installs it before handing off to the payload. The container still has no compiler and no CUDA toolkit, so the comparison the lane exists to make is intact. The payload arrives in CCCL_MINIMAL_PAYLOAD rather than as a parameter: a Param() block in the bootstrap binds the payload's own -py-version first and fails with "A positional parameter cannot be found that accepts argument '-ctk-mode'". Verified against the CI-built wheel in servercore:ltsc2022 -- reproducing the failure, then fixing it -- and end to end in the sibling container: 1789 tests collect with no errors, and test_reduce.py passes 133/133 on a GPU. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 971b0c1 |
Inline the MSVC runtime install into ci/windows/run_compute_tests.ps1. There is one Windows payload, so a separate bootstrap script bought an extra file, an extra env var and an extra layer of indirection for six lines of installer; minimal_container_bootstrap.ps1 and CCCL_MINIMAL_PAYLOAD both go away. Single-source Invoke-Checked in build_common_python.psm1, which the minimal container can import, and re-export it from build_common.psm1 so the scripts that import only that still see it. Invoke-Step was a renamed copy of it. Hoist the TLS 1.2 bump to module scope while there, which removes its second copy and lets the payload rely on it before Get-Python runs. Drop CCCL_MINIMAL_CONTAINER_NO_GPU, which nothing set, and the `docker version` probe, which spawned a process to pre-empt an error `docker run` reports anyway. Revert the ci/matrix.yaml bring-up override. Cut ~100 lines of comment. The rationale for the whole arrangement was written out in five places and the payload contract in four; both now point at "Testing Python in a minimal container" in ci_overview.rst, which is where it belongs. Comments that narrated an investigation rather than the resulting rule are trimmed to the rule. Verified on Windows: Invoke-Checked resolves through build_common.psm1 in both import forms, and the full compute suite passes in the sibling container -- 1764 + 117 + 22 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 3b08942 |
|
@coderabbitai review |
✅ Action performedReview finished.
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughChangesThe CI scripts now separate wheel preparation from test execution. Linux and Windows lanes run payload scripts directly or inside minimal sibling containers. Shared helpers support fallback CUDA detection and environment setup. Compute bindings preload Containerized Python CI
CUDA runtime loading
Suggested reviewers: Merge Risk: 🟠 High · up to The change runs tests in sibling containers, but the current scripts can silently mount the wrong workspace on Linux and execute unverified downloaded binaries with CI privileges on Windows. These can cause misleading CI failures or create a security exposure, so the issues should be fixed before merge. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
ci/windows/build_common_python.psm1 (1)
46-46: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winimportant: Verify every remote executable before CI execution. Both sites execute mutable downloaded content with CI-job privileges.
ci/windows/build_common_python.psm1#L46-L46: download a pinned uv installer and verify its signature or SHA-256 before execution.ci/windows/run_compute_tests.ps1#L36-L38: pin the VC++ redistributable and verify its Authenticode signature or SHA-256 beforeStart-Process.
🧹 Nitpick comments (1)
ci/util/python/run_in_minimal_container.sh (1)
78-87: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winsuggestion: Pass
--user "$(id -u):$(id -g)"todocker run. With the defaultpython:3.14-slimimage, payloads run asroot;pytestcan then create root-owned.pytest_cachefiles in the bind-mounted workspace, blocking latercodercleanup or updates.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1898c566-210d-4f49-a013-e94dcb484fe4
📒 Files selected for processing (16)
ci/pyenv_helper.shci/test_cuda_cccl_examples_python.shci/test_cuda_compute_minimal_python.shci/test_cuda_compute_python.shci/util/python/run_compute_minimal_tests.shci/util/python/run_compute_tests.shci/util/python/run_examples_tests.shci/util/python/run_in_minimal_container.shci/windows/build_common.psm1ci/windows/build_common_python.psm1ci/windows/run_compute_tests.ps1ci/windows/run_in_minimal_container.ps1ci/windows/test_cuda_compute_python.ps1docs/infrastructure/ci/references/ci_overview.rstpython/cuda_cccl/cuda/compute/_bindings.pypython/cuda_cccl/pyproject.toml
Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.
| Neither image ships Python: ``uv`` installs the interpreter the lane asked for, exactly as | ||
| it does in the devcontainer. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
important: Correct the Python image description. The Linux launcher’s image assignment in ci/util/python/run_in_minimal_container.sh defaults to python:3.14-slim, which already contains a bootstrap Python. State that Linux uses this interpreter to bootstrap uv, while uv installs the lane’s requested interpreter. Only the Windows Server Core image starts without Python.
As per path instructions, documentation changes must focus on technical accuracy, buildable examples, API/version consistency, migration impact, and matching documentation for public API changes.
Source: Path instructions
| # The test extras do not include CuPy. The cuda.compute examples (run via | ||
| # tests/test_examples.py) require it; install it separately (e.g. | ||
| # `pip install cupy-cuda13x`). | ||
| # `pip install cupy-cuda13x[ctk]`). Use the `ctk` extra unless a system CUDA |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- python/cuda_cccl/pyproject.toml lines 84-96 ---'
sed -n '84,96p' python/cuda_cccl/pyproject.toml
printf '%s\n' '--- relevant repository convention files ---'
find /tmp/coderabbit-repo-knowledge/nvidia-cccl-2f773195 -maxdepth 2 -type f \( -path '*/coding_guidelines*' -o -path '*/learnings/*' \) -print 2>/dev/null | head -20
printf '%s\n' '--- zsh glob behavior for the exact command argument ---'
if command -v zsh >/dev/null 2>&1; then
tmpdir="$(mktemp -d)"
trap 'rm -rf "$tmpdir"' EXIT
(
cd "$tmpdir"
touch 'cupy-cuda13x' 'cupy-cuda13x-extra'
zsh -f -c 'pip() { printf "<%s>\n" "$@"; }; pip install cupy-cuda13x[ctk]' 2>&1
)
else
printf '%s\n' 'zsh unavailable'
fiRepository: NVIDIA/cccl
Length of output: 2792
🌐 Web query:
zsh official documentation filename generation square brackets nomatch unquoted brackets
💡 Result:
In Zsh, unquoted square brackets [...] are interpreted as part of a glob pattern (filename generation) [1][2]. When you use these in a command, the shell attempts to match them against existing files in the filesystem [1][3]. If no files match the pattern, Zsh's default behavior depends on the NOMATCH option [2][3]. 1. NOMATCH Option: By default, the NOMATCH option is enabled, causing Zsh to print an error message (e.g., "zsh: no matches found:...") and stop processing the command if a glob pattern fails to match [2][4][3]. 2. Handling Unquoted Brackets: If you want to use brackets as literal characters without triggering filename generation (and thus avoiding the NOMATCH error), you have several options: - Quoting: Wrap the pattern in single or double quotes (e.g., "[...]") [4][3]. This prevents the shell from interpreting the brackets as glob operators [4]. - Unsetting NOMATCH: Run unsetopt NOMATCH to instruct the shell to leave the pattern unchanged if it fails to match any files, rather than throwing an error [4][3][5]. - NULL_GLOB: Alternatively, you can use setopt NULL_GLOB, which causes unmatched glob patterns to be removed from the argument list entirely instead of leaving them or reporting an error [3][5]. Note that bracketed expressions in Zsh can also be interpreted as index subscripts (e.g., when accessing array elements), which may sometimes lead to unexpected parsing behavior if the shell interprets the brackets in a different context than you intended [6]. Always quote strings containing brackets if you intend for them to be treated as literal text [4][3].
Citations:
- 1: https://zsh.sourceforge.io/Intro/intro_2.html
- 2: https://zsh.sourceforge.io/Doc/Release/Options.html
- 3: https://unix.stackexchange.com/questions/478813/why-does-a-failed-filename-generation-make-zsh-stop-processing-a-script
- 4: https://superuser.com/questions/1471106/zsh-is-treating-a-string-as-a-filename-pattern-and-complaining-about-it-nomatch
- 5: https://stackoverflow.com/questions/70904966/different-shell-behaviors-of-unmatched-glob-in-zsh-and-bash
- 6: https://github.com/zsh-users/zsh/blob/c96606cc0617b85d3bf0784d0bf1ecd71e44cbd7/Src/glob.c
🌐 Web query:
CuPy official installation documentation cupy-cuda13x[ctk] quote
💡 Result:
According to the official CuPy installation documentation, the [ctk] tag is used to install NVIDIA’s CUDA component wheels as optional dependencies [1][2]. This allows users to quickly spin up a virtual environment without installing a system-wide CUDA Toolkit, requiring only the CUDA driver [1][3]. The official documentation explains the usage as follows: By default, the above command only installs CuPy itself, assuming a CUDA Toolkit is already installed on the system. To use NVIDIA’s CUDA component wheels (so as to quickly spinning up a fresh virtual environment without installing a system-wide CUDA Toolkit – only the CUDA driver is needed – and allowing smaller installation footprint and better interoperability with other Python GPU libraries), you can pass [ctk] to install them all as optional dependencies, e.g.: $ pip install "cupy-cuda13x[ctk]" [1][4] When using this command, ensure you enclose the package name and tag in quotes to avoid shell interpretation errors [1][2].
Citations:
- 1: https://docs.cupy.dev/en/stable/install.html
- 2: https://github.com/cupy/cupy/blob/main/docs/source/install.rst
- 3: https://docs.cupy.dev/en/stable/upgrade.html
- 4: https://pypi.org/project/cupy-cuda13x/
suggestion: Quote the CuPy extras requirement.
In zsh, unquoted [...] can trigger filename generation and prevent pip from receiving the requirement. Use pip install "cupy-cuda13x[ctk]".
Source: MCP tools
The three entry points differed by one line -- the payload they hand off to -- so the rule deciding which lanes run in the minimal container was written out three times. Move it into dispatch_python_lane, where changing it means editing one file. Likewise the payloads opened with an identical preamble; python_payload_init now covers the argument parsing, the cuda-toolkit pin and the interpreter. Replace the `ls wheelhouse/cuda_cccl-*.whl` glob with cuda_cccl_wheel_path, which errors unless there is exactly one match. The glob yielded an empty string for none and a space-joined pair for two, and passed either to pip. Windows already resolved the wheel this way via Get-OnePathMatch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fall back to the working directory only when running directly on the host. Inside the devcontainer it is a path the host daemon cannot resolve, so require HOST_WORKSPACE there and say which knob to reach for instead. The Linux image does ship a Python, just not the one the lane asked for; say what it is actually for. Quote the CuPy extra in the example: zsh globs the unquoted brackets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 3eeacd4 |
Array splatting binds positionally, so `& run_compute_tests.ps1 @payloadArgs` handed the payload the literal "-py-version" as its version and tripped the ValidatePattern. The container path was unaffected: there the same array becomes a real `docker run ... -File payload.ps1 -py-version 3.14` command line, where the flag parses as a parameter name. Only `sysctk`, the one mode that skips the container, took the broken branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/ok to test 73033d2 |
1 participant
Description
Contributes to #10777. Supersedes #10918, which took a more invasive route (see below).
cuda.computeis meant to work with nothing installed beyond its declared pipdependencies — no host compiler, no system CUDA toolkit. Every CI job runs in the CCCL
devcontainer, which supplies both, so a passing test cannot distinguish "we depend only
on our wheels" from "we happened to find
gccand/usr/local/cudalying around".Each Python test lane is now two scripts:
ci/test_<lane>.shprovisions the wheel (whichneeds
gh) and hands the payload to a sibling container holding nothing but Python,launched through the host's docker daemon — the same docker-outside-of-docker arrangement
ci/build_cuda_cccl_python.shalready uses for wheel builds.ci/util/python/run_<lane>_tests.shis the payload, and must survive in the minimal image.
Applies to both the v1 (NVRTC) and v2 (HostJIT) backends.
Two real dependency gaps this surfaced
Both were invisible in CI because
/usr/local/cudawas always present, and both affectusers doing a plain
pip installon a driver-only machine:libcudartmissing at import for v2.libnvcc.soandlibcccl.c.parallel.v2.soboth carry a
DT_NEEDEDonlibcudart.so.<major>, auditwheel excludes it from thewheel, and
_bindings.pypreloaded onlynvrtcandnvJitLink. Bindings failed toimport with a
RuntimeWarning, degrading to anAttributeErroron first use.libcurandmissing for the examples. They usecp.random, and CuPy loadscurandlazily through
cuda-pathfinder. Fixed by requesting CuPy's ownctkextra on thepip-toolkit lanes.
Deliberately unchanged
ci/matrix.yaml,build-workflow.py,workflow-run-job-linux/action.ymland thedispatch workflows are untouched — lane selection lives in the scripts, which already
know
ctk_mode. These lanes stay in the devcontainer because they need what it provides:py_ctk_mode: sysctk(tests a system-provided toolkit),test_headers(compiles C++),python_tsan(LD_PRELOADs the runner'slibtsanviagcc),test_py_stf(separatewheel and script), and all Windows lanes (
ci/windows/*.ps1, no equivalent yet).CCCL_MINIMAL_CONTAINER=0runs the payload in the devcontainer instead, for local workor for comparing the two environments.
Validation
The mechanism is exercisable locally, and was: running the helper from inside a CCCL
devcontainer, a successful payload exits 0, a failing one propagates 126, and the mount,
GPU flag and env forwarding are correct. The
cudartfix was verified against theCI-built wheel in
python:3.14-slim— reproducing the failure, then fixing it.Not yet proven: that GHA GPU runners mount the docker socket into the devcontainer.
Every existing docker-outside-of-docker script in the repo is a
gpu: falsejob, sothere is no precedent. The helper fails with a plain message rather than an opaque
docker runerror if the socket is absent. One CI run settles it.Windows minimal containers remain out of scope, so #10777 is not fully closed by this.
Checklist