Repository navigation
[ROCm] Test ROCm failures on PR#1554 - #1642
akashveramd wants to merge 81 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1642
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit a324f14 with merge base d249ad5 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Co-authored-by: Cursor <cursoragent@cursor.com>
35e74f6 to
61016ce
Compare
librocjpeg links against libva.so.2 and libva-drm.so.2 (VA-API, used by rocJPEG's HYBRID GPU-JPEG backend). auditwheel was pulling both into the wheel because they weren't excluded, and repair_wheel.py's check_bundling() then rejected them as unexpected. libva/libva-drm are system-provided display-stack libraries, present on any ROCm install alongside libdrm (which is already excluded). Add "libva*" to the same exclude list so auditwheel leaves them on the system rather than bundling them. Co-authored-by: Cursor <cursoragent@cursor.com>
ROCm >= 7.14 distributes the full ROCm stack (including rocJPEG) as pip wheels (_rocm_sdk_core / _rocm_sdk_devel site-packages) rather than system RPMs. The rocjpeg-devel, libva-amdgpu and mesa-amdgpu-va-drivers DNF packages therefore don't exist on the ROCm 7.14 builder image and the install was failing with "No package rocjpeg available". Check for librocjpeg.so under /opt/conda (pip-wheel install path) and /opt/rocm (classic RPM install path) before attempting dnf install. If already present, skip the install entirely. The existing dnf path is preserved for ROCm 7.2 and earlier where RPMs are the only source. Co-authored-by: Cursor <cursoragent@cursor.com>
librocjpeg links libva.so.2 as a DT_NEEDED entry and needs it at dlopen time even when only the HARDWARE backend is used. The previous fix skipped all dnf installs when librocjpeg was found via ROCm 7.14 pip wheels, which left libva absent on the test machine and caused libtorchcodec_image.so to fail to load with OSError. libva ships in AlmaLinux standard repos so install it unconditionally. Co-authored-by: Cursor <cursoragent@cursor.com>
_find_rocjpeg_lib() only searched ROCM_HOME/lib, torch's ROCM_HOME, and /opt/rocm/lib. For ROCm >= 7.14, librocjpeg lives inside the _rocm_sdk_core pip-wheel's site-packages (e.g. /opt/conda/lib/python3.11/site-packages/_rocm_sdk_core/lib/librocjpeg.so), none of which were on those search paths. The function therefore returned None, printed a WARNING, and auditwheel never bundled librocjpeg. At test time libtorchcodec_image.so had DT_NEEDED: librocjpeg.so but no copy was on the dynamic-linker search path → OSError: Could not load this library. Fix: add a site-packages-first fallback that checks _rocm_sdk_core/lib and _rocm_sdk_devel/lib in the current interpreter's site-packages, then falls back to a broad /opt/conda/** glob (mirrors install_rocjpeg.sh). Also add ldd diagnostics before pytest in the install-and-test workflow so that any future missing transitive dep is visible in the log rather than hidden behind a bare OSError, and update the stale comment that said the wheel does not bundle librocjpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
…yout PyTorch ROCm wheels may bundle the ROCm runtime (libamdhip64, libhsa-runtime64 etc.) inside torch/lib/ rather than depending on a separate _rocm_sdk_core pip wheel. Add \$ORIGIN/../../torch/lib to the RPATH patched onto the bundled librocjpeg-HASH.so so it can find those deps in either layout: - torch/lib/ (torch bundles ROCm runtime, like it does for CUDA) - _rocm_sdk_core/lib/ (TheRock separate pip-wheel layout) - /opt/rocm/lib (ROCm <= 7.2 system install) Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
7435c2f to
2efd33f
Compare
…nostics The bundled librocjpeg-HASH.so has DT_NEEDED entries for librocm_sysdeps_va.so.2, librocm_sysdeps_drm_amdgpu.so.1, librocprofiler-register.so.0 etc. These live in _rocm_sdk_core/lib/ and _rocm_sdk_core/lib/rocm_sysdeps/lib/ but are not on the system search path and are not loaded by torch at import time. The RPATH on the bundled lib uses $ORIGIN which ldd cannot expand without execute permission (ldd shows all $ORIGIN-based entries as "not found"). Whether $ORIGIN resolves correctly at actual dlopen time is unclear, so set LD_LIBRARY_PATH explicitly before pytest as belt-and-suspenders. Also fix the rocjpeg_so find path (torchcodec.libs/ is a sibling of torchcodec/, not a child) and add objdump RPATH output so we can verify the RPATH was patched correctly on the bundled librocjpeg. Co-authored-by: Cursor <cursoragent@cursor.com>
Bundling librocjpeg moves it out of _rocm_sdk_core/lib, breaking AMD's own relative RPATH inside it that points to librocm_sysdeps_* transitive deps. We then have to re-patch those paths ourselves, which is fragile and was causing the missing librocm_sysdeps_va.so.2 / libamdhip64.so.7 failures at runtime. The cleaner approach: exclude librocjpeg from auditwheel bundling and instead append _rocm_sdk_core/lib (ROCm >= 7.14) and /opt/rocm/lib (ROCm <= 7.2) to the RPATH of libtorchcodec_image.so itself. At runtime the dynamic linker finds librocjpeg in its original location; AMD's own RPATH on that library then correctly resolves all transitive deps without any intervention from us. _find_rocjpeg_lib() is kept: auditwheel still needs librocjpeg in LD_LIBRARY_PATH to resolve the DT_NEEDED during its analysis pass, even though we exclude it from bundling. Co-authored-by: Cursor <cursoragent@cursor.com>
…ness Co-authored-by: Cursor <cursoragent@cursor.com>
check_bundling() previously required librocjpeg to be bundled in every ROCm wheel, which is now intentionally wrong. Update it to: - Treat "not bundled" as correct for ROCm wheels - Instead verify that libtorchcodec_image.so has _rocm_sdk_core/lib in its RPATH (confirming _patch_image_so_rpath_in_wheel ran correctly) - Raise if librocjpeg IS bundled (that would be a regression) - Remove _is_rocjpeg from the bundled-lib allowlist for the same reason - Add top-level `import tempfile` (used by the new RPATH check) Co-authored-by: Cursor <cursoragent@cursor.com>
… rocjpeg - bundle_third_party_licenses: warn (not raise) if rocjpeg LICENSE is missing, since librocjpeg is no longer redistributed in the wheel - check_bundling license check: only enforce rocjpeg keyword if the license file was actually bundled (it's optional, not mandatory) - Fix stale "ROCm wheels bundle librocjpeg" comment - Fix stale _find_rocjpeg_license docstring Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
$ORIGIN/../../_rocm_sdk_core/lib resolves to python3.x/_rocm_sdk_core/lib which does not exist. The correct path is $ORIGIN/../_rocm_sdk_core/lib: libtorchcodec_image.so lives in site-packages/torchcodec/, so one ../ brings us to site-packages/ where _rocm_sdk_core/ lives. This matches auditwheel's own convention: it sets $ORIGIN/../torchcodec.libs for the same reason. Co-authored-by: Cursor <cursoragent@cursor.com>
rocJPEG's HYBRID backend calls vaInitialize() to initialize the AMD GPU video decoder. libva.so.2 alone is not enough; it also needs the AMD VA-API backend driver (mesa-amdgpu-va-drivers) which provides the radeonsi/amdgpu DRI plugin that vaInitialize() dlopen-s at runtime. Without it, vaInitialize() fails and rocJPEG returns ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED, causing all jpeg_cuda tests to fail with "Failed to initialize rocJPEG with the hybrid backend". Install mesa-amdgpu-va-drivers alongside libva in the ROCm 7.14 pip-wheel path; fall back to libva-only if the AMD graphics repo is not available (build-only runners). Co-authored-by: Cursor <cursoragent@cursor.com>
Since ROCm 7.14, mesa (including the amdgpu/radeonsi VA-API backend driver) ships inside the _rocm_sdk_core pip wheel. There is no need to install mesa-amdgpu-va-drivers via dnf. However, libva's driver discovery (vaInitialize) needs to know where to find the DRI driver plugin. Set LIBVA_DRIVERS_PATH to _rocm_sdk_core/lib/dri so the vendored librocm_sysdeps_va.so.2 can find the bundled mesa backend, fixing the ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED failures. Also add a fallback diagnostic that lists _rocm_sdk_core/lib/ if the dri subdirectory is not found, to help debug future layout changes. Reverts the mesa-amdgpu-va-drivers dnf install added in the previous commit (442ac6c) as it is not needed. Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-line python block inside $() in the YAML run: | block had lines starting at column 0, which breaks the YAML scalar parser (same issue as before). Collapse to a single line. Also improve the *_drv_video.so search: use rglob instead of checking a hardcoded lib/dri/ path so the mesa VA-API backend driver is found regardless of the exact subdirectory AMD chose inside _rocm_sdk_core. The fallback diagnostic now greps the full recursive listing for dri/va/video/mesa/gallium/radeon keywords to pinpoint the layout if the driver isn't found. Co-authored-by: Cursor <cursoragent@cursor.com>
…iver libva auto-detects the DRM driver name as "radeonsi" from the kernel and looks for radeonsi_drv_video.so. AMD vendored it as librocm_sysdeps_gallium_drv_video.so, so LIBVA_DRIVER_NAME must be set to "librocm_sysdeps_gallium" to override that. Verified locally: vaInitialize returns 0 with both LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME set. Co-authored-by: Cursor <cursoragent@cursor.com>
1. test/utils.py: assert_frames_equal crashes with TypeError when cuda_version_used_for_building_torch() returns None on ROCm. Handle None explicitly, treating it as >= CUDA 13 (use atol=3). 2. DecodeJpegRocm: ROCJPEG_OUTPUT_RGB_PLANAR via the HW (VCN) backend produces ~51% correct pixels for colour (YCbCr) JPEG sources on MI350X: the hardware returns unconverted YCbCr planes instead of RGB. Fix: set force_hybrid=true in make_plan() for colour JPEGs requesting RGB output, routing them to the HYBRID backend which handles YCbCr->RGB in software. 3. DecodeJpegRocm: rocJpegDecodeBatched writes with an internally-aligned pitch that does not match our tensor row stride, producing completely wrong output when the batch mixes images of different dimensions. Fix: replace rocJpegDecodeBatched with individual rocJpegDecode calls in decode_batched_hardware(). 4. test_decoders.py: test_cuda_jpeg_errors expects corrupt JPEG input to raise RuntimeError. rocJPEG silently "succeeds" on this corrupt JPEG (nvJPEG raises, rocJPEG does not). Skip the corrupt-JPEG assertion on ROCm; the CPU-tensor-on-GPU error check still runs. Co-authored-by: Cursor <cursoragent@cursor.com>
These scripts are no longer needed with the TheRock/pip-wheel layout introduced in PR meta-pytorch#1642. rocDecode and RPP are now provided by the _rocm_sdk_core and _rocm_sdk_devel pip packages. Added ROCM_ROCDECODE_RPP_NOTES.md documenting the repair_wheel.py changes needed after PR meta-pytorch#1642 lands (adding librocdecode* and librpp* to the auditwheel exclude list). Co-Authored-By: Claude <noreply@anthropic.com>
The nightly index and builder image publish 10.1, so the wheel build drops 10.0 and the GPU job installs torch and rocm[devel] from that index. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
rocJPEG needs ROCm 10.1 or newer. Drop the channel-specific rewrite and filter the nightly matrix with a numeric comparison. Co-authored-by: Cursor <cursoragent@cursor.com>
Use the first Python version and each ROCm version from that matrix so the download artifact and install indexes match the built wheel. Co-authored-by: Cursor <cursoragent@cursor.com>
The manylinux ROCm image has no system rocm-smi. Put the SDK bin dirs on PATH and prefer amd-smi once the devel tree is initialized. Co-authored-by: Cursor <cursoragent@cursor.com>
Simulates merge-to-main wheel coverage on this PR by undoing test-infra LIMIT_PR_BUILDS. Revert this commit before landing; the GPU test job still uses only the first Python. Co-authored-by: Cursor <cursoragent@cursor.com>
…ns." This reverts commit a3f4909.
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Isolated diagnostic for whether rocJPEG resolves via $ORIGIN RPATH on ROCm 10.1, so it can be reverted once VA-API init is understood. Co-authored-by: Cursor <cursoragent@cursor.com>
A/B whether setting ROCM_PATH alone fixes rocJPEG InitVAAPI; revert if it does not help. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep exporting ROCM_PATH from _rocm_sdk_devel; it is required for rocJPEG InitVAAPI. Fail the job if the devel package is missing. Co-authored-by: Cursor <cursoragent@cursor.com>
TheRock documents this as the canonical way to locate the devel tree; avoid hardcoding the _rocm_sdk_devel package layout. Co-authored-by: Cursor <cursoragent@cursor.com>
| build-python-only: "disable" | ||
|
|
||
| build: | ||
| filter-matrix: |
There was a problem hiding this comment.
there should be a selector available from the test-infra's generate-matrix job? If not, we should add it there instead of having this logic here
There was a problem hiding this comment.
@NicolasHug Would you want us to get rid of the entire filtering logic in filter-matrix and move it to generate-matrix? If so, we will need the ability to get a single-python-version matrix from generate_binary_build_matrix.py, since torchcodec (and many other repos might) only wants to run tests for a single python version, regardless of PR or main branch commit. And we are trying to not hardcode the python version in the yml file, so that it auto-bumps (less maintenance, even for CUDA workflows).
But currently, generate_binary_build_matrix.py returns a single-python-version matrix only for PR-triggers, without any ability for the user to say "always give me just the first python version".
@huydhn Would you be okay with this suggestion:
- Introduce an optional
python-version-limitworkflow input togenerate_binary_build_matrix.pythat defaults toauto(current default). Caller can alternatively set it tofirst, which will limit the python versions to the first one in the matrix.
python-versions-limit:
description: "auto: all Python versions, except pull requests without ciflow/binaries/all get only the first. first: only the first Python version on every event."
default: "auto"
type: string
- Feed it into the existing
LIMIT_PR_BUILDSexpression sofirstforces the reduced matrix even when the label is present, andautoleaves every current caller unchanged:
LIMIT_PR_BUILDS: ${{ inputs.python-versions-limit == 'first' || (github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ciflow/binaries/all')) }}
|
|
||
|
|
||
| def _patch_image_so_rpath_in_wheel(wheel_path: Path) -> None: | ||
| """Append ROCm library search paths to libtorchcodec_image.so's RPATH. |
There was a problem hiding this comment.
I'm wary of modifying rpath here. Does that mean we expect the user to have a rocm installation via the TheRock/pip-wheel layout, or does it support more general setups?
What are the .so files that live in the folders that we now set as rpath - are there risks that some of those .so might shadow those from the user's system (libc.so etc.)?
There was a problem hiding this comment.
@NicolasHug: Currently we only handle TheRock/pip ROCm. Other ROCm installs (e.g. tarball) are not covered by the RPATH we set.
TheRock separates ROCm libs and System/sysdeps libs. ROCm libs are located under- _rocm_sdk_core/lib and System/sysdeps libs are located under- _rocm_sdk_core/lib/rocm_sysdeps/lib.
We set RPATH on libtorchcodec_image.so pointing at the ROCm libs directory only ($ORIGIN/../_rocm_sdk_core/lib), and do not add the sysdeps path. Hence, there is no risk of user's system libs getting shadowed.
Please see the attached screen shot which lists ROCm libs.
| """Find librocjpeg.so at wheel repair time so auditwheel can resolve it. | ||
|
|
||
| auditwheel needs librocjpeg to be resolvable in LD_LIBRARY_PATH during | ||
| `auditwheel repair` even though we exclude it from bundling (so it stays |
There was a problem hiding this comment.
we exclude it from bundling
I think it'd be much easier if we could just include the rocdecode .so with the torchcodec wheel. That's what we do for every other image decoder (jpeg, png, avif, etc.) including nvjpeg. It would also avoid the rpath setting below which I'm concerned about. Is this something we can do?
There was a problem hiding this comment.
@NicolasHug: Bundling librocjpeg like nvJPEG is what we’d prefer for symmetry, but moving it breaks TheRock-relative RPATHs inside librocjpeg; that’s why we RPATH instead. librocjpeg is built to live in a specific folder. Moreover for ROCm setting RPATH is common and we do it for PyTorch as well https://github.com/pytorch/pytorch/blob/fbab7cc90f5699714d5785aaa2f153c739435d6f/.ci/wheel/linux/repair_wheel.py#L102
Use python-version / rocm-version in the install-and-test matrix for consistency with other torchcodec workflows. Remove temporary rocJPEG debug logging from DecodeJpegRocm. Co-authored-by: Cursor <cursoragent@cursor.com>
Clarify that the filtered matrix feeds the wheel build job, matching the build-matrix input name used by other torchcodec workflows. Co-authored-by: Cursor <cursoragent@cursor.com>
Creating this PR to-
Validation-
With all the changes we got successful build and test run https://github.com/meta-pytorch/torchcodec/actions/runs/38807500701/job/116699414031?pr=1642
Also, obtained successful run for a temporary change made to simulate merge to main so the build matrix includes all nightly Pythons (3.11 … 3.15t).
https://github.com/meta-pytorch/torchcodec/actions/runs/38821460509/job/116726844584?pr=1642