Skip to content

[ROCm] Test ROCm failures on PR#1554 - #1642

Open
akashveramd wants to merge 81 commits into
rocm_decoderfrom
rocm_decoder_analysis
Open

akashveramd wants to merge 81 commits into
rocm_decoderfrom
rocm_decoder_analysis

Conversation

@akashveramd

@akashveramd akashveramd commented Aug 18, 2026 •

Copy link
Copy Markdown
Collaborator

Creating this PR to-

Validation-
With all the changes we got successful build and test run https://github.com/meta-pytorch/torchcodec/actions/runs/38807500701/job/116699414031?pr=1642

Also, obtained successful run for a temporary change made to simulate merge to main so the build matrix includes all nightly Pythons (3.11 … 3.15t).
https://github.com/meta-pytorch/torchcodec/actions/runs/38821460509/job/116726844584?pr=1642

@akashveramd akashveramd self-assigned this Aug 18, 2026
@pytorch-bot

pytorch-bot Bot commented Aug 18, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1642

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit a324f14 with merge base d249ad5 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 18, 2026
Co-authored-by: Cursor <cursoragent@cursor.com>
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch from 35e74f6 to 61016ce Compare August 18, 2026 23:21
akashveramd and others added 7 commits August 18, 2026 23:52
librocjpeg links against libva.so.2 and libva-drm.so.2 (VA-API, used
by rocJPEG's HYBRID GPU-JPEG backend).  auditwheel was pulling both
into the wheel because they weren't excluded, and repair_wheel.py's
check_bundling() then rejected them as unexpected.

libva/libva-drm are system-provided display-stack libraries, present
on any ROCm install alongside libdrm (which is already excluded).
Add "libva*" to the same exclude list so auditwheel leaves them on
the system rather than bundling them.

Co-authored-by: Cursor <cursoragent@cursor.com>
ROCm >= 7.14 distributes the full ROCm stack (including rocJPEG) as pip
wheels (_rocm_sdk_core / _rocm_sdk_devel site-packages) rather than
system RPMs. The rocjpeg-devel, libva-amdgpu and mesa-amdgpu-va-drivers
DNF packages therefore don't exist on the ROCm 7.14 builder image and
the install was failing with "No package rocjpeg available".

Check for librocjpeg.so under /opt/conda (pip-wheel install path) and
/opt/rocm (classic RPM install path) before attempting dnf install. If
already present, skip the install entirely. The existing dnf path is
preserved for ROCm 7.2 and earlier where RPMs are the only source.

Co-authored-by: Cursor <cursoragent@cursor.com>
librocjpeg links libva.so.2 as a DT_NEEDED entry and needs it at
dlopen time even when only the HARDWARE backend is used. The previous
fix skipped all dnf installs when librocjpeg was found via ROCm 7.14
pip wheels, which left libva absent on the test machine and caused
libtorchcodec_image.so to fail to load with OSError.

libva ships in AlmaLinux standard repos so install it unconditionally.

Co-authored-by: Cursor <cursoragent@cursor.com>
_find_rocjpeg_lib() only searched ROCM_HOME/lib, torch's ROCM_HOME, and
/opt/rocm/lib.  For ROCm >= 7.14, librocjpeg lives inside the
_rocm_sdk_core pip-wheel's site-packages (e.g.
/opt/conda/lib/python3.11/site-packages/_rocm_sdk_core/lib/librocjpeg.so),
none of which were on those search paths.  The function therefore returned
None, printed a WARNING, and auditwheel never bundled librocjpeg.  At test
time libtorchcodec_image.so had DT_NEEDED: librocjpeg.so but no copy was
on the dynamic-linker search path → OSError: Could not load this library.

Fix: add a site-packages-first fallback that checks _rocm_sdk_core/lib and
_rocm_sdk_devel/lib in the current interpreter's site-packages, then falls
back to a broad /opt/conda/** glob (mirrors install_rocjpeg.sh).

Also add ldd diagnostics before pytest in the install-and-test workflow so
that any future missing transitive dep is visible in the log rather than
hidden behind a bare OSError, and update the stale comment that said the
wheel does not bundle librocjpeg.

Co-authored-by: Cursor <cursoragent@cursor.com>
…yout

PyTorch ROCm wheels may bundle the ROCm runtime (libamdhip64, libhsa-runtime64
etc.) inside torch/lib/ rather than depending on a separate _rocm_sdk_core pip
wheel. Add \$ORIGIN/../../torch/lib to the RPATH patched onto the bundled
librocjpeg-HASH.so so it can find those deps in either layout:
  - torch/lib/ (torch bundles ROCm runtime, like it does for CUDA)
  - _rocm_sdk_core/lib/ (TheRock separate pip-wheel layout)
  - /opt/rocm/lib (ROCm <= 7.2 system install)

Co-authored-by: Cursor <cursoragent@cursor.com>
@akashveramd
akashveramd force-pushed the rocm_decoder_analysis branch from 7435c2f to 2efd33f Compare August 19, 2026 16:50
akashveramd and others added 12 commits August 19, 2026 17:32
…nostics

The bundled librocjpeg-HASH.so has DT_NEEDED entries for
librocm_sysdeps_va.so.2, librocm_sysdeps_drm_amdgpu.so.1,
librocprofiler-register.so.0 etc. These live in _rocm_sdk_core/lib/ and
_rocm_sdk_core/lib/rocm_sysdeps/lib/ but are not on the system search path
and are not loaded by torch at import time. The RPATH on the bundled lib
uses $ORIGIN which ldd cannot expand without execute permission (ldd shows
all $ORIGIN-based entries as "not found"). Whether $ORIGIN resolves
correctly at actual dlopen time is unclear, so set LD_LIBRARY_PATH
explicitly before pytest as belt-and-suspenders.

Also fix the rocjpeg_so find path (torchcodec.libs/ is a sibling of
torchcodec/, not a child) and add objdump RPATH output so we can verify
the RPATH was patched correctly on the bundled librocjpeg.

Co-authored-by: Cursor <cursoragent@cursor.com>
Bundling librocjpeg moves it out of _rocm_sdk_core/lib, breaking AMD's
own relative RPATH inside it that points to librocm_sysdeps_* transitive
deps. We then have to re-patch those paths ourselves, which is fragile
and was causing the missing librocm_sysdeps_va.so.2 / libamdhip64.so.7
failures at runtime.

The cleaner approach: exclude librocjpeg from auditwheel bundling and
instead append _rocm_sdk_core/lib (ROCm >= 7.14) and /opt/rocm/lib
(ROCm <= 7.2) to the RPATH of libtorchcodec_image.so itself. At runtime
the dynamic linker finds librocjpeg in its original location; AMD's own
RPATH on that library then correctly resolves all transitive deps without
any intervention from us.

_find_rocjpeg_lib() is kept: auditwheel still needs librocjpeg in
LD_LIBRARY_PATH to resolve the DT_NEEDED during its analysis pass, even
though we exclude it from bundling.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ness

Co-authored-by: Cursor <cursoragent@cursor.com>
check_bundling() previously required librocjpeg to be bundled in every
ROCm wheel, which is now intentionally wrong. Update it to:
- Treat "not bundled" as correct for ROCm wheels
- Instead verify that libtorchcodec_image.so has _rocm_sdk_core/lib in
  its RPATH (confirming _patch_image_so_rpath_in_wheel ran correctly)
- Raise if librocjpeg IS bundled (that would be a regression)
- Remove _is_rocjpeg from the bundled-lib allowlist for the same reason
- Add top-level `import tempfile` (used by the new RPATH check)

Co-authored-by: Cursor <cursoragent@cursor.com>
… rocjpeg

- bundle_third_party_licenses: warn (not raise) if rocjpeg LICENSE is
  missing, since librocjpeg is no longer redistributed in the wheel
- check_bundling license check: only enforce rocjpeg keyword if the
  license file was actually bundled (it's optional, not mandatory)
- Fix stale "ROCm wheels bundle librocjpeg" comment
- Fix stale _find_rocjpeg_license docstring

Co-authored-by: Cursor <cursoragent@cursor.com>
$ORIGIN/../../_rocm_sdk_core/lib resolves to python3.x/_rocm_sdk_core/lib
which does not exist.  The correct path is $ORIGIN/../_rocm_sdk_core/lib:
libtorchcodec_image.so lives in site-packages/torchcodec/, so one ../ brings
us to site-packages/ where _rocm_sdk_core/ lives.  This matches auditwheel's
own convention: it sets $ORIGIN/../torchcodec.libs for the same reason.

Co-authored-by: Cursor <cursoragent@cursor.com>
rocJPEG's HYBRID backend calls vaInitialize() to initialize the AMD
GPU video decoder. libva.so.2 alone is not enough; it also needs the
AMD VA-API backend driver (mesa-amdgpu-va-drivers) which provides the
radeonsi/amdgpu DRI plugin that vaInitialize() dlopen-s at runtime.

Without it, vaInitialize() fails and rocJPEG returns
ROCJPEG_STATUS_NOT_INITIALIZED / ROCJPEG_STATUS_NOT_IMPLEMENTED,
causing all jpeg_cuda tests to fail with "Failed to initialize rocJPEG
with the hybrid backend".

Install mesa-amdgpu-va-drivers alongside libva in the ROCm 7.14
pip-wheel path; fall back to libva-only if the AMD graphics repo is
not available (build-only runners).

Co-authored-by: Cursor <cursoragent@cursor.com>
Since ROCm 7.14, mesa (including the amdgpu/radeonsi VA-API backend
driver) ships inside the _rocm_sdk_core pip wheel. There is no need
to install mesa-amdgpu-va-drivers via dnf.

However, libva's driver discovery (vaInitialize) needs to know where
to find the DRI driver plugin. Set LIBVA_DRIVERS_PATH to
_rocm_sdk_core/lib/dri so the vendored librocm_sysdeps_va.so.2 can
find the bundled mesa backend, fixing the ROCJPEG_STATUS_NOT_INITIALIZED
/ ROCJPEG_STATUS_NOT_IMPLEMENTED failures.

Also add a fallback diagnostic that lists _rocm_sdk_core/lib/ if the
dri subdirectory is not found, to help debug future layout changes.

Reverts the mesa-amdgpu-va-drivers dnf install added in the previous
commit (442ac6c) as it is not needed.

Co-authored-by: Cursor <cursoragent@cursor.com>
The multi-line python block inside $() in the YAML run: | block had
lines starting at column 0, which breaks the YAML scalar parser (same
issue as before). Collapse to a single line.

Also improve the *_drv_video.so search: use rglob instead of checking
a hardcoded lib/dri/ path so the mesa VA-API backend driver is found
regardless of the exact subdirectory AMD chose inside _rocm_sdk_core.

The fallback diagnostic now greps the full recursive listing for
dri/va/video/mesa/gallium/radeon keywords to pinpoint the layout if
the driver isn't found.

Co-authored-by: Cursor <cursoragent@cursor.com>
…iver

libva auto-detects the DRM driver name as "radeonsi" from the kernel and
looks for radeonsi_drv_video.so. AMD vendored it as
librocm_sysdeps_gallium_drv_video.so, so LIBVA_DRIVER_NAME must be set to
"librocm_sysdeps_gallium" to override that. Verified locally: vaInitialize
returns 0 with both LIBVA_DRIVERS_PATH and LIBVA_DRIVER_NAME set.

Co-authored-by: Cursor <cursoragent@cursor.com>
1. test/utils.py: assert_frames_equal crashes with TypeError when
   cuda_version_used_for_building_torch() returns None on ROCm. Handle
   None explicitly, treating it as >= CUDA 13 (use atol=3).

2. DecodeJpegRocm: ROCJPEG_OUTPUT_RGB_PLANAR via the HW (VCN) backend
   produces ~51% correct pixels for colour (YCbCr) JPEG sources on
   MI350X: the hardware returns unconverted YCbCr planes instead of RGB.
   Fix: set force_hybrid=true in make_plan() for colour JPEGs requesting
   RGB output, routing them to the HYBRID backend which handles
   YCbCr->RGB in software.

3. DecodeJpegRocm: rocJpegDecodeBatched writes with an internally-aligned
   pitch that does not match our tensor row stride, producing completely
   wrong output when the batch mixes images of different dimensions.
   Fix: replace rocJpegDecodeBatched with individual rocJpegDecode calls
   in decode_batched_hardware().

4. test_decoders.py: test_cuda_jpeg_errors expects corrupt JPEG input to
   raise RuntimeError. rocJPEG silently "succeeds" on this corrupt JPEG
   (nvJPEG raises, rocJPEG does not). Skip the corrupt-JPEG assertion on
   ROCm; the CPU-tensor-on-GPU error check still runs.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread packaging/repair_wheel.py Resolved
Comment thread packaging/install_rocjpeg.sh Resolved
LakshmiKumar23 added a commit to LakshmiKumar23/torchcodec that referenced this pull request Sep 8, 2026
These scripts are no longer needed with the TheRock/pip-wheel layout
introduced in PR meta-pytorch#1642. rocDecode and RPP are now provided by the
_rocm_sdk_core and _rocm_sdk_devel pip packages.

Added ROCM_ROCDECODE_RPP_NOTES.md documenting the repair_wheel.py
changes needed after PR meta-pytorch#1642 lands (adding librocdecode* and librpp*
to the auditwheel exclude list).

Co-Authored-By: Claude <noreply@anthropic.com>
The nightly index and builder image publish 10.1, so the wheel build drops 10.0 and the GPU job installs torch and rocm[devel] from that index.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread .github/workflows/linux_rocm.yaml Resolved
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Comment thread .github/workflows/linux_rocm.yaml Resolved
Comment thread .github/workflows/linux_rocm.yaml Resolved
AryanSalmanpour and others added 7 commits October 14, 2026 19:50
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
rocJPEG needs ROCm 10.1 or newer. Drop the channel-specific rewrite and filter the nightly matrix with a numeric comparison.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use the first Python version and each ROCm version from that matrix so the download artifact and install indexes match the built wheel.

Co-authored-by: Cursor <cursoragent@cursor.com>
The manylinux ROCm image has no system rocm-smi. Put the SDK bin dirs on PATH and prefer amd-smi once the devel tree is initialized.

Co-authored-by: Cursor <cursoragent@cursor.com>
Simulates merge-to-main wheel coverage on this PR by undoing test-infra LIMIT_PR_BUILDS. Revert this commit before landing; the GPU test job still uses only the first Python.

Co-authored-by: Cursor <cursoragent@cursor.com>
Comment thread .github/workflows/linux_rocm.yaml Resolved
Comment thread .github/workflows/linux_rocm.yaml Resolved
Comment thread .github/workflows/linux_rocm.yaml Resolved
Comment thread .github/workflows/linux_rocm.yaml Resolved
akashveramd and others added 8 commits October 16, 2026 14:02
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Co-authored-by: Jithun Nair <37884920+jithunnair-amd@users.noreply.github.com>
Isolated diagnostic for whether rocJPEG resolves via $ORIGIN RPATH on ROCm 10.1, so it can be reverted once VA-API init is understood.

Co-authored-by: Cursor <cursoragent@cursor.com>
A/B whether setting ROCM_PATH alone fixes rocJPEG InitVAAPI; revert if it does not help.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keep exporting ROCM_PATH from _rocm_sdk_devel; it is required for
rocJPEG InitVAAPI. Fail the job if the devel package is missing.

Co-authored-by: Cursor <cursoragent@cursor.com>
TheRock documents this as the canonical way to locate the devel tree;
avoid hardcoding the _rocm_sdk_devel package layout.

Co-authored-by: Cursor <cursoragent@cursor.com>
build-python-only: "disable"

build:
filter-matrix:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be a selector available from the test-infra's generate-matrix job? If not, we should add it there instead of having this logic here

@jithunnair-amd jithunnair-amd Nov 3, 2026 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NicolasHug Would you want us to get rid of the entire filtering logic in filter-matrix and move it to generate-matrix? If so, we will need the ability to get a single-python-version matrix from generate_binary_build_matrix.py, since torchcodec (and many other repos might) only wants to run tests for a single python version, regardless of PR or main branch commit. And we are trying to not hardcode the python version in the yml file, so that it auto-bumps (less maintenance, even for CUDA workflows).

But currently, generate_binary_build_matrix.py returns a single-python-version matrix only for PR-triggers, without any ability for the user to say "always give me just the first python version".

@huydhn Would you be okay with this suggestion:

  1. Introduce an optional python-version-limit workflow input to generate_binary_build_matrix.py that defaults to auto (current default). Caller can alternatively set it to first, which will limit the python versions to the first one in the matrix.
python-versions-limit:
  description: "auto: all Python versions, except pull requests without ciflow/binaries/all get only the first. first: only the first Python version on every event."
  default: "auto"
  type: string
  1. Feed it into the existing LIMIT_PR_BUILDS expression so first forces the reduced matrix even when the label is present, and auto leaves every current caller unchanged:
    LIMIT_PR_BUILDS: ${{ inputs.python-versions-limit == 'first' || (github.event_name == 'pull_request' && !contains(github.event.pull_request.labels.*.name, 'ciflow/binaries/all')) }}

Comment thread .github/workflows/linux_rocm.yaml Resolved
Comment thread src/torchcodec/_core/DecodeJpegRocm.cpp Resolved
Comment thread packaging/repair_wheel.py


def _patch_image_so_rpath_in_wheel(wheel_path: Path) -> None:
"""Append ROCm library search paths to libtorchcodec_image.so's RPATH.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wary of modifying rpath here. Does that mean we expect the user to have a rocm installation via the TheRock/pip-wheel layout, or does it support more general setups?

What are the .so files that live in the folders that we now set as rpath - are there risks that some of those .so might shadow those from the user's system (libc.so etc.)?

@akashveramd akashveramd Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NicolasHug: Currently we only handle TheRock/pip ROCm. Other ROCm installs (e.g. tarball) are not covered by the RPATH we set.

TheRock separates ROCm libs and System/sysdeps libs. ROCm libs are located under- _rocm_sdk_core/lib and System/sysdeps libs are located under- _rocm_sdk_core/lib/rocm_sysdeps/lib.
We set RPATH on libtorchcodec_image.so pointing at the ROCm libs directory only ($ORIGIN/../_rocm_sdk_core/lib), and do not add the sysdeps path. Hence, there is no risk of user's system libs getting shadowed.
Please see the attached screen shot which lists ROCm libs.

ROCm_libs

Comment thread packaging/repair_wheel.py
"""Find librocjpeg.so at wheel repair time so auditwheel can resolve it.

auditwheel needs librocjpeg to be resolvable in LD_LIBRARY_PATH during
`auditwheel repair` even though we exclude it from bundling (so it stays

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we exclude it from bundling

I think it'd be much easier if we could just include the rocdecode .so with the torchcodec wheel. That's what we do for every other image decoder (jpeg, png, avif, etc.) including nvjpeg. It would also avoid the rpath setting below which I'm concerned about. Is this something we can do?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@NicolasHug: Bundling librocjpeg like nvJPEG is what we’d prefer for symmetry, but moving it breaks TheRock-relative RPATHs inside librocjpeg; that’s why we RPATH instead. librocjpeg is built to live in a specific folder. Moreover for ROCm setting RPATH is common and we do it for PyTorch as well https://github.com/pytorch/pytorch/blob/fbab7cc90f5699714d5785aaa2f153c739435d6f/.ci/wheel/linux/repair_wheel.py#L102

akashveramd and others added 2 commits November 3, 2026 18:59
Use python-version / rocm-version in the install-and-test matrix for
consistency with other torchcodec workflows. Remove temporary rocJPEG
debug logging from DecodeJpegRocm.

Co-authored-by: Cursor <cursoragent@cursor.com>
Clarify that the filtered matrix feeds the wheel build job, matching
the build-matrix input name used by other torchcodec workflows.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/rocm CLA Signed This label is managed by the Meta Open Source bot. module: rocm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants