[https://nvbugs/6523820][fix] Waive NCCL-EP dispatch-only CUDA graph replay test by nv-lschneider · Pull Request #16959 · NVIDIA/TensorRT-LLM

nv-lschneider · 2026-07-28T16:30:48Z

Dev Engineer Review

Temporarily waived test_nccl_ep_cuda_graph_replay_uses_updated_routing with @pytest.mark.skip.
The waiver cites the pending NCCL-EP CUDA-graph replay fix tracked by NVBug 6523820.
The change is limited to one regression test. No API, dependency, or production-code changes are included.
Validation includes git diff --check, Ruff, and Ruff-format pre-commit hooks.

QA Engineer Review

Modified test: tests/unittest/_torch/modules/moe/test_moe_comm.py::test_nccl_ep_cuda_graph_replay_uses_updated_routing.
The test is skipped pending the NCCL-EP replay fix.
No tests/integration/test_lists/, test-db/, or qa/ files changed.
No CI test-list coverage entry was added or removed.
Verdict: sufficient.

Description

Temporarily waive TestMoEComm.test_nccl_ep_cuda_graph_replay_uses_updated_routing.

The test captures and replays an NCCL-EP low-latency dispatch-only CUDA graph while changing routing between replays. This exposes an NCCL-EP internal ping-pong-buffer replay issue and is tracked to be fix soon in NCCL-EP.

This does not affect TRT-LLM’s standard CUDA-graph execution path: TRT-LLM captures the complete MoE forward path, including both NCCL-EP dispatch and combine. Those paired operations advance the internal buffer state together.

The regression test remains in-tree and will be re-enabled after updating to an NCCL-EP release containing the fix.

Test Coverage

Retains the dispatch-only CUDA-graph regression test, temporarily skipped pending NVBug 6523820.
git diff --check
ruff and ruff-format pre-commit hooks passed.

PR Checklist

No API changes or dependencies.
No documentation or CODEOWNERS updates required.
Existing regression coverage retained for future re-enablement.

nv-lschneider · 2026-07-28T16:32:10Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

coderabbitai · 2026-07-28T16:33:01Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8fb549fb-5d09-4445-809f-4b36234d488a

📥 Commits

Reviewing files that changed from the base of the PR and between 8e588da and ee8e18e.

📒 Files selected for processing (1)

tests/unittest/_torch/modules/moe/test_moe_comm.py

🚧 Files skipped from review as they are similar to previous changes (1)

tests/unittest/_torch/modules/moe/test_moe_comm.py

Walkthrough

The NCCL-EP CUDA-graph replay routing test is marked with pytest.mark.skip, including a waiver reason, so pytest does not execute it.

Changes

NCCL-EP test waiver

Layer / File(s)	Summary
Skip CUDA-graph replay verification `tests/unittest/_torch/modules/moe/test_moe_comm.py`	Adds a skip decorator with a waiver reason to `test_nccl_ep_cuda_graph_replay_uses_updated_routing`.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17352: Updates CUDA-graph runner and replay memory references related to this waived test.

Suggested labels: ci: full pre-merge approved

Suggested reviewers: rosong11

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title identifies NVBug 6523820, uses the valid fix type, and clearly states that the NCCL-EP dispatch-only CUDA graph replay test is waived.
Description check	✅ Passed	The description explains the issue, scope, resolution, test coverage, and checklist items, and it matches the pull request objectives.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-28T16:44:38Z

PR_Github #62235 [ run ] triggered by Bot. Commit: 00c9826 Link to invocation

tensorrt-cicd · 2026-07-28T20:08:23Z

PR_Github #62235 [ run ] completed with state FAILURE. Commit: 00c9826
/LLM/main/L0_MergeRequest_PR pipeline #50404 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-28T20:13:52Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-28T20:19:37Z

PR_Github #62295 [ run ] triggered by Bot. Commit: 00c9826 Link to invocation

tensorrt-cicd · 2026-07-28T21:35:08Z

PR_Github #62295 [ run ] completed with state FAILURE. Commit: 00c9826
/LLM/main/L0_MergeRequest_PR pipeline #50464 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-28T21:47:58Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-28T22:03:53Z

PR_Github #62313 [ run ] triggered by Bot. Commit: 00c9826 Link to invocation

tensorrt-cicd · 2026-07-28T23:50:42Z

PR_Github #62313 [ run ] completed with state FAILURE. Commit: 00c9826
/LLM/main/L0_MergeRequest_PR pipeline #50482 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-30T12:24:27Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-30T12:30:05Z

PR_Github #62769 [ run ] triggered by Bot. Commit: 00c9826 Link to invocation

tensorrt-cicd · 2026-07-30T13:19:49Z

PR_Github #62769 [ run ] completed with state SUCCESS. Commit: 00c9826
/LLM/main/L0_MergeRequest_PR pipeline #50898 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-31T01:21:03Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-31T01:26:53Z

PR_Github #62899 [ run ] triggered by Bot. Commit: 00c9826 Link to invocation

tensorrt-cicd · 2026-07-31T02:05:37Z

PR_Github #62899 [ run ] completed with state SUCCESS. Commit: 00c9826
/LLM/main/L0_MergeRequest_PR pipeline #51020 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-31T12:13:48Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-31T12:19:24Z

PR_Github #63034 [ run ] triggered by Bot. Commit: 00c9826 Link to invocation

tensorrt-cicd · 2026-07-31T13:34:45Z

PR_Github #63034 [ run ] completed with state SUCCESS. Commit: 00c9826
/LLM/main/L0_MergeRequest_PR pipeline #51138 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-31T19:27:14Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-31T19:34:07Z

PR_Github #63108 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

tensorrt-cicd · 2026-07-31T19:57:14Z

PR_Github #63108 [ run ] completed with state FAILURE. Commit: bd6aa12
/LLM/main/L0_MergeRequest_PR pipeline #51198 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-07-31T20:37:32Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-07-31T20:45:13Z

PR_Github #63123 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

tensorrt-cicd · 2026-07-31T23:08:57Z

PR_Github #63123 [ run ] completed with state FAILURE. Commit: bd6aa12
/LLM/main/L0_MergeRequest_PR pipeline #51212 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-08-03T13:24:18Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-08-03T13:32:09Z

PR_Github #63454 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

tensorrt-cicd · 2026-08-03T13:43:07Z

PR_Github #63454 [ run ] completed with state FAILURE. Commit: bd6aa12
/LLM/main/L0_MergeRequest_PR pipeline #51426 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-08-03T15:12:25Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-08-03T15:18:55Z

PR_Github #63475 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

tensorrt-cicd · 2026-08-03T17:16:37Z

PR_Github #63475 [ run ] completed with state SUCCESS. Commit: bd6aa12
/LLM/main/L0_MergeRequest_PR pipeline #51446 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-08-03T19:46:20Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-08-03T19:52:41Z

PR_Github #63547 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

tensorrt-cicd · 2026-08-03T21:31:46Z

PR_Github #63547 [ run ] completed with state SUCCESS. Commit: bd6aa12
/LLM/main/L0_MergeRequest_PR pipeline #51513 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

nv-lschneider · 2026-08-05T13:44:23Z

/bot run --extra-stage "GB200-4_GPUs-PyTorch-4"

tensorrt-cicd · 2026-08-05T13:50:32Z

PR_Github #64061 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

tensorrt-cicd · 2026-08-05T17:06:57Z

PR_Github #64061 [ run ] completed with state SUCCESS. Commit: bd6aa12
/LLM/main/L0_MergeRequest_PR pipeline #51990 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

nv-lschneider · 2026-08-05T18:28:35Z

/bot run --disable-reuse-test --stage-list "GB200-4_GPUs-PyTorch-1, GB200-4_GPUs-PyTorch-2, GB200-4_GPUs-PyTorch-3, GB200-4_GPUs-PyTorch-4, GB200-4_GPUs-PyTorch-5"

tensorrt-cicd · 2026-08-05T18:34:49Z

PR_Github #64094 [ run ] triggered by Bot. Commit: bd6aa12 Link to invocation

Signed-off-by: Ludwig Schneider <lschneider@nvidia.com>

coderabbitai · 2026-08-06T12:46:05Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

nv-lschneider · 2026-08-06T12:46:20Z

/bot run --stage-list "GB200-4_GPUs-PyTorch-*"

tensorrt-cicd · 2026-08-06T12:53:10Z

PR_Github #64347 [ run ] triggered by Bot. Commit: ee8e18e Link to invocation

tensorrt-cicd · 2026-08-06T12:53:16Z

PR_Github #64094 [ run ] completed with state ABORTED. Commit: bd6aa12

Link to invocation

tensorrt-cicd · 2026-08-06T13:53:24Z

PR_Github #64347 [ run ] completed with state FAILURE. Commit: ee8e18e
/LLM/main/L0_MergeRequest_PR pipeline #52241 (Partly Tested) completed with status: 'UNSTABLE'

CI Report

⚠️ Multi-GPU Label Required:
Multi-GPU tests require the ci: full pre-merge approved label on this PR. Ask a member of NVIDIA/trt-llm-ci-approvers to add the label, then re-trigger CI with the same bot command (no rebase needed).

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

nv-lschneider requested a review from a team as a code owner July 28, 2026 16:30

nv-lschneider requested review from rosong11 and xxi-nv July 28, 2026 16:30

github-actions Bot assigned nv-lschneider Jul 28, 2026

nv-lschneider force-pushed the fix/waive-nccl-ep-cuda-graph-replay-6523820 branch from fe6e8a8 to 00c9826 Compare July 28, 2026 16:32

nv-lschneider changed the title ~~[https://nvbugspro.nvidia.com/bug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test~~ [https://nvbugspro.nvidia.com/bug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test Jul 28, 2026

nv-lschneider changed the title ~~[https://nvbugspro.nvidia.com/bug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test~~ [https://nvbug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test Jul 28, 2026

nv-lschneider changed the title ~~[https://nvbug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test~~ [https://nvbug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test Jul 28, 2026

nv-lschneider changed the title ~~[https://nvbug/6523820][test] Waive NCCL-EP dispatch-only CUDA graph replay test~~ [https://nvbug/6523820][fix] Waive NCCL-EP dispatch-only CUDA graph replay test Jul 28, 2026

nv-lschneider changed the title ~~[https://nvbug/6523820][fix] Waive NCCL-EP dispatch-only CUDA graph replay test~~ [https://nvbugs/6523820][fix] Waive NCCL-EP dispatch-only CUDA graph replay test Jul 28, 2026

nv-lschneider mentioned this pull request Jul 30, 2026

[https://nvbugs/6517844][fix] Fall back to DeepEP when NCCL-EP lacks shared memory #16956

Merged

9 tasks

xxi-nv approved these changes Jul 31, 2026

View reviewed changes

nv-lschneider force-pushed the fix/waive-nccl-ep-cuda-graph-replay-6523820 branch from 00c9826 to bd6aa12 Compare July 31, 2026 19:26

[None][test] waive NCCL-EP CUDA graph replay test

ee8e18e

Signed-off-by: Ludwig Schneider <lschneider@nvidia.com>

nv-lschneider force-pushed the fix/waive-nccl-ep-cuda-graph-replay-6523820 branch from bd6aa12 to ee8e18e Compare August 6, 2026 12:45

brnguyen2 added the ci: full pre-merge approved label Aug 7, 2026

Uh oh!

Conversation

nv-lschneider commented Jul 28, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

Uh oh!

nv-lschneider commented Jul 28, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

nv-lschneider commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

nv-lschneider commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

nv-lschneider commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

tensorrt-cicd commented Jul 30, 2026

Uh oh!

nv-lschneider commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

nv-lschneider commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

nv-lschneider commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

nv-lschneider commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

tensorrt-cicd commented Jul 31, 2026

Uh oh!

nv-lschneider commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

nv-lschneider commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

nv-lschneider commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

nv-lschneider commented Aug 5, 2026

nv-lschneider commented Jul 28, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading