[https://nvbugs/6567403][fix] Waive hanging MoE multi-GPU tests on Hopper

[xxi-nv]

Summary

Waive three MoE multi-GPU unit tests on Hopper only, tracked by NVBug 6567403.

On DGX H100 (SM90) these three cases hang instead of finishing in their usual ~1 minute. Each is killed by the inner pytest --timeout=2400 after ~2427s and reported as FAILED:

• unittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu -k "CUTLASS and W4A16_MXFP4"
• unittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu -k "CUTLASS and W8A16"
• unittest/_torch/modules/moe/test_moe_module.py::test_configurable_moe_multi_gpu_eplb

The accumulated hangs push DGX_H100-4_GPUs-PyTorch-Others-1 past its 240-minute Slurm wall limit, so the stage is scancel'd and never uploads results.xml.

Observed signature (DGX_H100-4_GPUs-PyTorch-Others-1):

...test_configurable_moe_multi_gpu -k "CUTLASS and W4A16_MXFP4"] <- test_unittests.py FAILED [ 20%]
[INFO] Completed 1 cases in the last 2427 seconds

Scope of the waiver

The entries are scoped with full:DGX_H100/, so only the Hopper stage is affected. B200, B300, H200 and GB200/GB300 keep full coverage.

Resolution was verified against the repo's own parse_test_list_lines with the real --test-prefix values used by CI:

--test-prefix	resolves to	matches the failing CI test IDs
DGX_H100-4_GPUs-PyTorch-Others-1	DGX_H100-4_GPUs-PyTorch-Others-1/test_unittests.py::test_unittests_v2[...]	3 / 3
DGX_B200-4_GPUs-PyTorch-Others-1	DGX_H100/test_unittests.py::test_unittests_v2[...]	0 / 3
DGX_H200-4_GPUs-PyTorch-Post-Merge-1	DGX_H100/test_unittests.py::test_unittests_v2[...]	0 / 3
GB300-4_GPUs-PyTorch-Others-1	DGX_H100/test_unittests.py::test_unittests_v2[...]	0 / 3

Test plan

N/A — test waiver only. No product code is touched.

Dev Engineer Review

• Added three DGX_H100 waiver entries in tests/integration/test_lists/waives.txt.
• The entries cover the two CUTLASS MoE configurations and the configurable MoE EPLB test.
• All entries reference nvbugs/6567403.
• The waiver scope is limited to DGX_H100.
• No product code or public API changed.
• The change adds no duplicate or unintended platform coverage.

QA Engineer Review

• Modified tests/integration/test_lists/waives.txt.
• Added three DGX H100 waiver entries:
  • test_configurable_moe_multi_gpu -k "CUTLASS and W4A16_MXFP4"
  • test_configurable_moe_multi_gpu -k "CUTLASS and W8A16"
  • test_configurable_moe_multi_gpu_eplb
• No test code changed.
• CBTS coverage data is unavailable.
• Verdict: needs follow-up.

[coderabbitai]

Walkthrough

The integration test waiver list now skips three configurable multi-GPU MoE tests on DGX_H100. Each skip entry references nvbugs/6567403.

Changes

Configurable multi-GPU MoE test waivers

Layer / File(s)	Summary
DGX_H100 MoE skip entries tests/integration/test_lists/waives.txt	Added skip records for CUTLASS W4A16_MXFP4, CUTLASS W8A16, and configurable MoE EPLB tests. Each record references nvbugs/6567403.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

• NVIDIA/TensorRT-LLM#17276: Updates related GPU integration test waivers.
• NVIDIA/TensorRT-LLM#17294: Updates MoE-related test waivers.
• NVIDIA/TensorRT-LLM#17301: Updates multi-GPU MoE test waivers.

Suggested reviewers: brnguyen2

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the NVBug, fix type, affected MoE multi-GPU tests, and Hopper scope.
Description check	✅ Passed	The description explains the issue, affected tests, Hopper-only scope, validation, and test plan; the checklist is omitted but non-critical.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/test_lists/waives.txt`:
- Around line 202-204: Add the repository-standard NVIDIA copyright header at
the beginning of tests/integration/test_lists/waives.txt, using 2026 as the
year, before the existing waiver entries.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 660873c6-53d2-49d3-bba1-186be3044b2c

📥 Commits

Reviewing files that changed from the base of the PR and between b7a9d6f and 5110068.

📒 Files selected for processing (1)

• tests/integration/test_lists/waives.txt

[xxi-nv]

/bot run --stage-list "DGX_H100-4_GPUs-PyTorch-Others-1"

[tensorrt-cicd]

PR_Github #64255 [ run ] triggered by Bot. Commit: 5110068 Link to invocation

[longlee0622]

/bot skip --comment "premerge hang"

[tensorrt-cicd]

PR_Github #64260 [ skip ] triggered by Bot. Commit: 5110068 Link to invocation

[BowenFu]

Format checks out. full:DGX_H100/ is the right token here: parse_test_name takes the test_prefix.split("-")[0] in s branch, and for stage DGX_H100-4_GPUs-PyTorch-Others-1 that first segment is DGX_H100, so all three entries resolve to exactly the node IDs l0_dgx_h100.yml:176,177,180 generate. full:H100/ would have silently missed this stage ("DGX_H100" is not a substring of H100/...) — that is the usual mis-scoping failure mode, and you avoided it.

No leakage to B200/B300/H200/GB200/GB300 or to single-GPU H100-* stages. Not blocking on it, but this trades away the only DGX-H100 EPLB case plus CUTLASS W8A16/W4A16_MXFP4 — a hang, not a flake — so 6567403 wants a real root-cause, not just the unblock.

[tensorrt-cicd]

PR_Github #64255 [ run ] completed with state ABORTED. Commit: 5110068

Link to invocation

[tensorrt-cicd]

PR_Github #64260 [ skip ] completed with state SUCCESS. Commit: 5110068
Skipping testing for commit 5110068

Link to invocation