[https://nvbugs/6544407][fix] unwaive glm5 stress test

[chuangz0]

Dev Engineer Review

• Removed the waiver entry for the B200 GLM5 NVFP4 TP4/EP4/DP stress test.
• The change is limited to tests/integration/test_lists/waives.txt.
• The change has no code, API, performance, or error-handling impact.
• The waiver removal is consistent with the reported inability to reproduce the error on main.
• The test-list format and bug-reference consistency require verification against the file contents.

The error was not reproduced after running the test five times on the b200 x8 node，
link https://prod.blsm.nvidia.com/swqa-tensorrt-qa-test/job/DEBUG_LLM_FUNCTION_CLUSTER_TEST/1845/

QA Engineer Review

• Modified test-list file: tests/integration/test_lists/waives.txt.
• Removed the B200 GLM5 NVFP4 TP4/EP4/DP stress-test waiver.
• No test-db or QA entries were added or modified.
• Verdict: needs follow-up until CBTS coverage data confirms that the test runs without the waiver.

Description

unable to reproduce the error on latest main

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

Walkthrough

The integration waive list removes the skip entry for the B200 GLM5 NVFP4 TP4/EP4/DP stress test.

Changes

Cohort / File(s)	Summary
Waive list update tests/integration/test_lists/waives.txt	Removes the waived skip entry for the B200 GLM5 NVFP4 TP4/EP4/DP stress test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Suggested reviewers: schetlur-nv, qijune

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description states the reproduction result but leaves Test Coverage empty and does not explain the change or validation in enough detail.	Add the relevant GLM5 stress-test validation and briefly explain why removing the waiver resolves the issue.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the NVBugs fix and the removal of the GLM5 stress-test waiver.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[chuangz0]

/bot run

[tensorrt-cicd]

PR_Github #63916 [ run ] triggered by Bot. Commit: fd85d6f Link to invocation

[brnguyen2]

The deletion is correct and complete — the removed entry matches the pytest node ID exactly, and no duplicate or leftover waiver for this test exists elsewhere in test_lists.

The gap is evidence. This test is listed only in tests/integration/test_lists/qa/llm_function_stress.txt:8, not in any test-db/l0_*.yml, so /bot run on this PR will not execute it — nothing in CI can confirm the unwaive. Combined with what the test does (tests/integration/defs/disaggregated/test_disaggregated.py:2807: 8×B200, 35k requests at concurrency 512, 0.90 accuracy gate, 10% request cancellation), "unable to reproduce the error on latest main" is thin: that profile fails intermittently, which is usually why it gets waived in the first place.

Before merging, please add to the description: how many runs of disaggregated/test_disaggregated.py::test_disaggregated_stress_test[input8k-output1k-conc512-glm5_nvfp4_tp4_ep4_dp_stress] you did on main, and a link to at least one passing run. Also state the disposition of https://nvbugs/6544407 — if the failure is genuinely gone the bug should be closed (or closed as not-repro) in the same pass, otherwise the waiver and the bug drift apart and the next person hitting it has no trail. If the root cause was actually fixed by a specific commit, naming it would make this a one-line approve.

[brnguyen2]

Worth noting for whoever reviews the QA side: the two sibling stress waivers here (deepseek_r1_v2_fp4_stress and qwen3_32b_fp8_stress, both https://nvbugs/6472256) stay in place, so this unwaive is genuinely scoped to the GLM5 case — good. But since none of these run in pre-merge CI, the first signal that the GLM5 unwaive was premature will be a red QA stress campaign. If you're not confident the underlying issue is fixed (vs. just not reproducing), consider holding this until a couple of clean QA stress runs land rather than using the campaign as the experiment.

[BowenFu]

Holding on this one — I don't think the evidence retires the risk yet.

nvbugs/6544407 is still open P0 (Dev - Open - To fix, Open issue), and the recorded failure is a hard segfault, not a flaky assertion: worker_ctx_0.log ends in nixlUcxSharedThread::run -> nixlUcxWorker::arm -> ucp_worker_arm -> uct_cuda_base_iface_event_fd_arm -> cuEventQuery, followed by MPI worker rank 0 exited unexpectedly. For a race in a progress thread, one non-reproduction is weak evidence — the bug comment is just "failed to reproduce the error" with no hardware, commit, or iteration count.

The triage on the bug also names a concrete, still-unapplied mitigation: the run starts NixlTransferAgent with num_threads 8 (the default at _torch/disaggregation/native/transfer.py), and you capped TRTLLM_NIXL_NUM_THREADS=1 across the disaggregated unit tests in #16708 for exactly this reason — this integration stress case never got that cap. AIPerf logged 3,439 RequestCancellationError before the crash, so cancellation racing arm() is the stated suspect.

Two ways I'd be happy to approve:

1. Cite the run — GPU/board, commit, and how many stress iterations passed clean; or
2. Land the TRTLLM_NIXL_NUM_THREADS=1 cap (or whatever the real fix is) alongside the unwaive, so the entry comes back with the root cause addressed rather than on non-reproduction.

Nothing here is blocking on my side beyond that evidence — the diff itself is a one-line removal of a full:B200/ post-merge entry.

[tensorrt-cicd]

PR_Github #63916 [ run ] completed with state SUCCESS. Commit: fd85d6f
/LLM/main/L0_MergeRequest_PR pipeline #51853 completed with status: 'SUCCESS'

CI Report

Link to invocation