fix(update_weight): restore FlashInfer MoE layout after BF16 hot updates#2192
Open
LLMShark wants to merge 4 commits into
Open
fix(update_weight): restore FlashInfer MoE layout after BF16 hot updates#2192LLMShark wants to merge 4 commits into
LLMShark wants to merge 4 commits into
Conversation
LLMShark
force-pushed
the
fix/bf16-weight-postprocess
branch
from
July 10, 2026 10:31
f453551 to
0a9eeff
Compare
LLMShark
force-pushed
the
fix/bf16-weight-postprocess
branch
from
July 10, 2026 13:19
0a9eeff to
0af60fc
Compare
Author
|
Hi maintainers, just a gentle ping on this PR. It fixes the missing post-load finalization step for BF16 weight hot updates, and the CI checks are passing. I would appreciate a review when you have time. Thanks! |
Author
|
@huang3eng Please help review~ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
During BF16 hot updates with the
flashinfer_trtllmMoE backend, SGLang restores runtime block-layout weights to canonical layout before copying updated weights. After the copy, the weights must be finalized back into the backend-specific runtime layout.slime previously ran the post-load
post_process_weights(...)step only forcompressed-tensors. Unquantized/BF16 updates skipped it, leaving FlashInfer TRT-LLM MoE weights in canonical layout for the next forward pass.Reproduction and before/after
Environment:
flashinfer_trtllmMoE backendWith the corresponding SGLang pre-load canonical-shape restore fix applied, the BF16 hot-update copy completed successfully. Without this PR, the next generation forward pass failed with:
The updated MoE weights remained in canonical layout, while the FlashInfer TRT-LLM kernel expected its runtime block layout.
After applying this PR together with the SGLang restore fix:
ref_log_probs, andactor_trainiterations completed.This comparison validates the restore-copy-finalize control flow and crash fix. It does not by itself establish logit or log-probability parity between the training and rollout engines.
Changes
Updated both hot-update implementations:
slime/backends/megatron_utils/update_weight/update_weight_from_tensor.pyslime/backends/megatron_utils/update_weight/update_weight_from_distributed.pyAdded a shared
requires_post_process_after_update(...)predicate with an explicit scope:quantization_configNonecompressed-tensorsThe predicate deliberately checks
quantization_config is Noneinstead of treating every falsy value as an unquantized model. It also avoids assuming that every quantization backend'sprocess_weights_after_loadinghook is a safe or idempotent no-op.Additional behavior preserved:
compressed-tensorspre-load restore call remains conditional.continue_generationruns only after required post-processing succeeds.Relationship to SGLang #28015
This PR and SGLang #28015 address different phases of the update lifecycle:
flashinfer_trtllmBF16 MoE.For the non-routed backend, both changes, or equivalent downstream patches, are required:
Tests
tests/test_bf16_weight_post_process.pycovers:None,compressed-tensors, FP8, empty, and unknown configurations.compressed-tensorspreserves restore-then-post-process ordering.The test is included in the CPU CI matrix. The existing empty colocated bucket test also provides the shared helper in its isolated fake dependency module.
Validation
git diff --checkpassed.tests/test_empty_colocated_weight_bucket.py: 2 passed locally.pre-commitpassed onfb35d5e.PR Testpassed onfb35d5e.