Skip to content

perf: avoid full-vocab all-gather for draft greedy sampling - #1517

Open
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:exact-vocab-parallel-greedy
Open

perf: avoid full-vocab all-gather for draft greedy sampling#1517
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:exact-vocab-parallel-greedy

Conversation

@sufubao

@sufubao sufubao commented Aug 27, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Draft/MTP proposal generation only consumes greedy argmax tokens, so gathering every vocabulary logit onto every TP rank wastes communication and memory. #1512 addresses the same bottleneck with gathered top-K candidates; this version keeps both token selection and selected-token probability exact over the complete vocabulary.

This is the first half of the previously broader PR. It deliberately covers draft models only. The target-model opt-in, sampling eligibility checks, dense fallback, Gemma4 handling, and generic backend integration are isolated in a follow-up branch.

Design

  • Draft models are detected through the existing is_mtp_draft_model model invariant and use the sparse path automatically.
  • Each TP rank computes its local maximum, local logsumexp, and local argmax, then performs one fixed-size all-gather of three FP32 words per token.
  • Global ties deterministically select the smallest token ID. Argmax IDs are bit-preserved through the FP32 communication payload.
  • ModelOutput.logits remains a real logit tensor. Global IDs and full-vocabulary normalization are carried explicitly as logits_token_ids and logits_logsumexp.
  • Draft argmax and confidence paths reconstruct global token IDs and exact probabilities as exp(selected_logit - full_vocab_logsumexp).
  • CUDA Graph padding/unpadding, Eagle accepted-row selection, dynamic MTP, and DSpark confidence sampling preserve the explicit metadata.
  • Target-model sampling is unchanged in this PR; no feature flag or sampling fallback is introduced here.

Verification

  • 26 focused model-output, Triton reducer, Eagle, and DFlash tests pass.
  • The reducer tests compare token IDs, selected logits, logsumexp, and logprobs against a materialized full vocabulary, including local and cross-rank ties.
  • Black, flake8, and git diff --check pass.

Follow-up

The isolated target-model extension is available as a stacked comparison. It will be proposed to upstream after this draft-only contract lands, so reviewers do not need to review the shared kernel and output plumbing twice.

Related: #1511, #1512.

@sufubao
sufubao force-pushed the exact-vocab-parallel-greedy branch from 5d5e7a5 to 74841a4 Compare August 27, 2026 02:28
@sufubao
sufubao force-pushed the exact-vocab-parallel-greedy branch from 74841a4 to 8d4eaf2 Compare August 27, 2026 04:20
@sufubao sufubao changed the title perf: make vocab-parallel greedy exact and explicit perf: avoid full-vocab all-gather for draft greedy sampling Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant