Skip to content

perf: enable exact vocab-parallel greedy for target models - #7

Draft
sufubao wants to merge 1 commit into
vocab-parallel-greedy-draftfrom
vocab-parallel-greedy-target
Draft

perf: enable exact vocab-parallel greedy for target models#7
sufubao wants to merge 1 commit into
vocab-parallel-greedy-draftfrom
vocab-parallel-greedy-target

Conversation

@sufubao

@sufubao sufubao commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Stack

This is the clean second layer on top of ModelTC/LightLLM#1517. It is opened in the fork so its diff contains only the target-model extension while ModelTC#1517 is under review. After ModelTC#1517 lands, this change can be proposed to upstream against main without repeating the shared reducer and draft-output contract.

Scope

  • Add the opt-in LIGHTLLM_VOCAB_PARALLEL_GREEDY=1 target-model fast path.
  • Enable it only when every request is unmodified greedy sampling.
  • Retain dense logits for penalties, temperature changes, invalid-token masks, minimum-token EOS masking, constrained/diverse decoding, reward heads, and prompt-logprob capture.
  • Keep CUDA Graph capture and replay on a stable dense-or-sparse output contract.
  • Preserve sparse metadata through DP overlap row concatenation and backend sampling.
  • Apply Gemma4 final-logit softcapping in FP32 on each local vocabulary shard before distributed reduction.

Verification

  • 98 focused model-output, Triton, Gemma4, sampling, and speculative-decoding tests pass.
  • Black, flake8, and git diff --check pass.
  • The previously recorded full-stack Qwen3.5-27B TP=4/MTP=3 run improved median output throughput from 6791.56 to 7593.38 tok/s (+11.81%) and reduced median ITL from 8.589 to 7.562 ms (-11.96%). This measurement validates the combined draft + target stack; it does not isolate the marginal gain of this second layer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant