Skip to content

perf(pd): compact token forwarding packets - #1487

Open
WANDY666 wants to merge 1 commit into
perf/pd-omit-unused-logprob-metadatafrom
perf/pd-compact-token-packets
Open

perf(pd): compact token forwarding packets#1487
WANDY666 wants to merge 1 commit into
perf/pd-omit-unused-logprob-metadatafrom
perf/pd-compact-token-packets

Conversation

@WANDY666

@WANDY666 WANDY666 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Encode the common no-logprobs PD token update as a fixed tuple containing only fields consumed by PD master.
  • Preserve current main metadata, including MTP accepted/verify counters, first-token node mode, and multimodal input usage.
  • Fall back to the legacy packet whenever optional metadata is present.
  • Split mixed compact/legacy batches into contiguous wire packets while preserving token order.
  • Reconstruct the existing metadata and FinishStatus contract at PD master.

Dependency

This is a stacked PR on perf/pd-omit-unused-logprob-metadata (#1486), because that PR supplies the request-level marker selecting the common no-logprobs path. After #1486 merges, this PR can be retargeted to main without carrying unrelated changes.

Benchmark context

The historical cfce087 32K microbenchmark measured packet size around 8.4 KB → 3.6 KB, D-side serialization around 168 µs → 19 µs, and PD-master deserialization around 868–1366 µs/packet → 32 µs. The full q15 result remained within run-to-run noise, so this is presented as CPU/transport headroom rather than a proven QPS-capacity increase.

Validation

  • Production-code diff passes repository black and flake8 hooks.

@WANDY666
WANDY666 force-pushed the perf/pd-omit-unused-logprob-metadata branch from 788db8c to c5f5eb0 Compare August 24, 2026 09:11
@WANDY666
WANDY666 force-pushed the perf/pd-compact-token-packets branch from 1834566 to ddb1e01 Compare August 24, 2026 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant