Skip to content

feat(pd): expose stage and node load metrics - #1509

Open
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:feat/pd-master-metrics
Open

feat(pd): expose stage and node load metrics#1509
sufubao wants to merge 2 commits into
ModelTC:mainfrom
sufubao:feat/pd-master-metrics

Conversation

@sufubao

@sufubao sufubao commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

What

  • Export lightllm_pd_node_token_usage_ratio{role,endpoint} from the PD master using the load reports it already receives from P/D nodes.
  • Export lightllm_pd_master_stage_waiting_requests{stage} for prefill and decode waits.
  • Keep stage counts balanced across success, timeout, cancellation, and disconnect paths.
  • Copy RPyC label proxies into local dictionaries before updating labeled Prometheus gauges.

Why

PD deployments currently expose end-to-end latency but not enough information on the master to tell whether requests are waiting on prefill or decode capacity. These two gauges provide that distinction with no PD protocol changes.

Validation

  • PYTHONPATH=. pytest -q test/test_pd_selector/test_pd_master_metrics.py test/test_pd_selector/test_pd_master_multi_choice.py — 12 passed.
  • Black, Flake8, and git diff --check passed.
  • Real local Qwen3.5-27B P1D1 NIXL smoke on two H200 GPUs: all components became healthy, a 3617-token request returned HTTP 200 in 2.064s, P/D node load gauges were exported, and the prefill waiting gauge transitioned 0 -> 1 -> 0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant