Skip to content

fix(pd): add configurable decode capacity admission limit - #1514

Merged
shihaobai merged 1 commit into
ModelTC:mainfrom
sufubao:improve-429-handling
Aug 26, 2026
Merged

fix(pd): add configurable decode capacity admission limit#1514
shihaobai merged 1 commit into
ModelTC:mainfrom
sufubao:improve-429-handling

Conversation

@sufubao

@sufubao sufubao commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add a PD-master admission guard that computes the current capacity for every incoming request by summing running_max_req_size across all registered decode nodes. If the Master already has that many in-flight requests, it raises the existing ServerBusyError before multimodal preload, tokenization, or PD-stage waits.

The guard is enabled by default. To explicitly restore the previous behavior, start the PD Master with:

--disable_pd_master_decode_capacity_limit

Motivation

Overload may otherwise be detected only after entering the PD flow and waiting 60s/180s, so clients can wait a long time before receiving 429. Decode nodes already register their complete startup arguments with the Master, and disconnected nodes are removed from decode_nodes, allowing the current node list to remain the single source of truth without a cached capacity value.

This is intentionally temporary:

  • counts end-to-end Master requests rather than exact instantaneous D occupancy
  • uses the existing static per-D request-count limits; it is not token/KV-aware admission control
  • changes no streaming APIs or timeout behavior
  • capacity is local to each Master; a global cap across multiple Masters remains the gateway responsibility

Checks

  • pre-commit (Black and flake8)
  • git diff --check

@sufubao
sufubao force-pushed the improve-429-handling branch from ea8ce21 to 7b862dd Compare August 26, 2026 10:26
@sufubao sufubao changed the title fix(pd): reject excess requests before queueing fix(pd): reject requests when master is at capacity Aug 26, 2026
@sufubao
sufubao force-pushed the improve-429-handling branch 2 times, most recently from 40e9708 to 1f36388 Compare August 26, 2026 12:39
@sufubao sufubao changed the title fix(pd): reject requests when master is at capacity fix(pd): reject requests when decode capacity is reached Aug 26, 2026
@sufubao
sufubao force-pushed the improve-429-handling branch 3 times, most recently from 7465e88 to 75f12ad Compare August 26, 2026 13:10
@sufubao sufubao changed the title fix(pd): reject requests when decode capacity is reached fix(pd): add opt-in decode capacity admission limit Aug 26, 2026
@sufubao sufubao changed the title fix(pd): add opt-in decode capacity admission limit fix(pd): add configurable decode capacity admission limit Aug 26, 2026
@sufubao
sufubao force-pushed the improve-429-handling branch from 75f12ad to e225ca9 Compare August 26, 2026 13:12
@sufubao
sufubao force-pushed the improve-429-handling branch from e225ca9 to 1637038 Compare August 26, 2026 13:14
@shihaobai
shihaobai merged commit 73cf11d into ModelTC:main Aug 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants