Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix package loss for small models #2717

Merged
merged 1 commit into from
Jan 3, 2025
Merged

Fix package loss for small models #2717

merged 1 commit into from
Jan 3, 2025

Conversation

merrymercy
Copy link
Contributor

@merrymercy merrymercy commented Jan 2, 2025

Fix for #2548

fix packet loss when deploy small llm like qwen 0.5B or fishauido 

## Motivation

We found that when deploying some small LLMs, SGlang experiences packet loss under high load because the scheduler's results return too quickly, and at the same time, the rid is not updated in rid_to_state

## Modifications

we move send_to_scheduler after rid_to_state update

Co-authored with @sdli1995

@merrymercy merrymercy force-pushed the pr-fix-package-loss branch from 5fc3384 to 63264bd Compare January 3, 2025 01:47
@merrymercy merrymercy merged commit ffb5816 into main Jan 3, 2025
16 of 17 checks passed
@merrymercy merrymercy deleted the pr-fix-package-loss branch January 3, 2025 02:09
merrymercy added a commit that referenced this pull request Jan 3, 2025
XiaotongJiang pushed a commit to XiaotongJiang/sglang that referenced this pull request Jan 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant