Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Model] PP support for Mamba-like models #10992

Merged
merged 31 commits into from
Dec 11, 2024
Merged

[Model] PP support for Mamba-like models #10992

merged 31 commits into from
Dec 11, 2024
+229 −81

Conversation

mzusman
Copy link
Contributor

@mzusman mzusman commented Dec 8, 2024 •
edited by github-actions bot
Loading

Adds PP support for Mamba-like models (Jamba, Mamba, MambaFalcon, etc..)

CC @tlrmchlsmth

@github-actions GitHub Actions
Copy link

github-actions bot commented Dec 8, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@mergify Mergify
Copy link

mergify bot commented Dec 8, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mzusman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mzusman added 10 commits December 8, 2024 17:58
@mzusman
Add PP support to Jamba
90d0ff3 
Signed-off-by: mzusman <[email protected]>
@mzusman
GPU Runner adaptations
d30f741 
Signed-off-by: mzusman <[email protected]>
@mzusman
Add tests and adaptations
47e961b 
Signed-off-by: mzusman <[email protected]>
@mzusman
Add PP to mamba
e7e3ea4 
Signed-off-by: mzusman <[email protected]>
@mzusman
Introduce LayerBlockType
ebbf7e3 
Signed-off-by: mzusman <[email protected]>
@mzusman
Fix jamba
5514aa6 
Signed-off-by: mzusman <[email protected]>
@mzusman
Fix MS for Mamba
f9fcff4 
Signed-off-by: mzusman <[email protected]>
@mzusman
Format
2b5b525 
Signed-off-by: mzusman <[email protected]>
@mzusman
Format
01c7d93 
Signed-off-by: mzusman <[email protected]>
@mzusman
Format
50f2a8c 
Signed-off-by: mzusman <[email protected]>
DarkLight1337
DarkLight1337 approved these changes Dec 9, 2024
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding this!

tlrmchlsmth
tlrmchlsmth reviewed Dec 9, 2024
View reviewed changes
tests/distributed/test_pipeline_parallel.py Outdated Show resolved Hide resolved
vllm/config.py Outdated Show resolved Hide resolved
tlrmchlsmth
tlrmchlsmth requested changes Dec 9, 2024
View reviewed changes
Copy link
Collaborator

@tlrmchlsmth tlrmchlsmth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm requesting changes for the get_num_layers_by_block_type issue but the PR looks good otherwise!

vllm/model_executor/models/jamba.py Show resolved Hide resolved
@mzusman
Use only needed number layers per PP
46e3c25 
Signed-off-by: mzusman <[email protected]>
@mzusman
Fix get_num_layers_by_block_type to always return num of layers
f354468 
regardless of given block_type

Signed-off-by: mzusman <[email protected]>
@mzusman
Format
529ade6 
Signed-off-by: mzusman <[email protected]>
tlrmchlsmth
tlrmchlsmth reviewed Dec 9, 2024
View reviewed changes
vllm/config.py Outdated Show resolved Hide resolved
@mzusman
Add comment for workaround
e7f6338 
Signed-off-by: mzusman <[email protected]>
@mzusman
Fix comment
6d863ae 
Signed-off-by: mzusman <[email protected]>
@mzusman
Add IsHybrid interface to return 0 layers on non-hybrid models asking
d9ef20d 
about non-attention layers

Signed-off-by: mzusman <[email protected]>
@mzusman
Typo
029d710 
Signed-off-by: mzusman <[email protected]>
@mergify Mergify
Copy link

mergify bot commented Dec 9, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mzusman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Dec 9, 2024
@mergify mergify bot removed the needs-rebase label Dec 9, 2024
@mzusman
Format
3bc3823 
Signed-off-by: mzusman <[email protected]>
@mzusman
Fix mamba logic
0f70131 
Signed-off-by: mzusman <[email protected]>
@mergify Mergify
Copy link

mergify bot commented Dec 10, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mzusman.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Dec 10, 2024
@mergify mergify bot removed the needs-rebase label Dec 10, 2024
@mzusman
Small fix to logic
fcce742 
Signed-off-by: mzusman <[email protected]>
tlrmchlsmth
tlrmchlsmth reviewed Dec 10, 2024
View reviewed changes
vllm/config.py Outdated Show resolved Hide resolved
@mzusman
FIx logic
e3d8343 
Signed-off-by: mzusman <[email protected]>
tlrmchlsmth
tlrmchlsmth reviewed Dec 10, 2024
View reviewed changes
vllm/config.py Outdated Show resolved Hide resolved
@mzusman
Fixing according to comment
6d6b3d4 
Signed-off-by: mzusman <[email protected]>
@mzusman
Format
c034ffe 
Signed-off-by: mzusman <[email protected]>
tlrmchlsmth
tlrmchlsmth approved these changes Dec 10, 2024
View reviewed changes
Copy link
Collaborator

@tlrmchlsmth tlrmchlsmth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@mzusman
Fix comment
75ff1ef 
Signed-off-by: mzusman <[email protected]>
@tlrmchlsmth tlrmchlsmth merged commit ffa48c9 into vllm-project:main Dec 11, 2024
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mzusman @tlrmchlsmth @DarkLight1337