-
Notifications
You must be signed in to change notification settings - Fork 137
AWQ Apply Scales Bugfix when smooth layer output length doesn't match balance layer input length #1451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
587ec3e
to
72d51b0
Compare
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
72d51b0
to
5f06c06
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the test results! Looks good to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚀
SUMMARY: I wanted to create a PR showing users how they can add more mappings to AWQ to account for more models. Turns out qwen has the exact same as Llama, so I added one for Phi as well. I also updated the naming and used the infer pattern employed in SmoothQuant, rather than requiring user to set it TEST PLAN: `examples/awq/llama_example.py` works on this branch for ```python MODEL_ID = "microsoft/Phi-4-mini-reasoning" ``` TODOs: - [x] Merge in after #1451 lands --------- Signed-off-by: Brian Dellabetta <[email protected]>
Summary
We are hitting an edge case in AWQ we had not previously hit with the initial Llama/Qwen testing models. When a smooth layer's # of output_features does not match a balance layer's # of input_features, the code as it is currently will error out when trying to update the smooth layer's weights with
weights.div(scales)
, due to a shape mismatch error. We are hitting this in #1440 for Phi3 models, which include a mapping between the fusedqkv_proj
smooth layer ando_proj
balance layer in AutoAWQ (see here).The resolution in AutoAWQ is to only use the last rows of the smooth layer so that the shapes line up, as shown here. This PR includes that update, and with #1440 will allow Phi3 models to be quantizable with AWQModifier. Like with v_proj -> o_proj, if shapes don't match up, they will be excluded from resolved mappings. This allows phi-3-mini to include the mapping because
qkv_proj out_features == 3*o_proj in_features == 9216
, but excludes it from phi-3-medium which hasqkv_proj out_features == 7680
ando_proj in_features==5120
. If the mapping is included for phi-3-medium, the model blows up with wikitext eval perplexities >2000. This implementation was agreed upon with @anmarques .PS: I also moved
mul
&div
tomul_
&div_
, to avoid unnecessary memory allocation.Test Plan
With these changes and with #1440 ,
examples/awq/llama_example.py
works with"microsoft/Phi-3-mini-128k-instruct"
and produces similar results as when qkv_proj to o_proj mapping is includedWithout mapping:
With mapping:
I also confirmed re-running with
meta-llama/Llama-3.2-3B-Instruct
andmeta-llama/Llama-2-7b-hf
does not deviate in PPL scores from what is currently onmain