Skip to content

AWQ Apply Scales Bugfix when smooth layer output length doesn't match balance layer input length #1451

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
May 21, 2025

Conversation

brian-dellabetta
Copy link
Collaborator

@brian-dellabetta brian-dellabetta commented May 19, 2025

Summary

We are hitting an edge case in AWQ we had not previously hit with the initial Llama/Qwen testing models. When a smooth layer's # of output_features does not match a balance layer's # of input_features, the code as it is currently will error out when trying to update the smooth layer's weights with weights.div(scales), due to a shape mismatch error. We are hitting this in #1440 for Phi3 models, which include a mapping between the fused qkv_proj smooth layer and o_proj balance layer in AutoAWQ (see here).

The resolution in AutoAWQ is to only use the last rows of the smooth layer so that the shapes line up, as shown here. This PR includes that update, and with #1440 will allow Phi3 models to be quantizable with AWQModifier. Like with v_proj -> o_proj, if shapes don't match up, they will be excluded from resolved mappings. This allows phi-3-mini to include the mapping because qkv_proj out_features == 3*o_proj in_features == 9216, but excludes it from phi-3-medium which has qkv_proj out_features == 7680 and o_proj in_features==5120. If the mapping is included for phi-3-medium, the model blows up with wikitext eval perplexities >2000. This implementation was agreed upon with @anmarques .

PS: I also moved mul & div to mul_ & div_, to avoid unnecessary memory allocation.


Test Plan

With these changes and with #1440 , examples/awq/llama_example.py works with "microsoft/Phi-3-mini-128k-instruct" and produces similar results as when qkv_proj to o_proj mapping is included

Without mapping:

Tasks Version Filter n-shot Metric Value Stderr
wikitext 2 none 5 bits_per_byte 0.6474 ± N/A
none 5 byte_perplexity 1.5664 ± N/A
none 5 word_perplexity 11.0201 ± N/A

With mapping:

Tasks Version Filter n-shot Metric Value Stderr
wikitext 2 none 5 bits_per_byte 0.6482 ± N/A
none 5 byte_perplexity 1.5672 ± N/A
none 5 word_perplexity 11.0527 ± N/A

I also confirmed re-running with meta-llama/Llama-3.2-3B-Instruct and meta-llama/Llama-2-7b-hf does not deviate in PPL scores from what is currently on main

Copy link

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

@brian-dellabetta brian-dellabetta mentioned this pull request May 20, 2025
1 task
@brian-dellabetta brian-dellabetta force-pushed the bdellabe/awq-scales-bugfix branch from 587ec3e to 72d51b0 Compare May 20, 2025 18:59
@brian-dellabetta brian-dellabetta added the ready When a PR is ready for review label May 20, 2025
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
Signed-off-by: Brian Dellabetta <[email protected]>
@brian-dellabetta brian-dellabetta force-pushed the bdellabe/awq-scales-bugfix branch from 72d51b0 to 5f06c06 Compare May 21, 2025 17:31
Copy link
Collaborator

@shanjiaz shanjiaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the test results! Looks good to me.

Copy link
Collaborator

@rahul-tuli rahul-tuli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@kylesayrs kylesayrs enabled auto-merge (squash) May 21, 2025 17:44
@kylesayrs kylesayrs merged commit cfaa23f into main May 21, 2025
11 checks passed
@kylesayrs kylesayrs deleted the bdellabe/awq-scales-bugfix branch May 21, 2025 18:10
brian-dellabetta added a commit that referenced this pull request May 21, 2025
SUMMARY:
I wanted to create a PR showing users how they can add more mappings to
AWQ to account for more models. Turns out qwen has the exact same as
Llama, so I added one for Phi as well. I also updated the naming and
used the infer pattern employed in SmoothQuant, rather than requiring
user to set it

TEST PLAN:
`examples/awq/llama_example.py` works on this branch for 

```python
MODEL_ID = "microsoft/Phi-4-mini-reasoning"
```


TODOs:
- [x] Merge in after #1451 lands

---------

Signed-off-by: Brian Dellabetta <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready When a PR is ready for review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants