Flux quantized with lora #10990
Flux quantized with lora #10990
Conversation
| # TODO (sayakpaul): We still need to consider if the module we're expanding is | ||
| # quantized and handle it accordingly if that is the case. |
There was a problem hiding this comment.
cc @sayakpaul should be ok to remove these TODO now
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Fantastic! I left some nits, let me know if they make sense.
Somewhat related to #10588, which I had planned to work on. But glad that you beat me to it. Do we want to club that in this PR?
Also, let's add a test to test_4bit.py?
| @@ -18,6 +18,7 @@ | |||
| import torch | |||
| from huggingface_hub.utils import validate_hf_hub_args | |||
|
|
|||
| from ..quantizers.bitsandbytes import dequantize_bnb_weight | |||
There was a problem hiding this comment.
Should this be guarded with is_bitsandbytes_available()?
There was a problem hiding this comment.
Guard added, it is guarded internally to quantizers.bitsandbytes so would also be ok without
| module_weight = ( | ||
| dequantize_bnb_weight(module.weight, state=module.weight.quant_state).data | ||
| if module.weight.__class__.__name__ == "Params4bit" | ||
| else module.weight.data | ||
| ) |
There was a problem hiding this comment.
This looks good. But to be extra cautious, I would first determine if module.weight.__class__.__name__ == "Params4bit". Then do
if not is_bistandbytes_available():
raise ....
else:
....Also I think the data device needs to be on a device where bitsanbdytes is known to be supported. Otherwise, its won't be able to dequantize.
| # TODO (sayakpaul): We still need to consider if the module we're expanding is | ||
| # quantized and handle it accordingly if that is the case. |
|
Test also added, needs slice updating from runner. |
Co-authored-by: Sayak Paul <[email protected]>
|
@bot /style |
|
Style fixes have been applied. View the workflow run here. |
|
What's the easiest way to trigger the slow bnb tests? |
|
Would have been this workflow: Easier would be to use with the big GPU ( |
What does this PR do?
Fixes #10989
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.