Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Need to test] Remove max_memory usage and adapter float16 initialization by default #3610

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

[Need to test] Remove max_memory usage and adapter float16 initialization by default #3610

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions ludwig/models/llm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,14 +293,14 @@ def to_device(self, device):
if device == torch.device("cuda") and num_gpus > 1:
# TODO: make this configurable in the future. These parameters are from FastChat:
# https://github.com/lm-sys/FastChat/blob/0e958b852a14f4bef5f0e9d7a5e7373477329cf2/fastchat/serve/inference.py#L90 # noqa
# TODO: Wrap device_map="auto" in a try-except block since it may not be supported for all models (E.g. BertLMHead) # noqa
# TODO: Wrap device_map="auto" in a try-except block since it may not be supported for all models
# (E.g. BertLMHead) # noqa
# We don't add quantization here (float16 or bfloat16) since we may not always want to quantize. We should
# make quantization configurable in the future via the trainer config.
model_kwargs.update(
dict(
low_cpu_mem_usage=True,
device_map="auto",
max_memory={i: "13GiB" for i in range(num_gpus)},
)
)

Expand All @@ -321,7 +321,6 @@ def to_device(self, device):
self.model = PeftModel.from_pretrained(
self.model,
tmpdir,
torch_dtype=torch.float16,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need a way to plumb this in, otherwise it's going to load in fp32, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true yeah, I think we actually need a way to plumb in

quantization:
    bits: 16
    type: bfloat16

and

quantization:
    bits: 16
    type: float16

and then trickle this down appropriately to both the base model and the adapter.

This will also allow users to experiment with non-deepspeed-based training if they'd like (say they have a H100 or A100)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think?

)
else:
self.model = AutoModelForCausalLM.from_pretrained(
Expand Down
Loading