Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable lora-adapters for dev mode #306

Open
gorkem opened this issue May 8, 2024 · 3 comments
Open

enable lora-adapters for dev mode #306

gorkem opened this issue May 8, 2024 · 3 comments
Labels
enhancement New feature or request

Comments

@gorkem
Copy link
Contributor

gorkem commented May 8, 2024

Describe the problem you're trying to solve
dev mode should be able to load lora-adapters if they are present in the modelkit

Describe the solution you'd like
dev mode should detect if there is a lora-adapter type on the modelkit and load it to the dev-mode.

@gorkem gorkem added the enhancement New feature or request label May 8, 2024
@srikary12
Copy link
Contributor

Are we looking for something like model.parts.type and adding those in dev mode.

@srikary12
Copy link
Contributor

Does the llamafile we are downloading support --lora tag to run lora adapters?

@gorkem
Copy link
Contributor Author

gorkem commented Dec 14, 2024

Yes it does. Here are the related options. They are just inherited from llama.cpp

--lora FNAME
Apply LoRA adapter (implies --no-mmap)
--lora-scaled FNAME S
Apply LoRA adapter with user defined scaling S (implies --no-mmap)
--lora-base FNAME
Optional model to use as a base for the layers modified by the LoRA adapter

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants