How to swap LoRA during a post request to llama-server?

[Ujjawal-K-Panchal]

I recently came across @ngxson's PR: #8332. The idea of swapping LoRAs during runtime is fascinating!
I went through the PR & the documentation of llama-server and wanted to try this out and then came across this question.

Question: For a custom deployment how does one actively swap the LoRA depending on the request from user? While I could not find any parameters in the POST request, is there any other way or workaround to let the model use a specific LoRA to serve a specific request?

PS: One way could be to launch multiple instances of llama-server, then let the user control which version he wants to query, but that limits it to the amount of RAM you have (since you need to host multiple copies of the base model). Instead if one could keep all LoRA weights in the same llama.cpp deployment, it would be much more memory efficient. Especially from the perspective of small devices and personal computers.

[ngxson]

I drafted a PR, could you try it out? Thanks.

#8857

[Ujjawal-K-Panchal]

Amazing! Grateful for the feature.

[MaTwickenham]

Hi @ngxson, really appreciate your work on lora!
Are there any docs about how to use hot-swap lora? I saw the #8332 but have no idea on how to use multi lora adapters.

[ngxson]

Please have a look on #8857

[MaTwickenham]

So it is necessary to set up a LLaMA.cpp HTTP Server to use adapters by adding post arguments? I didn't know if I got it right.

[Ujjawal-K-Panchal]

TLDR:
The deployment looks like this:

llama-server -m <base-model> --lora-scaled <lora-1> 0.0 --lora-scaled <lora-2> 0.0 --lora-init-without-apply 

Then when you want to apply a specific LoRA, you make a POST request to: /lora-adapters with the following in Body:

[
  {"id": 0, "scale": 1.0},
  {"id": 1, "scale": 0.0}
]

Generally id = 0 corresponds to lora-1, id = 1 corresponds to lora-2 ... likewise.

If you want to apply a combination of LoRA, you can also apply some combination of scales:
With the following as body (applying both LoRAs equally):

[
  {"id": 0, "scale": 0.5},
  {"id": 1, "scale": 0.5}
]

Then to disable all LoRAs, simply set all scales to 0.