Loading LORA Adapters with specific configuration

[bely66]

Hi Everyone,

I have been moving my model to TGI to accelerate the inference speed

Some issues I'm facing right now:

1. I can't pass a bnb config (which contains info like double_quant, dtype, loading in 4_bit and quant type), i only can pass quant_type nf4

2. A complete drop in performance (my adapter performance is completely bad, doesn't do the task) which i suspect because of the above configuration differences

3. I can't configure the tokenizer for my model because I change it with padding to right instead of left

So many configuration constraints that's making reproducing my model a hell

So any idea how can I fix or handle this? is the issue that I'm using qlora and only lora is supported? Should I export the model with the specific configuration and load it locally instead of configuring it on the spot?