FSDP + QLoRA use case

[kmehant]

Thank you for this wonderful project on bringing a unified format across quantization techniques. I am working on a usecase where I was trying to do FSDPv1 + QLoRA over compressed tensor type models on HF. Did we ever try such a usecase for CompressedLinear modules involved?

[brian-dellabetta]

Hi @kmehant , appreciate the kind words! While we do have flows for fine-tuning of sparse models (latest example of that here), we generally don't train after a model has been quantized. QLoRA is not currently in our roadmap.

CompressedLinear performs QDQ (quant + de-quant) on the model, this is a useful dev tool to test out quantized models within HF transformers but is really nonperformant. Compressed models loaded up in vllm use custom kernels over CompressedLinear to get the performance boost. It is not clear to me if this could be useful for implementing QLoRA

[kmehant]

we generally don't train after a model has been quantized. QLoRA is not currently in our roadmap.

Thanks for letting me know.