SpQR Quantization? ... for Near-Lossless LLM Weight Compression

[ianscrivener]

almost sounds too good to be true... but this technique even makes sense to the layman

SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression
https://arxiv.org/abs/2306.03078
https://github.com/Vahe1994/SpQR

via: https://www.superdatascience.com/podcast/near-lossless-llm-quantization

[Green-Sky]

#1602 (comment)
Thanks to @ikawrakow 's great work with his k-quants, spqr likely has nothing left to explore. (see comment)

[Green-Sky]

funnily, the spqr paper was released the same day k-quants where merged :)

[afg1]

Does k-quants do sparsity as well then? I really need to make some time to read up on it...

[ikawrakow]

k-quants offer quantization from 2 to 6 bits. They also offer quantization mixes, where some of the tensors are quantized with more bits. The Q4_K_M variant arrives at better than 1% difference in perplexity to fp16 for all LLaMA models. At 7B the Q4_K_M model size is 3.8 GiB. As far as I can tell this is slightly better than their "lossless compression", which needs 4 GiB for the same accuracy. One of the main ideas in their paper, which they call "double quantization" was first proposed in this repo in #1240 (I called it Super-Blocks instead). Their other idea is to spend more bits on "outliers". I found this to be less effective than using more bits for a subset of the tensors, which was first proposed in #1256 (less effective in the sense that if you spent the same amount of extra bits as you did on "outliers", you get a lower perplexity by using those extra bits on e.g. the feed_forward.w2 tensors).