Implement I-quants (IQ4XS, IQ4NL) #2785
Conversation
|
This PR is based on the following reference ggml quantization/dequantization functions: Dequantization: Quantization: |
| let expected_blocks = xs.len() / block_size; | ||
| let actual_blocks = ys.len(); | ||
|
|
||
| // Validate that the input is the right size | ||
| if expected_blocks != actual_blocks { | ||
| crate::bail!("quantize {dtype:?}: expected {expected_blocks} blocks but only {actual_blocks} were provided!") | ||
| } |
There was a problem hiding this comment.
I am not opposed to doing this check every time, as that is safest, but this may be a decent case for debug_assert. What do you think @LaurentMazare?
candle-core/tests/quantized_tests.rs
Outdated
| assert_eq!(diff, 0.); | ||
| assert!(diff < 0.96); |
There was a problem hiding this comment.
The assert_eq version actually passes on my machine for both the neon and default impl. Is this increase in tolerance because of the avx impl perhaps?
There was a problem hiding this comment.
Hmm, I think that was a remnant from when I did the other neon/default impls. I undid this change (I'm still working out a bug in the iq4xs avx impl, iq4nl works nicely).
This PR refactors the quantization parts of candle-core a bit and integrates some new I-quants!
There is no CUDA or Metal support yet; perhaps we could add that in a later PR. I have added Metal support on a local branch, and I'm working on syncing the latest GGML CUDA kernels, which should also give a nice performance boost!