Can run an int8 quantized model on CUDA?

[shencuifeng]

I want to run a model on CUDA with actual int8 instructions instead of FakeQuantised float32 instructions, and enjoy the efficiency gains. It is slower when I set weights=qint8, activations=qint8 than weights=qint8.

[dacorvo]

@shencuifeng this is because the cost of quantizing activations on the fly is not compensated by the faster int8 matmul, especially considering that the float x int8 matmul might benefit from an accelerated kernel depending on the float type.

[shencuifeng]

@dacorvo Is it possible to support static quantize activations? It seems https://github.com/mit-han-lab/nunchaku static quantize the activations to int4.