[Feature] request smoothquant (int8, W8A8) quantization on 40G A100

[Hao-YunDeng]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

We have been using smoothquant (int8, W8A8) quantization on A100 GPU with TensorRT-LLM and recently tested with vLLM as well. The performance is good: speed, memory, and accuracy all are advantageous compared to fp16 or other quantizations.

Can SGLang also support such quantization for A100 machine? My team is very eager to see it coming.

Thanks

Related resources

No response

[Hao-YunDeng]

AWQ and GPTQ both are W8A16; we need W8A8

[zhyncs]

AWQ and GPTQ are W4A16.

@HandH1998 and @ispobock are collaborating on the W8A8.

[Hao-YunDeng]

AWQ and GPTQ are W4A16.

@HandH1998 and @ispobock are collaborating on the W8A8.

thank you so much for your reply. Is this W8A8 feature going to be smoothquant? If so, when do you expect to have it available? @zhyncs @HandH1998 @ispobock

[ispobock]

AWQ and GPTQ are W4A16.
@HandH1998 and @ispobock are collaborating on the W8A8.

thank you so much for your reply. Is this W8A8 feature going to be smoothquant? If so, when do you expect to have it available? @zhyncs @HandH1998 @ispobock

1. Yes, it's smoothquant.
2. Maybe next two weeks.

[halexan]

Does sglang support w8a8 quantized model? Like this one: neuralmagic-ent/Qwen2.5-72B-Instruct-quantized.w8a8

If supported, how can I fly it?