[Feature]: support for fp8 marlin with MoE

[ehartford]

🚀 The feature, motivation and pitch

I wanna run Qwen3-235B-A22B on Ampere (A100) in fp8.

I quantized it to w8a16 using llm-compressor

https://huggingface.co/cognitivecomputations/Qwen3-235B-A22B-FP8-W8A16

but when I run it, I get the error

ERROR 05-02 03:16:53 [multiproc_executor.py:435] AssertionError: float16 is required for MoE compressed models. Set dtype=torch.float16

Please support FP8 with MoE in Marlin kernel

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[zhangxiaoxing]

I guess the following error is also related? It also happened on running FP8 MoE modal with Ampere cards. Qwen3 on FP8 is nice to have, and llama4 on FP8 could be even more interesting due to its potentially larger context length.

(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] triton.compiler.errors.CompilationError: at 1:0:
(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] def _per_token_group_quant_fp8(
(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] ^                                                                                                                

(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470] ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
(VllmWorker rank=1 pid=61141) ERROR 05-02 09:37:16 [multiproc_executor.py:470]                                                                                                                  

[sparkprime]

Is there any workaround for running 235b on a100? A different quantization maybe?

[ehartford]

AWQ should work theoretically

I haven't seen anyone quant this yet

Personally I'm working on it too.

But that also is failing for me.

vllm-project/llm-compressor#1406