Running DeepSeek-R1-AWQ on single A100 with vllm

[mjp9527]

I discovered an interesting thing, by placing the MOE experts' weights in the CPU's pinned memory during loading, Triton kernels for FusedMOE can support pinned CPU tensors. This allows the DeepSeek-R1 model to run on a GPU with only 40GB of VRAM. I successfully ran the AWQ model on an A800 GPU, although the performance was relatively poor. Model is cognitivecomputations/DeepSeek-R1-AWQ

[dailingcs]

Can you share your command? Thank you!

[jamin85cheng]

Can you share your command? Thank you!