A high-throughput and memory-efficient inference and serving engine for LLMs

The vLLM XPU kernels for Intel GPU

Community maintained hardware plugin for vLLM on Intel Gaudi

Intelligent Mixture-of-Models Router for Efficient LLM Inference

Cost-efficient and pluggable Infrastructure components for GenAI inference

Community maintained hardware plugin for vLLM on Ascend

Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

This repo hosts code for vLLM CI & Performance Benchmark infrastructure.