Development Roadmap (2024 Q4)

[Ying1123]

Here is the development roadmap for 2024 Q4. Contributions and feedback are welcome (Join Bi-weekly Development Meeting). Previous 2024 Q3 roadmap can be found in #634.

Performance

• Hide CPU overhead. See also a prototype.
• Support speculative decoding
  • Reference-based. See also a prototype.
  • Draft model based.
  • Medusa head [Feature] plan to support medusa? #859
• Efficient attention
  • Sparse Attention [WIP] Support double sparsity #1459
  • Parallel encoding
• Multi-layer radix cache (GPU/CPU/Disk)
• Faster grammar parsing library for constrained decoding

Parallelism

• Finish sequence parallelism [Feature] Add initial support for sequence parallelism #1436
• Support pipeline parallelism.
• Implement a better load balancer for data parallelism. See also preble.
• Support expert parallelism + data parallelism for DeepSeek models.
• Support disaggregated serving to separate prefill and decoding.

Hardware Coverage

• AMD optimizations. cc @HaiShaw
  • FlashInfer support for AMD
  • Setup CI (accuracy/performance) for AMD
• Intel XPU support.
  • [Feature, Hardware] Enable SGLang on XPU GPUs via PyTorch #1480

Model Coverage

• Multi-modal models
  • QWen2-VL
  • VILA
  • Phi-vision
  • FishSpeech audio model support
  • Ultravox
• Language models
  • Mamba models
• Reward models
  • [Feature] Support reward model LxzGordon/URM-LLaMa-3.1-8B #1525

LoRA support

• Performance optimizations for multi-LoRA serving [Performance] Add triton kernels for LoRA #1471

Quantization

@zhyncs @ispobock

• Torchao quantization
• Turbomind operators integration
• More CUTLASS mixed precision gemm integration
• KV cache quantization (more formats + scaling factor)

Server API

• Support directly taking embedding as inputs. [Feature] Generation Inputs: input_embeds #745
• Add APIs for using the inference engine in a single script without launching a separate server. See also examples.
  • [RFC] Add an LLM engine #1127
• Support endpoint other than OpenAI (Anthropic, Mistral) in the language frontend.
• Better APIs to support RL trainers https://github.com/huggingface/trl

Observability

• Integrate Grafana / Prometheus
  • [WIP] Prometheus Metrics #1461

Others

• Notebook-style interactive tutorials. @zhaochenyang20
• Compiler mode optimizations for the language (e.g. support sending a full serialized SGL program to the server). @hnyls2002
• Memory pool refactor to better support mixing different attention layers (e.g., interleaved window attention). @Ying1123
• Linear layers refactor. Make vLLM an optional dependency. @zhyncs

[fengyang95]

Are there any plans to optimize long context latency?