Can vllm serving clients by using multiple model instances?

[aoyulong]

Based on the examples, vllm can launch a server with a single model instances. Can vllm serving clients by using multiple model instances? With multiple model instances, the sever will dispatch the requests to different instances to reduce the overhead.

[zhuohan123]

Right now vLLM is a serving engine for a single model. You can start multiple vLLM server replicas and use a custom load balancer (e.g., nginx load balancer). Also feel free to checkout FastChat and other multi-model frontends (e.g., aviary). vLLM can be a model worker of these libraries to support multi-replica serving.

[hex-plex]

This is just a example of load balancer that I use hope its useful for others

Load Balancer Gist

[hughesadam87]

Is this still the case? If so, why does the API support a model parameter if the intents is not to host multiple models?

[dmarx]

maybe to be compliant with the OpenAI API spec, possibly also future proofing

[khurramkhalil]

Can someone please let me know what is best practice to serve multiple (different) models with vllm?

[huiyeruzhou]

有人可以告诉我使用 vllm 服务多个（不同）模型的最佳做法是什么吗？

Ray serve shold have been the best practice, However it seems that ray serve framework and ray infrastration used in AsyncLLMEngine are conflicting and result in a No GPU Error.