[Question] How to access the vLLM-Vineyard integration code mentioned in Distributed KV Cache documentation?

[cheyang]

The Distributed KV Cache documentation references a customized vLLM implementation with Vineyard integration.

However, I couldn't locate the corresponding code implementation.

Could you help me clarify:

1. Is the customized vLLM code with Vineyard support publicly accessible?
2. If available, is it in a separate branch/repository?

[gaocegege]

Hi Yang,

aibrix/pkg/controller/kvcache/kvcache_controller.go

Line 505 in 6feec99

fmt.Sprintf(`/usr/local/bin/vineyardd --sync_crds true --socket /var/run/vineyard.sock --size --stream_threshold 80 --etcd_cmd etcd --etcd_prefix /vineyard --etcd_endpoint http://%s-etcd-service:2379`, kvCache.Name),

I think it uses a customized vllm, something like vllm-project/vllm@main...aibrix:vllm:feat/distributed-kv-cache

From what I understand, this only works with v0.6.x of vLLM.

[Jeffwan]

Great to see you here @cheyang long time no see. Yeah, @gaocegege gave the code pointer in vLLM.

1. vLLM code will be refactor to adapt to v1 architecture and a RFC will be cut soon. this part will be definitely upstreamed.
2. @DwyaneShi made some changes like metadata optimization and advanced eviction policies in vineyard. @DwyaneShi feel free to comment if you have more information like to share.