[router] LSH based prefix cache aware router

[gaocegege]

🚀 Feature Description and Motivation

Right now, we're using xxhash in #641 for our prefix cache-aware router. We might consider switching to a consistent hash + LSH-based approach, which could reduce accuracy a bit but would simplify scaling. Here are some related discussions: vllm-project/production-stack#59 (comment).

Use Case

N/A

Proposed Solution

No response

[Jeffwan]

@varungup90 @DwyaneShi can you spend some time on this issue?

[gaocegege]

It's just a proposal; I don't know if it helps in the chat use case. But it works well with long document QA. Ref vllm-project/production-stack#59 (comment)