docs: add P/D disaggregation example in manifests/disaggregation #253
Conversation
googs1025
force-pushed
the
feature/pd_example
branch
from
6ce81a9 to
c9364a8
Compare
| metadata: | ||
| name: vllm-sim-p | ||
| labels: | ||
| app: my-llm-pool |
There was a problem hiding this comment.
why the "my-llm-pool" label is required? when the simulator is installed as part of llm-d - it defines the innference-pool, but here this is a stand-alone example
There was a problem hiding this comment.
removed, only use llm-d.ai/role label
manifests/disaggregation/README.md
Outdated
| Now, send a request to the forwarded Decode service port with the necessary headers: | ||
|
|
||
| ```bash | ||
| curl -v http://localhost:8001/v1/chat/completions \ |
There was a problem hiding this comment.
oh, thanks for reminding
| spec: | ||
| containers: | ||
| - name: routing-sidecar | ||
| image: ghcr.io/llm-d/llm-d-routing-sidecar:latest |
There was a problem hiding this comment.
there is no latest tag for side-car
There was a problem hiding this comment.
Yes, that's also where I have some questions. When I was testing, I used ghcr.io/llm-d/llm-d-routing-sidecar:v0.3.1-rc.1, which I referenced here: https://github.com/llm-d/llm-d-routing-sidecar/releases.
However, I see "latest" being used here, which I assumed was due to network issues I can't downloads.
googs1025
force-pushed
the
feature/pd_example
branch
2 times, most recently
from
a1137a6 to
842d281
Compare
|
@googs1025 I tested the configuration from the scheduler and turned out that there is a new version of nixl connector and simulator should be changed to by in-sync with it. |
Signed-off-by: googs1025 <[email protected]>
googs1025
force-pushed
the
feature/pd_example
branch
from
842d281 to
3c49142
Compare
change to |
|
Changing the connector type to nixlv2 is not enough, PD will work end-to-end after #255 fix. |
2 participants
fix: #251 (comment)