Setup at least one e2e test

[ahg-g]

To verify the end-to-end flow and CRD setup.

[ahg-g]

/assign @liu-cong

[danehans]

@liu-cong any status updates?

[liu-cong]

/assign @kaushikmitr

Example: https://github.com/kubernetes-sigs/jobset/blob/main/test/e2e/e2e_test.go
The ext proc image: us-central1-docker.pkg.dev/k8s-staging-images/llm-instance-gateway/epp:main

[k8s-ci-robot]

@liu-cong: GitHub didn't allow me to assign the following users: kaushikmitr.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @kaushikmitr

Example: https://github.com/kubernetes-sigs/jobset/blob/main/test/e2e/e2e_test.go
The ext proc image: us-central1-docker.pkg.dev/k8s-staging-images/llm-instance-gateway/epp:main

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[liu-cong]

@liu-cong any status updates?

@kaushikmitr is picking this up

[danehans]

/assign @danehans
/unassign @kaushikmitr

[danehans]

A Hugging Face Hub token is required to pull the model image used for e2e testing. How should this be handled when running e2e in CI, e.g. create a Hugging Face account used specifically for CI?

[ahg-g]

If we can create one for CI, then that would be great. Perhaps we can ask on the sig-testing channel on how to handle such accounts?

[danehans]

xref kubernetes/k8s.io#7698 for creating an HF account.

[danehans]

@ahg-g @liu-cong regarding this issue, Mistral will not work without training Mistral-compatible LoRA weights. See this gist for details.

[ahg-g]

Can we use a dummy LoRA? Check this in vllm: https://github.com/vllm-project/vllm/blob/28e0750847ded93158a66efdcbc869d87463b38f/vllm/lora/lora.py#L75

However, I am not sure if there is a way to configure vllm to actually set it up.

[liu-cong]

Mistral will not work without training Mistral-compatible LoRA weights

I think we just need to find the right adapter compatible with Mistral, right? BTW, you can find compatible adapters in the HF page, like this

Besides, the Qwen model doesn't require any HF login and it has adapters as well: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/discussions

[danehans]

I think we just need to find the right adapter compatible with Mistral, right?

The model architecture also needs to be supported by vLLM. Qwen appears to be supported but I have not verified, have you?

[liu-cong]

The model architecture also needs to be supported by vLLM. Qwen appears to be supported but I have not verified, have you?

I didn't try but Qwen appears in the vLLM get started guide so I am pretty sure it's supported.