Skip to content

feat: support ai-gateway v0.2.1 #478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

googs1025
Copy link
Member

@googs1025 googs1025 commented Jul 5, 2025

What this PR does / why we need it

Which issue(s) this PR fixes

Fixes #463 (comment)

Special notes for your reviewer

Does this PR introduce a user-facing change?

support ai-gateway v0.2.1

@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Jul 5, 2025
@InftyAI-Agent InftyAI-Agent requested review from carlory and cr7258 July 5, 2025 13:32
@@ -85,38 +85,62 @@ spec:
value: qwen2-0--5b
backendRefs:
- name: qwen2-0--5b
modelsOwnedBy: "llmaz"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a newly added field

kind: Backend
group: gateway.envoyproxy.io
---
apiVersion: gateway.envoyproxy.io/v1alpha1
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Support after v0.2.0

@googs1025
Copy link
Member Author

envoyproxy/ai-gateway#829

This is just a draft and there are still some issues that have not been resolved.

@googs1025
Copy link
Member Author

@kerthcet I have some problems, but I don't see the problem. If you have time, can you help me take a look? Thanks for your help.

  • use support/ai-gateway-v0.2.0 branch

  • make helm-install

  • patch configmaps

kubectl patch cm envoy-gateway-config -n llmaz-system -p='{"data":{"envoy-gateway.yaml":"apiVersion: gateway.envoyproxy.io/v1alpha1\nkind: EnvoyGateway\nextensionApis:\n  enableBackend: true\n  enableEnvoyPatchPolicy: true\nextensionManager:\n  hooks:\n    xdsTranslator:\n      post:\n      - VirtualHost\n      - Translation\n  service:\n    fqdn:\n      hostname: ai-gateway-controller.envoy-ai-gateway-system.svc.cluster.local\n      port: 1063\ngateway:\n  controllerName: gateway.envoyproxy.io/gatewayclass-controller\nlogging:\n  level:\n    default: info\nprovider:\n  kubernetes:\n    rateLimitDeployment:\n      container:\n        image: docker.io/envoyproxy/ratelimit:3e085e5b\n      patch:\n        type: StrategicMerge\n        value:\n          spec:\n            template:\n              spec:\n                containers:\n                - imagePullPolicy: IfNotPresent\n                  name: envoy-ratelimit\n    shutdownManager:\n      image: docker.io/envoyproxy/gateway:v1.4.0\n  type: Kubernetes"}}'
  • use example
root@VM-0-15-ubuntu:/home/ubuntu/llmaz# kubectl apply -f docs/examples/envoy-ai-gateway/basic.yaml
openmodel.llmaz.io/qwen2-0--5b created
playground.inference.llmaz.io/qwen2-0--5b created
openmodel.llmaz.io/qwen2--5-coder created
playground.inference.llmaz.io/qwen2--5-coder created
gatewayclass.gateway.networking.k8s.io/default-envoy-ai-gateway created
gateway.gateway.networking.k8s.io/default-envoy-ai-gateway created
aigatewayroute.aigateway.envoyproxy.io/default-envoy-ai-gateway created
aiservicebackend.aigateway.envoyproxy.io/qwen2-0--5b created
aiservicebackend.aigateway.envoyproxy.io/qwen2--5-coder created
backend.gateway.envoyproxy.io/qwen2-0--5b created
backend.gateway.envoyproxy.io/qwen2--5-coder created


According to the feedback here, it seems to be a webhook-related issue, but I haven’t found the configuration problem yet. . . .

image

@kerthcet
Copy link
Member

Sure, I'll take a look later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/needs-kind Indicates a PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support Envoy AI gateway v0.2.0
3 participants