Added the ability for plugins to receive the request headers and modify them #760

shmuelk · 2025-04-30T10:37:12Z

This PR adds support for the PreSchedule, Filter, Scorer, Picker, and PostSchedule plugins to receive all of the headers sent by the user in their request. This enables the writing of plugins that make use of additional headers.

In addition, this PR adds support for the above plugins to add and/or modify the headers in the user's request. This is needed in certain solutions of Disaggregated Prefill/Decode.

The unit tests added are very simplistic. A better unit test will be added later at the StreamingServer level

…fy them

linux-foundation-easycla · 2025-04-30T10:37:16Z

The committers listed above are authorized under a signed CLA.

✅ login: shmuelk / name: Shmuel Kallner (ab54af9, 4aa87db, 0a1093d, b7224e0, e5846d1, 8fd2ca5, 840f7d4, 796f5b5)

k8s-ci-robot · 2025-04-30T10:37:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shmuelk
Once this PR has been reviewed and has the lgtm label, please assign arangogutierrez for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-04-30T10:37:21Z

Welcome @shmuelk!

It looks like this is your first PR to kubernetes-sigs/gateway-api-inference-extension 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api-inference-extension has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-04-30T10:37:22Z

Hi @shmuelk. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2025-04-30T10:37:33Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`796f5b5`
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/68121d1202321a00085f931c
😎 Deploy Preview	https://deploy-preview-760--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

…fy them

nirrozenbaum · 2025-04-30T11:04:21Z

/ok-to-test

nirrozenbaum · 2025-04-30T11:06:26Z

pkg/epp/handlers/request.go

@@ -70,7 +70,9 @@ func (s *StreamingServer) HandleRequestBody(
 		Model:               model,
 		ResolvedTargetModel: modelName,
 		Critical:            modelObj.Spec.Criticality != nil && *modelObj.Spec.Criticality == v1alpha2.Critical,
+		Headers:             reqCtx.RequestHeaders,


lines 73 and 75 are duplicates?

nirrozenbaum · 2025-04-30T11:14:45Z

pkg/epp/scheduling/types/types.go

@@ -32,6 +32,7 @@ type LLMRequest struct {
 	// Target models is a map of target model name to weight.
 	TargetModels map[string]int
 	Prompt       string
+	Headers      map[string]string


can you add a comment with description like the other fields?

Done. I also added one for MutatedHeaders in the SchedulingContext

nirrozenbaum · 2025-04-30T15:25:25Z

/assign

liu-cong · 2025-04-30T14:57:44Z

pkg/epp/scheduling/types/types.go

@@ -32,6 +32,9 @@ type LLMRequest struct {
 	// Target models is a map of target model name to weight.
 	TargetModels map[string]int
 	Prompt       string
+	// Headers during request processing contains all of the request headers.
+	//         During response processing it contains all of the response headers.


nit: remove spaces

liu-cong · 2025-04-30T16:43:30Z

pkg/epp/handlers/server.go

@@ -377,6 +380,15 @@ func (s *StreamingServer) populateRequestHeaderResponse(reqCtx *RequestContext,
 			},
 		})
 	}
+	// Add headers added by filters/scorers
+	for key, value := range mutatedHeaders {


The headers is a list. If the plugins updated the value of an existing header key, should we update the header here instead of just appending?

k8s-ci-robot · 2025-05-01T05:34:10Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

nirrozenbaum

@shmuelk there have been some changes in main that affect this PR.
in main - the headers are now passed through the reqCtx arg to HandleRequestBody using the Request struct.
this is a step towards the direction you made in this PR.
please rebase to grap latest changes.

nirrozenbaum · 2025-05-02T06:55:28Z

pkg/epp/handlers/request.go

+	for _, header := range req.RequestHeaders.Headers.Headers {
+		reqCtx.RequestHeaders[header.Key] = header.Value
+	}
+


this part was already merged into main

nirrozenbaum · 2025-05-02T06:56:12Z

pkg/epp/handlers/server.go

@@ -86,6 +86,8 @@ type RequestContext struct {
 	RequestState         StreamRequestState
 	modelServerStreaming bool

+	RequestHeaders map[string]string


this is also available already in main but in different field - Request, which contains body and headers.

nirrozenbaum · 2025-05-02T06:56:30Z

pkg/epp/handlers/server.go

@@ -117,7 +119,8 @@ func (s *StreamingServer) Process(srv extProcPb.ExternalProcessor_ProcessServer)
 	// Create request context to share states during life time of an HTTP request.
 	// See https://github.com/envoyproxy/envoy/issues/17540.
 	reqCtx := &RequestContext{
-		RequestState: RequestReceived,
+		RequestState:   RequestReceived,
+		RequestHeaders: make(map[string]string),


already in main in Request field

Added the ability for plugins to receive the request headers and modi…

ab54af9

…fy them

k8s-ci-robot requested review from liu-cong and nirrozenbaum April 30, 2025 10:37

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Apr 30, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 30, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Apr 30, 2025

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 30, 2025

Merge branch 'main' into headers-main

4aa87db

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 30, 2025

Added the ability for plugins to receive the request headers and modi…

0a1093d

…fy them

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Apr 30, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 30, 2025

nirrozenbaum reviewed Apr 30, 2025

View reviewed changes

shmuelk added 5 commits April 30, 2025 15:03

Updated test after rebase

840f7d4

Removed duplicate field

8fd2ca5

Added a go doc comment for the new field

b7224e0

Added a go doc comment for the new field

e5846d1

Ran go fmt on the updated file

796f5b5

k8s-ci-robot assigned nirrozenbaum Apr 30, 2025

liu-cong reviewed Apr 30, 2025

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 1, 2025

nirrozenbaum reviewed May 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added the ability for plugins to receive the request headers and modify them #760

Added the ability for plugins to receive the request headers and modify them #760

shmuelk commented Apr 30, 2025

linux-foundation-easycla bot commented Apr 30, 2025 •

edited

Loading

k8s-ci-robot commented Apr 30, 2025

k8s-ci-robot commented Apr 30, 2025

k8s-ci-robot commented Apr 30, 2025

netlify bot commented Apr 30, 2025 •

edited

Loading

nirrozenbaum commented Apr 30, 2025

nirrozenbaum Apr 30, 2025

shmuelk Apr 30, 2025

nirrozenbaum Apr 30, 2025

shmuelk Apr 30, 2025

nirrozenbaum commented Apr 30, 2025

liu-cong Apr 30, 2025

liu-cong Apr 30, 2025

k8s-ci-robot commented May 1, 2025

nirrozenbaum left a comment

nirrozenbaum May 2, 2025

nirrozenbaum May 2, 2025

nirrozenbaum May 2, 2025

Added the ability for plugins to receive the request headers and modify them #760

Are you sure you want to change the base?

Added the ability for plugins to receive the request headers and modify them #760

Conversation

shmuelk commented Apr 30, 2025

linux-foundation-easycla bot commented Apr 30, 2025 • edited Loading

k8s-ci-robot commented Apr 30, 2025

k8s-ci-robot commented Apr 30, 2025

k8s-ci-robot commented Apr 30, 2025

netlify bot commented Apr 30, 2025 • edited Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

nirrozenbaum commented Apr 30, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirrozenbaum commented Apr 30, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 1, 2025

nirrozenbaum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

linux-foundation-easycla bot commented Apr 30, 2025 •

edited

Loading

netlify bot commented Apr 30, 2025 •

edited

Loading