Context
Split out of the perf program (AISIX-Cloud#1259, same-protocol passthrough gate). That gate measured, on the benchmark payload, a full-delete upper bound of 3.2–4.6% CPU (≈3.6–5.1 µs/req at c=128) for everything serde/render on the chat path — a correct no-go at that payload size (86 bytes).
The no-go does not generalize to large bodies. Serde cost scales roughly linearly with body size while the rest of the request path is roughly constant. The realistic worst case is multimodal chat: images embedded as base64 data URIs inside messages[].content, with clients resending the whole conversation history every turn — request bodies of hundreds of KB to MB. Today every such request pays, on /v1/chat/completions:
- full typed deserialization of the entire body (axum
Json<ChatFormat>, chat.rs:100)
- an unconditional typed →
serde_json::Value full-tree conversion (prepare_outbound_body, openai bridge.rs:314) — even when the target has no body overrides
- full re-serialization on the way out (
bridge.rs:410)
Three O(body) passes plus a heap allocation for every content string, per request, for bytes the gateway never reads. Peak memory per request is a small multiple of body size.
Proposal — skeleton parsing
Really parse only the fields the gateway reads or rewrites; keep bulk subtrees as raw bytes (serde_json::value::RawValue) and emit them verbatim.
Request side (no user-visible behavior change — the upstream receives the same bytes):
- really parse:
model (read + rewrite), stream, stream_options
- keep raw:
messages, tools (the two size dominants)
- gate: applies only when no content-consuming feature is enabled for the request (guardrails / PII redaction / semantic cache / token estimation fall back to full parse)
/v1/messages is a simpler first target: its request side is already a Value round-trip whose only rewrite is model (messages.rs:1005).
The response side is a separate decision: envelope fields would stay parsed (model restamp per AISIX-Cloud#410, created, usage extraction) with choices kept raw — but today's render emits only choices[0] and drops unmodeled fields, so raw response passthrough is a user-visible behavior change (n>1 choices and unknown fields would start passing through), not a pure optimization.
Known constraints
- The chat request model keeps unknown fields via
#[serde(flatten)]; serde's flatten is incompatible with RawValue, so this needs a hand-rolled partial deserializer or a reshaped extractor.
- Streaming chunks are small; skeleton parsing buys little on the SSE path. The win is the request side and the non-streaming response envelope.
Gate / definition of done
- Payload-size sweep first, no implementation before it: same-session anchors at c=128 with base64-image-shaped
messages at e.g. 1 KB / 8 KB / 64 KB / 512 KB; publish the serde-share growth curve and the crossover size here.
- Survey how at least three mainstream gateways handle large-body same-protocol forwarding, and cite the serde
RawValue documentation for the mechanism.
- Go: request-side skeleton for the chat family plus
/v1/messages, wire-equivalent for the default posture, with tests proving byte-for-byte upstream request equivalence. No-go: record the sweep numbers here and close.
Not scheduled — backlog until a workload with large multimodal payloads makes it worth pulling forward, or the current perf program items land.
Context
Split out of the perf program (AISIX-Cloud#1259, same-protocol passthrough gate). That gate measured, on the benchmark payload, a full-delete upper bound of 3.2–4.6% CPU (≈3.6–5.1 µs/req at c=128) for everything serde/render on the chat path — a correct no-go at that payload size (86 bytes).
The no-go does not generalize to large bodies. Serde cost scales roughly linearly with body size while the rest of the request path is roughly constant. The realistic worst case is multimodal chat: images embedded as base64 data URIs inside
messages[].content, with clients resending the whole conversation history every turn — request bodies of hundreds of KB to MB. Today every such request pays, on/v1/chat/completions:Json<ChatFormat>,chat.rs:100)serde_json::Valuefull-tree conversion (prepare_outbound_body, openaibridge.rs:314) — even when the target has no body overridesbridge.rs:410)Three O(body) passes plus a heap allocation for every content string, per request, for bytes the gateway never reads. Peak memory per request is a small multiple of body size.
Proposal — skeleton parsing
Really parse only the fields the gateway reads or rewrites; keep bulk subtrees as raw bytes (
serde_json::value::RawValue) and emit them verbatim.Request side (no user-visible behavior change — the upstream receives the same bytes):
model(read + rewrite),stream,stream_optionsmessages,tools(the two size dominants)/v1/messagesis a simpler first target: its request side is already aValueround-trip whose only rewrite ismodel(messages.rs:1005).The response side is a separate decision: envelope fields would stay parsed (
modelrestamp per AISIX-Cloud#410,created,usageextraction) withchoiceskept raw — but today's render emits onlychoices[0]and drops unmodeled fields, so raw response passthrough is a user-visible behavior change (n>1 choices and unknown fields would start passing through), not a pure optimization.Known constraints
#[serde(flatten)]; serde's flatten is incompatible withRawValue, so this needs a hand-rolled partial deserializer or a reshaped extractor.Gate / definition of done
messagesat e.g. 1 KB / 8 KB / 64 KB / 512 KB; publish the serde-share growth curve and the crossover size here.RawValuedocumentation for the mechanism./v1/messages, wire-equivalent for the default posture, with tests proving byte-for-byte upstream request equivalence. No-go: record the sweep numbers here and close.Not scheduled — backlog until a workload with large multimodal payloads makes it worth pulling forward, or the current perf program items land.