[Bug] A target's extra_body is applied to another target that shares its model id

[himorishige]

Environment

• Repo / commit: NVIDIA-NeMo/Switchyard @ f30498d3 (refactor(python): remove legacy routing profiles #268) — also present on main @ fb3fc308
• Released switchyard-server 0.2.0 (crates.io): not affected (see below)
• Upstream: Fireworks AI (https://api.fireworks.ai/inference/v1), openai_chat format
• Client: curl / plain HTTP, no launcher

Description

When two targets share one model id and are reachable from the same route, the extra_body declared
on one target is applied to calls that the router sends to the other one.

In an llm_classifier deployment this is easy to hit, because using the cheap tier as its own judge
is a natural configuration: the classifier target and the weak target point at the same model, and
only their extra_body differs. The result is that user-facing generations on the weak tier are
served with the judge's parameters — in our case reasoning_effort: "none" and temperature: 0,
i.e. every routed weak answer came back with reasoning suppressed, while the same model called
through a passthrough route reasoned normally.

Nothing in the logs or the routing stats indicates this. The route reports the correct model, the
decision is correct, and the response is a valid completion — just generated under parameters the
operator declared for a different target.

Giving the two targets separate llm_clients (the workaround for the (llm client, model id)
dedupe warning added in #180) does not help: the per-route map is keyed by model id alone.

Reproduction

schema_version = 1

[llm_clients.fireworks]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"

# Distinct client, so the #180 dedupe warning does not fire and neither target is dropped.
[llm_clients.fireworks_judge]
format = "openai_chat"
base_url = "https://api.fireworks.ai/inference/v1"
api_key_env = "FIREWORKS_API_KEY"

[targets.weak]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"
llm_client = "fireworks"
# no extra_body: this target should reason at provider defaults

[targets.strong]
id = "accounts/fireworks/models/kimi-k3"
llm_client = "fireworks"

[targets.classifier]
id = "accounts/fireworks/models/deepseek-v4-flash-0731"   # same id as [targets.weak]
llm_client = "fireworks_judge"
extra_body = { reasoning_effort = "none", temperature = 0 }

[routes.auto]
id = "auto"
type = "llm_classifier"
mode = "capability"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
base_threshold = 0.75
threshold_step = 0.1
session_affinity = true

[routes.weak-only]
id = "weak-only"
type = "passthrough"
target = "weak"

Send the same prompt to auto (judged onto the weak tier) and to weak-only, and compare
reasoning_content on the response. Both are served by deepseek-v4-flash-0731.

Expected: both reason at provider defaults, because [targets.weak] declares no extra_body.
Actual on f30498d3: the auto response has no reasoning at all.

The released 0.2.0 build does not reproduce it

Running the identical config against two builds side by side, same prompts, same minute:

build	weak-only reasoning	auto reasoning	verdict
switchyard-server 0.2.0 (crates.io)	881 / 1,565 / 993 / 1,070	289 / 2,358 / 1,360 / 1,346	both tiers reason
f30498d3 (#268)	951 / 1,629 / 1,439 / 297	0 / 0 / 0 / 0	auto loses its reasoning

The released crate was published from d0b9d50b, which is 57 minutes before #268 merged, so the
regression window is #268 itself. Note that both binaries report switchyard-server 0.2.0, so the
version string does not distinguish them.

Root cause (as far as we traced it)

• crates/switchyard-server/src/config.rs — LlmTarget.semantic_name is set to the model id
  (config.id.clone()), not the target's TOML key, so two targets on one model are indistinguishable
  downstream.
• crates/switchyard-server/src/config.rs — build_client_router() builds a per-route map keyed by
  target.id. Two targets sharing a model id collide on insert, and the surviving entry (with its
  extra_body) serves both.
• crates/libsy/src/algorithms/fall_through.rs — route() resolves the decision by model-id string
  (score.target), so no target identity survives to the call site.

Before #268, build_targets() attached the client to each LlmTarget directly
(llm_client: Some(client)), so each target carried its own backend and the ids never had to be
disambiguated. The per-route model-id map was introduced with the #268 refactor.

Why the #180 warning does not catch this

#180 warns when two targets reuse a model id on one llm client, and the workaround is to give
them separate clients. That prevents the drop, but the per-route map added in #268 is keyed by model
id regardless of client, so the collision reappears one layer down — with no warning this time,
because at the client level the config is now legal.

Impact

Silent, and in the direction that is hardest to notice: the response is well-formed, the routing
decision is right, the reported model is right, and the only difference is the sampling/reasoning
parameters. We ran an A/B, a rubric calibration and a 52-run benchmark before noticing, and had to
re-derive our classifier threshold afterwards because the judge's own configuration had to change to
work around it.

Happy to test any patch against the reproduction above.

[ting-hong-shieh]

I'd like to work on this.

I traced the issue on current main (b256d936). build_targets() maps each configured target name to its ModelId, and build_client_router() keys the per-route client map by that same ModelId. When two callable targets share a model ID, the later entry replaces the earlier one, so one target's client and extra_body can serve calls intended for the other. This collision predates #373; the recent ModelId refactor preserved the same model-ID-based lookup.

I can add focused server regression coverage for an llm_classifier route whose classifier and weak targets share a model ID but use different clients and extra_body.

Before implementing the fix, could a maintainer confirm the intended behavior?

1. Reject duplicate callable model IDs within one route at startup. This is the smaller safety fix and prevents the silent crossover, but it does not support the configuration described in this issue.
2. Preserve configured target identity separately from the upstream model ID, allowing both targets to work with their own client and extra_body. This would require a wider change across the server, routing algorithm, and client boundary.

I'm happy to proceed with either direction once confirmed.

[himorishige]

Thanks for picking this up, @ting-hong-shieh — and for re-tracing it on current main. That matches
what we saw on f30498d3.

For whatever an operator's view is worth here, since we happen to run the configuration in question:
my own leaning is toward option 2. The call is yours, of course.

In our setup the weak tier doubling as its own judge isn't really a convenience — it's what keeps
the classifier from eating the savings. With Fireworks and DeepSeek V4 Flash as both weak and
classifier, the classifier came to 8% of total spend across a 39-run agentic benchmark, and that
figure rests on two things at once:

• the judge runs on the cheapest model already in the pool rather than a second, pricier one
• the judge's extra_body suppresses its reasoning — without reasoning_effort: "none" the same
  judge calls go from 63 completion tokens flat to 303–636, and from 0.9–1.4 s to 3.2–4.8 s

As I read it, option 1 would make those two mutually exclusive: once the judge needs its own
extra_body it needs its own target, and the cheapest target is by definition the one weak traffic
already uses.

More generally — and this is the part I'm least sure generalises beyond our own setup — the
underlying request feels legitimate to me rather than a corner case: a judge and a serving call want
different sampling parameters from the same model. Determinism and suppressed reasoning are right
for a routing decision and wrong for the user-facing generation, even when one model serves both.
The same shape can appear outside classifiers too, e.g. one model exposed on two routes at different
temperatures. Option 1 would treat that as a configuration error rather than a valid intent.

Moving the judge to a different model isn't quite a free workaround either: base_threshold is
calibrated against the confidence distribution of one specific judge, so swapping the judge means
re-deriving the threshold rather than editing a line.

None of that is an argument against option 1 landing first, if that's the faster path to stopping
the bleeding. Being rejected at startup is strictly better than the silent parameter crossover we
hit — it would have saved us an A/B, a rubric calibration and a re-derived threshold. I'd only hope
it wouldn't end up as the final shape.

If it's any use for option 2 — you've traced this further than I have, so take it or leave it: the
identity we need already exists in the config as the TOML table key. LlmTarget.semantic_name
currently carries config.id; carrying the target key there instead, and keying the per-route map
on it, might keep both targets addressable without changing the shape of the routing algorithm.

Happy to test a branch against our setup, if that helps — the reproduction is wired up and we have
both builds running side by side.

[ting-hong-shieh]

Thanks — those cost and latency numbers are helpful. I agree that option 2 is the right outcome. Rejecting the configuration would prevent the silent crossover, but it would also rule out a valid use case.

I’ve added a focused regression test against b256d936. It configures the classifier and weak targets with the same model ID but separate clients, with reasoning_effort = "none" only on the classifier target. On current main, the judge client receives both calls and the serving client receives none. The existing multi-client test using different model IDs still passes.

I also tested using the TOML target key as a route-local identity. That correctly separates the clients and makes the regression pass, but the target key then leaks into stage-router statistics and fallback rationale, breaking two existing server tests. I dropped that prototype.

The fix therefore needs to preserve the upstream model ID in decisions, headers, statistics, and fallback messages, while using a separate identity only for client/backend selection. I’ll wait for a maintainer to confirm that duplicate model IDs within one route should be supported before choosing where that identity belongs. The regression test is ready, and once the direction is confirmed I can share a branch for testing against your deployment.