engines: carry the parallel tool policy to Claude models on Converse - #95
Merged
Merged
Conversation
Converse has no parallel flag in toolConfig, so a chat client's
parallel_tool_calls stayed in additionalModelRequestFields and a Claude
model answered 400 "parallel_tool_calls: Extra inputs are not permitted"
while the same request succeeded on anthropic-messages and aws-anthropic.
Bedrock accepts the whole tool_choice (with disable_parallel_tool_use) in
additionalModelRequestFields as long as toolConfig.toolChoice is absent,
and refuses the pair ("conflicts with the existing field
toolConfig.toolChoice"). For a Claude model the policy now becomes
tool_choice.disable_parallel_tool_use and that choice rides in the extras
instead of toolConfig; other families keep parallel_tool_calls there.
Probed directly against Bedrock us-east-1 and seen live through the
gateway on haiku-4-5 and gpt-oss-20b.
claude_model had landed between the doc block and the fn; the one WHY line fits 100 columns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #94. On Converse a chat client's
parallel_tool_callsstayed inadditionalModelRequestFieldsbecausetoolConfighas no parallel flag; the OpenAI family accepts it there, a Claude model answers 400 "parallel_tool_calls: Extra inputs are not permitted" — while the same request succeeds onanthropic-messagesandaws-anthropic.Bedrock accepts the whole
tool_choice(withdisable_parallel_tool_use) inadditionalModelRequestFieldsas long astoolConfig.toolChoiceis absent, and refuses the pair ("The additional field tool_choice/type conflicts with the existing field toolConfig.toolChoice.auto"); a flatdisable_parallel_tool_useis refused too. So for a Claude model the policy becomestool_choice.disable_parallel_tool_useand that choice rides in the extras instead oftoolConfig; other families keepparallel_tool_callsthere unchanged.converse::request:claude_model(model)(the existingcontains("claude")check, now shared) gates the routing; one line of comment carries the vendor constraint.claude_engine: theparallel_tool_calls→disable_parallel_tool_usemapping runs on Converse too when the model is Claude.Hot path
Zero increment by construction: the family check reuses the one already computed in
request, and a routed choice is one fewer move (it stays in the body that becomesadditionalModelRequestFields). Requests withouttool_choice, and every non-Converse wire, take the same path as before.A/B bench on the Linux testbed (384 cores), mock upstream,
oha -c 256 -z 15sper shape, six rounds with the arms interleaved and the order swapped each round. Arm A = main before #94 (a66250a), arm B = this head, so the table covers the #94 paths and this one:tool_choice+parallel_tool_calls→ openai-chat/v1/messages+ tools +{type: any, disable_parallel_tool_use}→ openai-chatparallel_tool_calls→ anthropic-messagesparallel_tool_calls→ aws-converse (this PR)Every per-round paired delta changes sign across the six rounds and the medians sit within ±4% with rps and p50 disagreeing on direction, so the run resolves nothing below its own ±10–40% round-to-round noise. Success rate 1.000 on every shape and arm.
The two shapes with a −4% median were re-run at
-c 64(noise sd ≈3%), eight rounds, and then once more as an A/A control with the baseline binary on both arms:tool_choice+parallel_tool_calls→ openai-chatparallel_tool_calls→ aws-converseThe identical binary on both arms reproduces the same signed offsets, so the residual is the harness's arm-position bias, not code; the source delta on the openai-chat path is two lookups on a three-key map (under 0.1 µs against a 227 µs p50). No regression.
Gates
cargo fmt --checkclean;cargo clippy -p gw-engines --all-targets -- -D warningsclean; engines tests 215 passed locally. Linux (rustc 1.98.0, kernel 6.12)cargo test --workspaceat this head: 32 suites, 632 passed, 0 failed, 4 ignored (the Redis- and Postgres-gated suites, no credentials on the testbed).Live (Linux testbed, real Bedrock us-east-1, Gemini, OpenAI, Anthropic: 22/22 on this head, gateway logs 0 WARN/ERROR)
parallel_tool_calls: false+ tools → Converseus.anthropic.claude-haiku-4-5openai.gpt-oss-20bglobal.anthropic.claude-haiku-4-5Lines: production +14, tests +33, comments +2 (two one-line docs), docs +5/−3.