Skip to content

engines: carry the parallel tool policy to Claude models on Converse - #95

Merged
CMGS merged 3 commits into
mainfrom
fix/converse-anthropic-parallel-policy
Sep 18, 2026
Merged

CMGS merged 3 commits into
mainfrom
fix/converse-anthropic-parallel-policy

Conversation

@CMGS

@CMGS CMGS commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

What

Follow-up to #94. On Converse a chat client's parallel_tool_calls stayed in additionalModelRequestFields because toolConfig has no parallel flag; the OpenAI family accepts it there, a Claude model answers 400 "parallel_tool_calls: Extra inputs are not permitted" — while the same request succeeds on anthropic-messages and aws-anthropic.

Bedrock accepts the whole tool_choice (with disable_parallel_tool_use) in additionalModelRequestFields as long as toolConfig.toolChoice is absent, and refuses the pair ("The additional field tool_choice/type conflicts with the existing field toolConfig.toolChoice.auto"); a flat disable_parallel_tool_use is refused too. So for a Claude model the policy becomes tool_choice.disable_parallel_tool_use and that choice rides in the extras instead of toolConfig; other families keep parallel_tool_calls there unchanged.

  • converse::request: claude_model(model) (the existing contains("claude") check, now shared) gates the routing; one line of comment carries the vendor constraint.
  • claude_engine: the parallel_tool_callsdisable_parallel_tool_use mapping runs on Converse too when the model is Claude.
  • tests: the Converse integration test now walks both families × both policies; a unit test pins the routing and the OpenAI-family passthrough.
  • docs: API page states the Converse behavior per family.

Hot path

Zero increment by construction: the family check reuses the one already computed in request, and a routed choice is one fewer move (it stays in the body that becomes additionalModelRequestFields). Requests without tool_choice, and every non-Converse wire, take the same path as before.

A/B bench on the Linux testbed (384 cores), mock upstream, oha -c 256 -z 15s per shape, six rounds with the arms interleaved and the order swapped each round. Arm A = main before #94 (a66250a), arm B = this head, so the table covers the #94 paths and this one:

shape A rps (median) B rps (median) Δ rps A p50 B p50 Δ p50 paired Δ rps per round
chat, no tools (untouched path) 237884 240585 +1.1% 0.358 ms 0.360 ms +0.7% +2 +5 −1 −4 −1 +6
chat + tools + tool_choice + parallel_tool_calls → openai-chat 217502 208700 −4.0% 0.339 ms 0.329 ms −2.8% −22 +14 −6 0 −2 −12
/v1/messages + tools + {type: any, disable_parallel_tool_use} → openai-chat 162964 164300 +0.8% 0.338 ms 0.342 ms +1.0% −19 −17 +27 +42 +7 −15
chat + tools + parallel_tool_calls → anthropic-messages 263682 271192 +2.8% 0.348 ms 0.362 ms +4.0% +9 −30 +4 +25 +1 +48
chat + tools + parallel_tool_calls → aws-converse (this PR) 159053 152609 −4.1% 0.312 ms 0.307 ms −1.4% +11 −6 −10 −23 +20 −14

Every per-round paired delta changes sign across the six rounds and the medians sit within ±4% with rps and p50 disagreeing on direction, so the run resolves nothing below its own ±10–40% round-to-round noise. Success rate 1.000 on every shape and arm.

The two shapes with a −4% median were re-run at -c 64 (noise sd ≈3%), eight rounds, and then once more as an A/A control with the baseline binary on both arms:

shape A/B mean Δ rps (8 rounds) A/A control mean Δ rps (8 rounds)
chat + tools + tool_choice + parallel_tool_calls → openai-chat −1.7% (sd 2.6) −1.7% (sd 3.3)
chat + tools + parallel_tool_calls → aws-converse +2.0% (sd 3.7) +1.8% (sd 8.4)

The identical binary on both arms reproduces the same signed offsets, so the residual is the harness's arm-position bias, not code; the source delta on the openai-chat path is two lookups on a three-key map (under 0.1 µs against a 227 µs p50). No regression.

Gates

cargo fmt --check clean; cargo clippy -p gw-engines --all-targets -- -D warnings clean; engines tests 215 passed locally. Linux (rustc 1.98.0, kernel 6.12) cargo test --workspace at this head: 32 suites, 632 passed, 0 failed, 4 ignored (the Redis- and Postgres-gated suites, no credentials on the testbed).

Live (Linux testbed, real Bedrock us-east-1, Gemini, OpenAI, Anthropic: 22/22 on this head, gateway logs 0 WARN/ERROR)

check before after
chat parallel_tool_calls: false + tools → Converse us.anthropic.claude-haiku-4-5 424 carrying Bedrock 400 "parallel_tool_calls: Extra inputs are not permitted" 200, one tool call
same → Converse openai.gpt-oss-20b 200 200
same → aws-anthropic InvokeModel global.anthropic.claude-haiku-4-5 200 200
Converse haiku without the field (control) 200 200
the #94 suite (Gemini signatures, DLP replay index, body cap, tool_choice shapes, stop_reason, parallel policy on direct wires) 18/18 18/18

Lines: production +14, tests +33, comments +2 (two one-line docs), docs +5/−3.

Converse has no parallel flag in toolConfig, so a chat client's
parallel_tool_calls stayed in additionalModelRequestFields and a Claude
model answered 400 "parallel_tool_calls: Extra inputs are not permitted"
while the same request succeeded on anthropic-messages and aws-anthropic.
Bedrock accepts the whole tool_choice (with disable_parallel_tool_use) in
additionalModelRequestFields as long as toolConfig.toolChoice is absent,
and refuses the pair ("conflicts with the existing field
toolConfig.toolChoice"). For a Claude model the policy now becomes
tool_choice.disable_parallel_tool_use and that choice rides in the extras
instead of toolConfig; other families keep parallel_tool_calls there.
Probed directly against Bedrock us-east-1 and seen live through the
gateway on haiku-4-5 and gpt-oss-20b.
claude_model had landed between the doc block and the fn; the one WHY line fits 100 columns.
@CMGS
CMGS merged commit c103564 into main Sep 18, 2026
2 checks passed
@CMGS
CMGS deleted the fix/converse-anthropic-parallel-policy branch September 18, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant