Skip to content

fix(llm-client): fall back when an upstream connect times out - #854

Draft
ting-hong-shieh wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
ting-hong-shieh:fix/853-connect-timeout-fallback
Draft

ting-hong-shieh wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
ting-hong-shieh:fix/853-connect-timeout-fallback

Conversation

@ting-hong-shieh

Copy link
Copy Markdown
Contributor

What

convert_reqwest_error now checks is_connect() before is_timeout(). A connection that times out before it is established becomes LlmClientError::Transport instead of Timeout, so candidate fallback treats it like a refused connection.

Closes #853.

Why

reqwest reports a connect timeout as both is_connect() and is_timeout(). It was mapped to Timeout, and since #702 fallback_reason does not fall back on Timeout. So when the efficient target of a stage_router is powered off or behind a firewall that drops SYNs, the request fails with 504 after the retries instead of falling back to the capable target. 0.2.0 fell back in this case. A refused connection still fell back because it was already Transport.

The failing connection never sent the HTTP request, so falling back is as safe as it is for a refused connection. This narrows #702's rule to timeouts after the connection is established. Response timeouts and timeout_ms expiry still stop routing.

Notes for reviewers

Start with convert_reqwest_error in crates/libsy-llm-client/src/client.rs.

Behavior changes, all limited to connect-phase timeouts:

  • Candidate fallback in run.rs now tries the next candidate.
  • When no candidate is left, the client gets 502 upstream_error instead of 504 upstream_timeout.
  • Error labels change from timeout to transport in error.type, RouteErrorKind, and the classifier/advisor failure reason.
  • The escalation router's efficient call and single-model judge calls do not gain a fallback. Only their status code and labels change.

Unchanged:

  • Retries. Both variants were already retryable.
  • timeout_ms. The deadline can expire during a connect, and it still returns Timeout without fallback. Fallback happens only after every attempt has failed. On Linux, reqwest's default TCP_USER_TIMEOUT makes each connect attempt give up after about 30 s, so with the default max_retries = 2 fallback starts after about 90 s. A timeout_ms shorter than that still returns 504. Other platforms use the OS connect timeout.

Tests:

  • connect_timeout_is_a_transport_error (client.rs, Linux only) uses a listener with a full accept queue, so the kernel drops later SYNs. It sets reqwest's connect_timeout to 100 ms as a stand-in for the 30 s OS timeout; both produce is_connect() && is_timeout(). It fails on main with Timeout and passes with this change.
  • refused_connection_falls_back_to_the_next_candidate (run.rs) checks that a connect failure falls back through run. It passes on main too, so it guards existing behavior rather than reproducing the bug.

Docs: docs/reference/toml_schema.md and crates/libsy-llm-client/README.md now say that a connect timeout counts as a connection failure.

Validation:

cargo fmt --all --check
cargo clippy --workspace --all-targets --locked -- -D warnings
cargo test --workspace --locked
cd docs && make publish

Signed-off-by: Ting-Hong Shieh <shiehharry@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] stage_router no longer falls back to the capable target when the connection to the efficient upstream times out (0.3.0; 0.2.0 fell back)

1 participant