fix(llm-client): fall back when an upstream connect times out - #854
Draft
ting-hong-shieh wants to merge 1 commit into
Draft
ting-hong-shieh wants to merge 1 commit into
ting-hong-shieh wants to merge 1 commit into
Conversation
Signed-off-by: Ting-Hong Shieh <shiehharry@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
convert_reqwest_errornow checksis_connect()beforeis_timeout(). A connection that times out before it is established becomesLlmClientError::Transportinstead ofTimeout, so candidate fallback treats it like a refused connection.Closes #853.
Why
reqwest reports a connect timeout as both
is_connect()andis_timeout(). It was mapped toTimeout, and since #702fallback_reasondoes not fall back onTimeout. So when the efficient target of astage_routeris powered off or behind a firewall that drops SYNs, the request fails with 504 after the retries instead of falling back to the capable target. 0.2.0 fell back in this case. A refused connection still fell back because it was alreadyTransport.The failing connection never sent the HTTP request, so falling back is as safe as it is for a refused connection. This narrows #702's rule to timeouts after the connection is established. Response timeouts and
timeout_msexpiry still stop routing.Notes for reviewers
Start with
convert_reqwest_errorincrates/libsy-llm-client/src/client.rs.Behavior changes, all limited to connect-phase timeouts:
run.rsnow tries the next candidate.upstream_errorinstead of 504upstream_timeout.error.type,RouteErrorKind, and the classifier/advisor failure reason.Unchanged:
timeout_ms. The deadline can expire during a connect, and it still returnsTimeoutwithout fallback. Fallback happens only after every attempt has failed. On Linux, reqwest's defaultTCP_USER_TIMEOUTmakes each connect attempt give up after about 30 s, so with the defaultmax_retries = 2fallback starts after about 90 s. Atimeout_msshorter than that still returns 504. Other platforms use the OS connect timeout.Tests:
connect_timeout_is_a_transport_error(client.rs, Linux only) uses a listener with a full accept queue, so the kernel drops later SYNs. It sets reqwest'sconnect_timeoutto 100 ms as a stand-in for the 30 s OS timeout; both produceis_connect() && is_timeout(). It fails onmainwithTimeoutand passes with this change.refused_connection_falls_back_to_the_next_candidate(run.rs) checks that a connect failure falls back throughrun. It passes onmaintoo, so it guards existing behavior rather than reproducing the bug.Docs:
docs/reference/toml_schema.mdandcrates/libsy-llm-client/README.mdnow say that a connect timeout counts as a connection failure.Validation: