http: stop using retry_back_off.max_interval as implicit per-try timeout#46238
http: stop using retry_back_off.max_interval as implicit per-try timeout#46238davidvo-lyft wants to merge 1 commit into
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
Utility::convertCoreToRouteRetryPolicy copied the computed backoff max_interval into the route retry policy per_try_timeout, so every client built from config.core.v3.RetryPolicy aborted individual tries after the backoff interval. Operators setting a small max_interval saw every try time out almost immediately (issue envoyproxy#38847). Leave per_try_timeout unset so tries fall back to the caller's request timeout, matching route retry policy semantics. The legacy behavior is kept behind the runtime guard envoy.reloadable_features.core_retry_policy_no_implicit_per_try_timeout. Signed-off-by: David Vo <davidvo@lyft.com>
a5b72de to
ba692f9
Compare
|
LGTM, but should we add a "per try timeout" field to the core RetryPolicy? |
|
@botengyao wdyt about adding a per try timeout to the core RetryPolicy? |
|
Thanks @paul-r-gall . I agree an explicit This PR is deliberately the prerequisite for that: as long as the conversion silently copies Happy to open a follow-up issue and take on the field addition once this lands. Does that split sound good? |
Utility::convertCoreToRouteRetryPolicyunconditionally copied the computed backoffmax_intervalinto the route retry policy'sper_try_timeout. Every client that builds a retry policy fromconfig.core.v3.RetryPolicy(ext_authz, oauth2, jwt_authn, gcp_authn, the gRPC access logger, and the Envoy gRPC async client) therefore got a per-try deadline equal to its maximum backoff interval, a knob that has nothing to do with timeouts. As reported in #38847, an operator who set a smallmax_interval(1 to 2 ms in the repro) saw every try aborted almost immediately, with no config surface to express intent. This change stops settingper_try_timeoutduring the conversion, so each try is bounded by the caller's overall request timeout, matching route retry policy semantics whereper_try_timeoutdefaults to unset. The change is guarded by a runtime feature so it can be reverted without a rollback.Commit Message: http: stop using retry_back_off.max_interval as implicit per-try timeout
Additional Description: The conversion no longer copies
max_intervalintoper_try_timeout. Guarded byenvoy.reloadable_features.core_retry_policy_no_implicit_per_try_timeout(default true); setting it to false restores the previous behavior exactly. Removing this implicit copy is also a prerequisite for any future explicit per-try-timeout field onconfig.core.v3.RetryPolicy(as the issue author suggested): a dedicated field cannot coexist with the accidentalmax_intervaldefault without the two fighting. That API addition needs separate api review and is intentionally out of scope here.Risk Level: Medium. Behavior change for all consumers of the core-to-route retry conversion, runtime guarded and default on; the four HTTP consumers that always set an overall request timeout fall back to that deadline.
Testing:
//test/common/http:utility_test(HttpUtility.TestConvertCoreToRouteRetryPolicy): with the guard default-on, the converted policy has noper_try_timeoutfor both the default policy and an explicitretry_back_off(1s/5s) case, whilenum_retries,base_interval,max_interval, andretry_onstill convert; aTestScopedRuntimeblock flips the guard false and asserts the legacy 5s and 10s copies return. Red before the fix (assertshas_per_try_timeout()false but it was true), green after. Verified in the CI clang docker image.//test/common/grpc:async_client_impl_test(EnvoyAsyncClientImplTest.ParsedRetryPolicyWillBeUsed): aretry_back_offof base 1s / max 2s on the gRPC service yieldsperTryTimeout() == 0mson the parsed policy (was 2000ms). Red before, green after.Docs Changes: N/A
Release Notes: changelogs/current/minor_behavior_changes/http__core-retry-policy-per-try-timeout.rst (names the runtime flag and the revert procedure)
Platform Specific Features: N/A
Fixes #38847
Generative AI disclosure: Developed with AI assistance (Claude Code); I reviewed and understand every line and take full ownership of the submission.