Skip to content

happy_eyeballs: sort upstream address list once per host#46240

Open
davidvo-lyft wants to merge 1 commit into
envoyproxy:mainfrom
davidvo-lyft:davidvo/32916-happy-eyeballs-sort-once
Open

happy_eyeballs: sort upstream address list once per host#46240
davidvo-lyft wants to merge 1 commit into
envoyproxy:mainfrom
davidvo-lyft:davidvo/32916-happy-eyeballs-sort-once

Conversation

@davidvo-lyft

Copy link
Copy Markdown
Contributor

Every upstream connection attempt to a multi-address host (logical DNS, EDS additional addresses) re-ran the RFC 8305 address interleave in the connection path: HappyEyeballsConnectionProvider's constructor called sortAddresses(), which per connection allocated a fresh result vector, a map of per-family buckets, and a family-order vector, then stored its own copy of the sorted list. The address list and the cluster happy_eyeballs config are immutable between address-list updates, so this was pure repeated work per connection. This change computes the sorted list once, where the address list is created or refreshed, and hands the pre-sorted shared vector to the provider. Issue #32916 asks for exactly this, and the header TODO endorsed sorting at resolution time.

HostDescriptionImpl caches the sorted list as a const member at construction; LogicalHost computes it in its constructor and inside setNewAddresses under the existing address_lock_, so the raw and sorted lists update atomically. HostImplBase gains a sortedAddressListOrNull() accessor implemented by HostImpl and LogicalHost (the only two subclasses), and the static createConnection now receives the pre-sorted vector; the provider no longer sorts or copies. addressListOrNull() still returns the original raw order for its existing consumers (admin config_dump, the HTTP/3 pool grid, and http3 conn_pool). Connection attempt order is byte-identical: the same sortAddresses function runs on the same input with the same config, just once and earlier.

sortAddresses cost (dual-family list) when it runs (before) when it runs (after)
2 addresses: 114 ns + ~6 heap allocs every connection attempt once per address-list update
8 addresses: 250 ns every connection attempt once per address-list update
16 addresses: 394 ns every connection attempt once per address-list update

Commit Message: happy_eyeballs: sort upstream address list once per host

Additional Description: Tradeoffs disclosed for transparency: multi-address hosts now cache a second vector of shared_ptrs (roughly 16 bytes per address plus one control block), and a host that always dials through an HTTP/1.1 proxy redirect sorts once per address-list refresh it never uses. Both are small and bounded versus the per-connection savings.

Risk Level: Low. Behavior preserving: the identical sort function runs on identical input with the identical config, only earlier and once; the existing sortAddresses ordering tests pass unchanged.

Testing:

  • New HostImplTest.HappyEyeballsSortsAddressListOncePerHost counts the existing sort address with happy_eyeballs config trace event: host creation sorts exactly once and two createConnection calls sort zero times. Red before the fix (creation 0, two connections 2), green after (1, 0). Verified in the CI clang docker image.
  • New LogicalHostSortedAddressListTest.SortedListFollowsAddressUpdates: the sorted list is interleaved at construction, recomputed by setNewAddresses, and cleared when there is no list, while the raw list order is preserved.
  • Existing sortAddresses ordering tests in happy_eyeballs_connection_provider_test pass unchanged (same static function), corroborating identical attempt order. Also run green: upstream_impl_test, multi_connection_base_impl_test, logical_host_test, logical_dns_cluster_test, eds_test.
  • New //test/common/network:happy_eyeballs_speed_test benchmark reports the sortAddresses cost that moves out of the connection path (2 addresses: 114 ns, 8: 250 ns, 16: 394 ns).

Docs Changes: N/A

Release Notes: changelogs/current/minor_behavior_changes/happy_eyeballs__sort-address-list-once.rst

Platform Specific Features: N/A

Fixes #32916

Generative AI disclosure: Developed with AI assistance (Claude Code); I reviewed and understand every line and take full ownership of the submission.

Previously every upstream connection attempt to a multi-address host re-ran
the RFC 8305 address interleave in HappyEyeballsConnectionProvider's
constructor, allocating a fresh sorted vector, per family buckets, and a
family order vector each time. The address list and the cluster happy
eyeballs config are immutable between address list updates, so the sort now
runs once: HostDescriptionImpl caches the sorted list at construction and
LogicalHost recomputes it under its existing address lock when addresses are
refreshed. The provider receives the pre-sorted shared vector and no longer
sorts or copies it, and addressListOrNull() still returns the original raw
order for its existing consumers. Connection attempt order is unchanged
because the same sortAddresses function runs, just earlier and once.

Fixes envoyproxy#32916

Signed-off-by: David Vo <davidvo@lyft.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@RyanTheOptimist RyanTheOptimist left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Are the failing tests real?

@davidvo-lyft

Copy link
Copy Markdown
Contributor Author

/retest

@davidvo-lyft

Copy link
Copy Markdown
Contributor Author

Thanks for taking a look. That failure was not related to this change: the one red test was contrib/peak_ewma's BasicLoadBalancing, which asserts power-of-two-choices spreads requests across at least 2 upstreams and that run happened to see only 1 (a randomized load-distribution check). This PR only changes how a single multi-address host's addresses are ordered for happy eyeballs and keeps the connection attempt order byte-identical; it does not touch contrib/peak_ewma or cross-host load balancing. I re-triggered CI and it is green now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Happy Eyeball routines should not call sortAddress() in data path during creating connection

2 participants