Title: Upstream TLS session cache in ClientContextImpl is not keyed by SNI, causing cross-host session resumption (SAN verification failures) in dynamic_forward_proxy clusters
Description:
A single dynamic_forward_proxy cluster that fronts multiple upstream hostnames sharing an IP address (e.g. hosts behind the same CDN anycast IP) will intermittently fail TLS verification with fail_verify_san, returning a 503 to the downstream client.
Root cause: the upstream TLS session cache in ClientContextImpl (source/common/tls/client_context_impl.{h,cc}) is a single unkeyed std::deque<bssl::UniquePtr<SSL_SESSION>> session_keys_ per TLS context (one context per cluster). newSessionKey() stores a
session from every completed handshake regardless of server name, and newSsl() blindly attaches session_keys_.front() (the most-recently-stored session) to every new outbound connection, with no check that the cached session's server identity matches
the SNI being requested for the new connection.
When two hostnames resolve to the same upstream IP, a session established for host A can be resumed on a brand-new connection intended for host B. The abbreviated handshake completes with no Certificate message, so the peer certificate is inherited from
host A's session. Envoy then validates that certificate against host B's expected SANs and fails.
Expected behavior: a cached TLS session should only ever be offered for resumption on a connection whose server identity (SNI, and ideally the verify-SAN override) matches the identity under which the session was originally established. Session
resumption must never cause a connection to inherit a certificate for a different server name. The session cache should be keyed by SNI/hostname rather than being a single shared per-context queue.
Repro steps:
Environment: Envoy 1.39.0-dev, BoringSSL. Reproduced deterministically.
Preconditions:
- Two hostnames that resolve to the same upstream IP and are served by different certificates — e.g. host-a.example (cert SANs [host-a.example]) and host-b.example (cert SANs [host-b.example]), both resolving to the same CDN anycast IP. Any two hosts
sharing an IP with distinct certs will do.
- A single dynamic_forward_proxy cluster with an UpstreamTlsContext that has upstream session resumption enabled (max_session_keys > 0, the default) and SAN verification configured (auto_sni / auto_san_validation, or per-request SAN override via
verifySubjectAltNameListOverride).
Steps:
- Send request 1 through the cluster with Host: host-a.example. Full handshake succeeds (200), and the resulting SSL_SESSION is stored at the front of session_keys_.
- Immediately send request 2 with Host: host-b.example. A new upstream connection is created (the pool may be empty/idle — this is not connection-pool reuse). newSsl() attaches the cached host-a.example session; the resumed/abbreviated handshake
completes with no Certificate message and the inherited host-a.example cert.
- SAN validation runs expected SANs: [host-b.example] against certificate SANs: [host-a.example] → CERTIFICATE_VERIFY_FAILED → upstream reset → 503 to the client.
Config:
Minimal UpstreamTlsContext on the DFP cluster (session resumption on by default + SAN verification):
clusters:
- name: dynamic_forward_proxy_cluster
lb_policy: CLUSTER_PROVIDED
cluster_type:
name: envoy.clusters.dynamic_forward_proxy
typed_config:
"@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
dns_cache_config:
name: dynamic_forward_proxy_cache_config
dns_lookup_family: V4_ONLY
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
auto_sni: true
auto_san_validation: true
common_tls_context:
validation_context:
trusted_ca: { filename: /etc/ssl/certs/ca-certificates.crt }
# max_session_keys defaults > 0 → upstream session resumption enabled → bug triggers.
# Setting common_tls_context.tls_params.max_session_keys: 0 is the current workaround.
Paired with a dynamic_forward_proxy HTTP filter on the listener so the upstream host is taken from the Host header. Any config that routes multiple distinct SNIs through one TLS context reproduces it.
Logs:
Envoy debug logs (connection, tls) for the failing request. C1002 is a brand-new connection (connecting to :443 — the pool was empty, so this is not pooled-connection reuse); it resumes the previously-cached host-a.example session and fails
SAN validation for host-b.example:
[connection] [connection_impl.cc:1183] [C1002] connecting to :443
[connection] [connection_impl.cc:1202] [C1002] connection in progress
[connection] [connection_impl.cc:878] [C1002] connected
[connection] [ssl_handshaker.cc:156] [C1002] ssl error occurred while read: WANT_READ
[connection] [default_validator.cc:267] verify cert failed: verify SAN list, expected SANs: [host-b.example], certificate SANs: [host-a.example]
[connection] [ssl_socket.cc:269] [C1002] remote address::443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:verify cert failed: verify SAN list, expected SANs: [host-b.example], certificate SANs:
[host-a.example]:TLS_error_end
[connection] [connection_impl.cc:361] [C1002] closing socket: 0
Resulting upstream reset and downstream 503:
[router] [router.cc:1596] [C1000][S1] upstream reset: reset reason: remote connection failure, transport failure reason: TLS_error:...CERTIFICATE_VERIFY_FAILED:verify cert failed: verify SAN list, expected SANs: [host-b.example], certificate SANs:
[host-a.example]:TLS_error_end
[http] [filter_manager.cc:1158] [C1000][S1] Sending local reply with details upstream_reset_before_response_started{remote_connection_failure|...CERTIFICATE_VERIFY_FAILED:...expected SANs: [host-b.example], certificate SANs: [host-a.example]...}
Access log: UF / URX response flags, response code 503, upstream_reset_before_response_started.
Title: Upstream TLS session cache in ClientContextImpl is not keyed by SNI, causing cross-host session resumption (SAN verification failures) in dynamic_forward_proxy clusters
Description:
A single dynamic_forward_proxy cluster that fronts multiple upstream hostnames sharing an IP address (e.g. hosts behind the same CDN anycast IP) will intermittently fail TLS verification with fail_verify_san, returning a 503 to the downstream client.
Root cause: the upstream TLS session cache in ClientContextImpl (source/common/tls/client_context_impl.{h,cc}) is a single unkeyed std::deque<bssl::UniquePtr<SSL_SESSION>> session_keys_ per TLS context (one context per cluster). newSessionKey() stores a
session from every completed handshake regardless of server name, and newSsl() blindly attaches session_keys_.front() (the most-recently-stored session) to every new outbound connection, with no check that the cached session's server identity matches
the SNI being requested for the new connection.
When two hostnames resolve to the same upstream IP, a session established for host A can be resumed on a brand-new connection intended for host B. The abbreviated handshake completes with no Certificate message, so the peer certificate is inherited from
host A's session. Envoy then validates that certificate against host B's expected SANs and fails.
Expected behavior: a cached TLS session should only ever be offered for resumption on a connection whose server identity (SNI, and ideally the verify-SAN override) matches the identity under which the session was originally established. Session
resumption must never cause a connection to inherit a certificate for a different server name. The session cache should be keyed by SNI/hostname rather than being a single shared per-context queue.
Repro steps:
Environment: Envoy 1.39.0-dev, BoringSSL. Reproduced deterministically.
Preconditions:
sharing an IP with distinct certs will do.
verifySubjectAltNameListOverride).
Steps:
completes with no Certificate message and the inherited host-a.example cert.
Config:
Minimal UpstreamTlsContext on the DFP cluster (session resumption on by default + SAN verification):
Paired with a dynamic_forward_proxy HTTP filter on the listener so the upstream host is taken from the Host header. Any config that routes multiple distinct SNIs through one TLS context reproduces it.
Logs:
Envoy debug logs (connection, tls) for the failing request. C1002 is a brand-new connection (connecting to :443 — the pool was empty, so this is not pooled-connection reuse); it resumes the previously-cached host-a.example session and fails
SAN validation for host-b.example:
[connection] [connection_impl.cc:1183] [C1002] connecting to :443
[connection] [connection_impl.cc:1202] [C1002] connection in progress
[connection] [connection_impl.cc:878] [C1002] connected
[connection] [ssl_handshaker.cc:156] [C1002] ssl error occurred while read: WANT_READ
[connection] [default_validator.cc:267] verify cert failed: verify SAN list, expected SANs: [host-b.example], certificate SANs: [host-a.example]
[connection] [ssl_socket.cc:269] [C1002] remote address::443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:verify cert failed: verify SAN list, expected SANs: [host-b.example], certificate SANs:
[host-a.example]:TLS_error_end
[connection] [connection_impl.cc:361] [C1002] closing socket: 0
Resulting upstream reset and downstream 503:
[router] [router.cc:1596] [C1000][S1] upstream reset: reset reason: remote connection failure, transport failure reason: TLS_error:...CERTIFICATE_VERIFY_FAILED:verify cert failed: verify SAN list, expected SANs: [host-b.example], certificate SANs:
[host-a.example]:TLS_error_end
[http] [filter_manager.cc:1158] [C1000][S1] Sending local reply with details upstream_reset_before_response_started{remote_connection_failure|...CERTIFICATE_VERIFY_FAILED:...expected SANs: [host-b.example], certificate SANs: [host-a.example]...}
Access log: UF / URX response flags, response code 503, upstream_reset_before_response_started.