AWS STS - STS cluster is destroyed on CDS update

[michaelfinch]

When attempting to follow method 3 here https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/aws_request_signing_filter#credentials, the STS cluster created by the filter is created as a dynamic cluster. If delta xDS is not being used, this dynamic cluster will be deleted on the next CDS update the envoy receives, which will cause AWS request signing to fail.

Repro steps:

1. Set envoy.reloadable_features.use_http_client_to_fetch_aws_credentials to true.
2. Confirm that AWS request signing works as expected.
3. Without using delta xDS, deliver a CDS update to the envoy.
4. Confirm that AWS request signing is now broken.

Here are the debug logs seen when a CDS update is received

[2024-07-02 17:38:06.614][15][debug][init] [external/envoy/source/common/init/watcher_impl.cc:31] init manager Cluster sts_token_service_internal destroyed
[2024-07-02 17:38:06.614][15][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:859] removing cluster sts_token_service_internal
[2024-07-02 17:38:06.614][15][debug][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:863] removing TLS cluster sts_token_service_internal
[2024-07-02 17:38:06.614][15][debug][upstream] [external/envoy/source/common/upstream/cds_api_helper.cc:66] cds: remove cluster 'sts_token_service_internal'

Here is where the STS cluster is created https://github.com/envoyproxy/envoy/blob/main/source/extensions/common/aws/credentials_provider_impl.cc#L150. The function is named createInternalClusterStatic, but I confirmed in config dump that the cluster is actually created as a dynamic cluster. Is there a way to create a static cluster that won't get wiped out by CDS updates?

[htuch]

@derekargueta @suniltheta @mattklein123 @marcomagdy @nbaws

[nbaws]

will grab this one