CIC not linking intermediate certificates

[eugene-eeo]

Describe the bug
Intermediate certificates are not presented during the TLS handshake,
even though the uploaded certificate contents contain the intermediate.

We are using CIC version v1.39.6, and NetScaler MPX 13.1 51.15.nc.

To Reproduce
For example, with the following configuration:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls
spec:
  dnsNames:
  - <redacted>
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-thgice
  secretName: tls
  usages:
  - digital signature
  - key encipherment
---
apiVersion: v1
kind: Service
metadata:
  name: cert-test
  annotations:
    service.citrix.com/secret: tls
    service.citrix.com/service-type-0: SSL
    service.citrix.com/ssl-termination-0: REENCRYPT
spec:
  loadBalancerIP: <redacted>
  type: LoadBalancer
  selector:
    app: nginx
  ports:
  - name: https
    port: 443
    targetPort: https

and a backend configured to accept TLS -- e.g. nginx with something like:

events {}
http {
    access_log /dev/stdout;
    server {
        listen              443 ssl;
        server_name         <redacted>;
        ssl_certificate     /etc/nginx/certs/tls.crt;
        ssl_certificate_key /etc/nginx/certs/tls.key;
        location / {
            add_header Content-Type text/plain;
            return 200 'hello\n';
        }
    }
}

the secret does contain the intermediate cert (R11):

$ kubectl get secret -n cert-test tls -o json | jq -r '.data["tls.crt"]' | base64 -d
-----BEGIN CERTIFICATE-----
... REDACTED -- for REDACTED_DOMAIN
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
... REDACTED, openssl x509 output:
... Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
... Validity
...     Not Before: Mar 13 00:00:00 2024 GMT
...     Not After : Mar 12 23:59:59 2027 GMT
... Subject: C=US, O=Let's Encrypt, CN=R11
-----END CERTIFICATE-----

But we see that openssl reports the intermediate certificate isn't served:

$ openssl s_client -connect REDACTED_IP:443 -showcerts -servername REDACTED_DOMAIN
CONNECTED(00000003)
depth=0 CN = REDACTED_DOMAIN
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = REDACTED_DOMAIN
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = REDACTED_DOMAIN
verify return:1
---
Certificate chain
 0 s:CN = REDACTED_DOMAIN
   i:C = US, O = Let's Encrypt, CN = R11
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 22 13:01:41 2024 GMT; NotAfter: Nov 20 13:01:40 2024 GMT
-----BEGIN CERTIFICATE-----
... REDACTED -- for REDACTED_DOMAIN
-----END CERTIFICATE-----
---
Server certificate
subject=CN = REDACTED_DOMAIN
issuer=C = US, O = Let's Encrypt, CN = R11
---
No client certificate CA names sent
---
SSL handshake has read 1472 bytes and written 675 bytes
Verification error: unable to verify the first certificate
---

Expected behavior
The intermediate cert should appear in openssl s_client.

Logs

2024-08-22 14:15:29,350  - INFO - [nitrointerface.py:handle_frontend_cs_vserver:5151] (MainThread) Configuring csvserver: autoiceub9js-cert-test_443_cert-test_svc and associated services
2024-08-22 14:15:29,855  - INFO - [nitrointerface.py:_create_nsapp_cs_vserver:3914] (MainThread) csvserver autoiceub9js-cert-test_443_cert-test_svc is created successfully
2024-08-22 14:15:29,856  - INFO - [NSAppInterfacePriorityPoolManager.py:createPriorityPool:424] (MainThread) NSPriorityPoolManagerPerCSVS: Skipping since autoiceub9js-cert-test_443_cert-test_svc CS VS already has pool
2024-08-22 14:15:29,856  - INFO - [referencemanager.py:process_unmanaged_add_event:1051] (MainThread) Adding unmanaged entity: cert-test.csvserver_lbsvc.cert-test - autoiceub9js-cert-test_443_cert-test_svc
2024-08-22 14:15:30,676  - INFO - [nitrointerface.py:nsapp_handle_certkeys:3148] (MainThread) ssl vserver:autoiceub9js-cert-test_443_cert-test_svc default cert, server cert to remove:[] , server cert to add:['autoiceub9js-XPTECDJF43U5DG5HWL']
2024-08-22 14:15:31,529  - INFO - [nitrointerface.py:_add_cert_key:3643] (MainThread) certkey autoiceub9js-XPTECDJF43U5DG5HWL create successful
2024-08-22 14:15:31,624  - INFO - [nitrointerface.py:create_entities_for_policy:2806] (MainThread) Processing lbvserver:autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu for csvserver:autoiceub9js-cert-test_443_cert-test_svc service type for lbvserver: http service type for servicegroup:ssl
2024-08-22 14:15:32,761  - INFO - [nitrointerface.py:_create_nsapp_vserver:1741] (MainThread) lbvserver autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu with ip: 0.0.0.0 and port:0  is created successfully
2024-08-22 14:15:33,130  - INFO - [nitrointerface.py:_update_lbvserver_params:1774] (MainThread) lbvserver autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu is updated successfully
2024-08-22 14:15:33,919  - INFO - [nitrointerface.py:_bind_default_cs_policy:4273] (MainThread) csvserver autoiceub9js-cert-test_443_cert-test_svc binding to lbvserver autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu as default policy is successful
2024-08-22 14:15:34,155  - INFO - [nitrointerface.py:_create_nsapp_service_group:2149] (MainThread) Servicegroup autoiceub9js-cert-test_443_sgp_da62s5lrkn3uuboq65r4trnlwrcxoqdu is created successfully
2024-08-22 14:15:34,222  - INFO - [nitrointerface.py:_update_servicegroup_params:1825] (MainThread) servicegroup autoiceub9js-cert-test_443_sgp_da62s5lrkn3uuboq65r4trnlwrcxoqdu is updated successfully
2024-08-22 14:15:34,766  - INFO - [nitrointerface.py:_bind_service_group_lb:2212] (MainThread) servicegroup autoiceub9js-cert-test_443_sgp_da62s5lrkn3uuboq65r4trnlwrcxoqdu bind to lbvserver autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu is successful
2024-08-22 14:15:35,021  - INFO - [nitrointerface.py:_configure_services_nondesired:2576] (MainThread) Binding 10.10.49.111:31982 from servicegroup autoiceub9js-cert-test_443_sgp_da62s5lrkn3uuboq65r4trnlwrcxoqdu is successful
2024-08-22 14:15:35,022  - INFO - [referencemanager.py:process_unmanaged_add_event:1051] (MainThread) Adding unmanaged entity: cert-test.lbvserver.autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu - autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu
2024-08-22 14:15:35,022  - INFO - [referencemanager.py:resolve_references:542] (MainThread) Resolve reference for CRD cert-test.lbvserver.autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu
2024-08-22 14:15:35,022  - INFO - [referencemanager.py:resolve:478] (MainThread) Resolving node cert-test.lbvserver.autoiceub9js-cert-test_443_lbv_da62s5lrkn3uuboq65r4trnlwrcxoqdu
2024-08-22 14:15:35,180  - INFO - [nitrointerface.py:_is_default_ssl_profile_enabled:7387] (MainThread) Default SSL profile is False
2024-08-22 14:15:35,232  - INFO - [nitrointerface.py:_update_ssl_vserver_sni_clientauth_status:3285] (MainThread) sni DISABLED, clientauth DISABLED set on ssl vserver autoiceub9js-cert-test_443_cert-test_svc
2024-08-22 14:15:35,232  - INFO - [nitrointerface.py:configure_ns_cs_app:5280] (MainThread) Finished processing instruction to configure autoiceub9js-cert-test_443_cert-test_svc app associated with autoiceub9js-cert-test_443_cert-test_svc csvserver
2024-08-22 14:15:35,232  - INFO - [nitrointerface.py:configure_apps_during_sync:4919] (MainThread) ADC-SYNC: Loadbalancer type service configuration completed. Number of services that are deleted and added are 1. Avergage down_time: 0:00:09.228632 Max down_time: 0:00:09.228632.

Additional context

Our certificates are generated using cert-manager + LetsEncrypt,
which gives us (today) a certificate signed using the R10/R11 intermediates.

What we find is that we need to manually link the certificate on the ADC UI
in order for the intermediate to be served.
It might be because in our environment, there's some existing automation that
uploads the R11 intermediate cert for other non-K8s LBs:

NAME	CERTIFICATE_TYPE	COMMON NAME	ISSUER NAME
LetsEncrypt-r11_CA_2024	INTM_CERT,...	R11	ISRG Root X1
autoiceub9js-XPTECDJF43U5DG5HWL	CLNT_CERT,SRVR_CERT	REDACTED_DOMAIN	R11