create secret for multiple SSL certificates in HTTPS load balancing with one Ingress

[silandrew]

I want to create a secrets for multiple SSL certificates with one Ingress controller, this is working with one domain ssl certificate config but deosent cretate secrets with 2

#domain A secret class

------
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: ingress-tls
  namespace: test
spec:
  provider: azure
  secretObjects:                                
  - secretName: ingress-tls-csi    #name of the secret that gets created - this is the value we provide to nginx
    type: kubernetes.io/tls
    data: 
    - objectName: a
      key: tls.key
    - objectName: a
      key: tls.crt
  parameters:
    usePodIdentity: "true"   #since we are using aad pod identity we set this to true
    #userAssignedIdentityID: "xxxxxxxxxxxx"
    keyvaultName: "xxxx"                        
    objects: |
      array:
        - |
          objectName: a
          objectType: secret
    tenantId: "xxxxxxxx"      # the tenant ID of KeyVaul

---

#domain B secret class

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: ingress-tls-b
  namespace: test
spec:
  provider: azure
  secretObjects:                                
  - secretName: ingress-b-tls-csi
  #name of the secret that gets created - this is the value we provide to nginx
    type: kubernetes.io/tls
    data: 
    - objectName: b
      key: tls.key
    - objectName: b
      key: tls.crt
  parameters:
        keyvaultName: "xxxx"                        
    objects: |
      array:
        - |
          objectName: b
          objectType: secret
    tenantId: "xxx"      # the tenant ID of KeyVaul

---

####helm config

helm install ingress-nginx/ingress-nginx --generate-name --namespace=portal --set controller.replicaCount=2 --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz --set controller.podLabels.aadpodidbinding=mi-xxxxxxx -f - <<EOF
controller:
  extraVolumes:
      - name: secrets-store-inline
        csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "ingress-tls"   #name of the SecretProviderClass we created above
  extraVolumeMounts:
      - name: secrets-store-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
EOF

and this is the config for ingress config

---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test
  annotations:
        kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
    
spec:
  tls:
  - hosts:
      - "*.a.com"
    secretName: ingress-tls-csi    
    
  - hosts:
      - "*.b.io"
    secretName: ingress-b-tls-csi
  rules:
  - host: "*.a.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: 
            port:
              number: 80
  - host: "test.b.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: b
            port:
              number: 80
---    

[aramase]

@silandrew Even though you have 2 secret provider classes, you're only referencing one in the pod volume mount. So only the sync secret specified in ingress-tls will be created. You can either:

1. Pool both secret provider classes into 1 and have 2 secret objects, one each for secret a and b or
2. Create 2 volumes (one for secret provider class ingress-tls and one for ingress-tls-b.

If both secrets are being fetched from same keyvault, I would recommend (1) as it reduces the scope for runtime (volume mount) failures.

[silandrew]

this works only with 2 ingress controllers because in ingress helm has been specifying secretProviderClass: "ingress-tls",
I've tried to add a second class in the following format bellow but this didnt work
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "ingress-tls" #name of the SecretProviderClass we created above
secretProviderClass: "ingress-tls-b"

---

working ingress helm config

helm install ingress-nginx/ingress-nginx --generate-name --namespace=a --set controller.ingressClass=ingress-a --set controller.ingressClassResource.name=ingress-a --set controller.replicaCount=2 --set controller.nodeSelector."beta.kubernetes.io/os"=linux --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux --set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz --set controller.podLabels.aadpodidbinding=mi-aksxxx-f - <<EOF
controller:
extraVolumes:
- name: secrets-store-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "ingress-tls" #name of the SecretProviderClass we created above

extraVolumeMounts:
- name: secrets-store-inline
mountPath: "/mnt/secrets-store"
readOnly: true
EOF