Skip to content

Latest commit

 

History

History
315 lines (252 loc) · 15.6 KB

tls.md

File metadata and controls

315 lines (252 loc) · 15.6 KB
Raw

TLS

Certificate management within the openstack-operator is by default done using cert-manager. For each component the openstack-operator creates one or multiple certificates and pass the name/names of the k8s secret, which contains the certificate, key and ca certificate, to the service component. By default cert-manager does not delete k8s secrets, holding the certificate information, when the cert-manager certificate resource gets deleted. Consult the OpenShift documentation on Deleting a TLS secret automatically upon Certificate removal.

In the context of TLS (Transport Layer Security) configuration within the OpenStackControlPlane, two primary levels of traffic encryption are distinguished: ingress and podLevel:

  • ingress refers to the public (external) traffic that enters the cluster via a route
  • podLevel refers to the internal traffic within the cluster, specifically, the communication between pods, services and edpm nodes.

TLS is enabled by default for both ingress and podLevel communication.

Endpoint Termination overview

In terms of how traffic is managed within the cluster:

  • Public Endpoints
    • Public (external) endpoints are by default terminated at the route. The openstack-operator is responsible for creating these routes for each of the services
    • Using the serviceOverride it is possible to use a LoadBalancer service for public endpoints as well.
  • Internal Endpoints
    • Internal endpoints terminate at the designated service, this can be either ClusterIP or LoadBalancer type of service (e.g. MetalLB).

More details on networking check Networking

General TLS control

The default TLS settings can be changed using the tls section of the OpenStackControlPlane:

Example of a basic TLS section:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: myctlplane
spec:
  tls:
    ingress:
      enabled: true
    podLevel:
      enabled: false
    caBundleSecretName: myAdditionalCACerts

TLS Certificates

CA Certificates

By default, the openstack-operator provisions the following CAs via cert-manager for securing endpoints:

  • rootca-public - used to issue certificates for routes and k8s services for public endpoint httpd vhost configuration.
  • rootca-internal - used to issue certificates for all internal communication, like internal endpoint httpd vhost config, database, rabbitmq ..., except of the bellow special use case CAs.
  • rootca-ovn - used to issue certificates for ovn/ovs services specifically.
  • rootca-libvirt - used to issue certificates for libvirt/qemu services specifically.

By default CA certificates are valid for 10 years and service certificates for 5. This can be customized using the duration and renewBefore parameters for CA and certificates, e.g.:

spec:
  tls:
    ingress:
      ca:
        duration: 43800h
      cert:
        duration: 5000h
        renewBefore: 100h
      enabled: true
    podLevel:
      enabled: true
      internal:
        ca:
          duration: 43800h
        cert:
          duration: 5000h
          renewBefore: 200h
      libvirt:
        ca:
          duration: 83800h
        cert:
          duration: 5000h
      ovn:
        ca:
          duration: 83800h
        cert:
          duration: 5000h

Combined CA Bundle

The combined-ca-bundle secret, generated by the openstack-operator, aggregates:

  • System default CAs,
  • Operator-created CAs,
  • Third-party CAs provided with tls.caBundleSecretName. This bundle ensures a trusted CA pool is available for all services, supporting both default and custom certificate validation. The bundle can be used to be direct mounted into a deployment to the environment wide CA location. The lib-common common module tls package provides funtionality for this.

The secret also has a default label which can be used to query:

  labels:
    combined-ca-bundle: ""

Using the global caBundleSecretName parameter a secret can be referenced containing any additional CA certificates, which should be added to the combined CA bundle:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: myctlplane
spec:
  ...
  tls:
    caBundleSecretName: mycasecret

Multiple entries are allowed in one secret. Whenever openstack-operator reconciles it will check for expired or new CA certs to be added to the bundle. Not expired CAs will be kept in the secret if they are not yet expired, even if they got removed from the source it was originally taken from. This allows to rotate certs for a service using a new CA.

Note: The CA provided via the apiOverride.route is right now not added to the CA bundle.

Service Certificates for Public and Internal Endpoints

Certificates for both public and internal service endpoints are automatically managed by cert-manager, but can be customized for specific requirements.

Public endpoints:

Certificates are created for each public endpoint by default. However, custom certificates can be specified by creating a secret with the following keys:

  • tls.crt: The TLS certificate,
  • tls.key: The corresponding private key for the TLS certificate,
  • ca.crt: The CA certificate that has issued the TLS certificate,

The information is then used to be set the Route for this public endpoint or in future if a LoadBalancer service is used to terminated the endpoint at the pod level.

  • An alternative would be to directly use the RouteOverride to set the cert, key and CA cert for the endpoint.

Example using the route override:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: myctlplane
spec:
  ...
  keystone:
    apiOverride:
      route:
        spec:
          tls:
            termination: edge
            certificate: |-
              -----BEGIN CERTIFICATE-----
              -----END CERTIFICATE-----
            key: |-
              -----BEGIN RSA PRIVATE KEY----
              -----END RSA PRIVATE KEY-----
            caCertificate: |-
              -----BEGIN CERTIFICATE-----
              -----END CERTIFICATE-----
            insecureEdgeTerminationPolicy: Redirect

For public endpoints, custom certificates can be specified within the apiOverride section for each service. This is not part of the global TLS configuration but is specific to each service that requires a custom public endpoint certificate.

Example of specifying custom certificates for ingress traffic:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: custom-ingress-cert-example
spec:
  ...
  keystone:
    apiOverride:
      tls:
        secretName: custom-ingress-cert-secret
    template:
  ...

Internal endpoints

For internal TLS, certificates are similarly managed with options for customization. CertManager issues certificates based on internal or public issuers, and the resulting secrets can be integrated into service configurations. To specify a custom certificates for internal endpoints, it's only possible providing the custom issuer.

TLS common package

The lib-common common module tls package defines several structs to organize and apply TLS settings for different service types within the OpenStack deployment.

Generic Service

The GenericService represents the basic configuration for a service requiring TLS. It includes the SecretName field, which refers to a Kubernetes secret containing the TLS certificate (tls.crt),private key (tls.key) and the CA certificate (ca.crt) used to sign the service certificate.

Simple Service

The SimpleService extends the GenericService by including CA information through embedding the Ca struct. This allows for specifying a custom CA bundle secret caBundleSecretName.

Example of a single, not endpoint specific service:

  tls:
    caBundleSecretName: combined-ca-bundle
    secretName: cert-nova-novncproxy-cell1-public-svc

API Service

The API struct focuses on services that provide an API, which might have different TLS requirements for its public and internal endpoints. It includes:

  • APIService: Encapsulates TLS configurations for both public and internal API endpoints.
  • Public: A GenericService for the public (external) endpoint of the API.
  • Internal: A GenericService for the internal endpoint of the API.

Example of a public/internal API service:

  tls:
    api:
      # secret holding tls.crt and tls.key for the APIs internal k8s service
      internal:
        secretName: cert-internal-svc
      # secret holding tls.crt and tls.key for the APIs public k8s service
      public:
        secretName: cert-public-svc
    # secret holding the tls-ca-bundle.pem to be used as a deploymend env CA bundle
    caBundleSecretName: combined-ca-bundle

Mounting the Certificates and Handling Certificate Updates

As mentioned above - certificates, including TLS certificates (tls.crt), private keys (tls.key), and CA certificates (ca.crt), are stored in secrets. These secrets are mounted into the containers of the services that require them. Depending on how the service gets started (using kolla or not), the cert and key get either mounted to a directory and using kolla to put into its final place, or directly mounted to where the service expects it. The CA bundle is always mounted as the global CA bundle to the final destination (bypassing kolla).

For k8s service a virtual host gets configured using the service name with its corresponding certificate.

Like with other config secrets and config maps, a hash is used to identify if the certificate changed and the deployment pod has to be restarted. For this, the service operators index the named secret fields to be able to watch those if they change. If a change is detected which results in a hash change, the default action is to restart the deployment in order to use the new certificates. There might be services which will handle this differently, but currently there's no option to put new certificates in use without restarting the deployment.

If internal tls is enabled, using spec.tls.podlevel.enabled: true, routes get configured with spec.tls.termination: reencrypt and CA added to spec.tls.destinationCACertificate to be able to validate the certificate of the k8s service.

Deployments Using k8s Services

The configuration of the service is reflecting the k8s service name, not the individual pod name. With this all pods in a scaling resource will share the same service certificate.

For each of the k8s services the openstack-operator requests a certificate for the hostname <service>.<namespace>.svc and <service>.<namespace>.svc.cluster.local using the internal and the public CertManager issuer. For each cert request, CertManger will create a secret holding a tls.crt, tls.key and ca.crt. Those secrets, certificate and the combined CA bundle described above, can be passed into the service operators CRD using the tls section.

Additional Subject Alternative Names

It is possible, for a given service, to define additional Subject Alternative Names that can be added to the CertificateRequest triggered by the openstack-operator. The definition of additional Subject Names is realized through an Annotation added to either public or internal services by the service operator. For example, the OpenStack Image Service (Glance) requires additional Subject Names DNS entries to ensure that the tls communication works properly when requests are proxied through replicas of the same glanceAPI resolved by an headless service. For this reason, the glance-operator introduces an annotation like the following:

apiVersion: v1
kind: Service
metadata:
  annotations:
    additionalSubjectNames: '*.glance-default-external-api.openstack.svc,*.glance-default-external-api.openstack.svc.cluster.local'
    core.openstack.org/ingress_create: "true"
    endpoint: public
  labels:
    component: glance-api
    endpoint: public
    glanceAPI: glance-default-external
...
...

The example above shows that a comma separated list of additional Subject Names can be added as the value of the additionalSubjectNames key. Such key is common to all the operators, and it's defined in the tls package of lib-common/common module. When the annotation described above is present, the openstack-operator is able to process its content, and it produces a CertificateRequest where additional Hostnames are added (as dnsNames) to the certificate generation process. Here's an example of the resulting Certificate CR processed by the openstack-operator via the EnsureEndpointConfig function:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: glance-default-public-svc
  namespace: openstack
spec:
  dnsNames:
  - glance-default-public.openstack.svc
  - glance-default-public.openstack.svc.cluster.local
  - '*.glance-default-external-api.openstack.svc'
  - '*.glance-default-external-api.openstack.svc.cluster.local'
...
...

Deployments Not Using k8s Services

TBD when e.g. ovn got added

Testing Scenarios

The kuttl test scenarios aim to cover the functionality of TLS within a cluster environment. General guidelines for writing a kuttl test for an operator are available in this section.

In openstack-operator

The tests for general TLS functions are implemented in the openstack-operator. Currently, the following scenarios are implemented:

  • Custom Internal Issuer
    • Deploy with the default internal issuer, then switch to a custom internal issuer.
    • Deploy with the custom internal issuer, then switch to a default internal issuer.
  • Custom Ingress Issuer
    • Deploy with the default ingress issuer, then switch to a custom ingress issuer.
    • Deploy with the custom ingress issuer, then switch to a default ingress issuer.
  • Service Certificate Rotation
    • After deployment, certificate secrets are deleted, triggering service restarts. The test checks whether new secrets are mounted in service pods and if route configurations use these new secrets.
  • Custom CA cert added to a bundle
    • Test whether a custom CA certificate provided via the caBundleSecretName is added to the combined-ca-bundle.
  • Customize cert and CA cert duration parameters
    • New durations are set to 500h0m0s for the service certificates and 1000h0m0s for the CA certificates.

Note: To triggert certificate recreation, secrets should not be deleted, this method was used for testing purposes only. Learn more about triggering renewal in the cert-manager ctml documentation.

In Service Operators

Each service operator implements its own kuttl tests, which can vary based on the specific operator and the type of TLS service each uses. Generally, the TLS kuttl tests cover the successful creation of volumes and volume mounts that hold the combined-ca-bundle, public, and internal certificates, ensuring the correct setup if the HTTPS scheme is used, etc. To facilitate kuttl testing, hardcoded certificate secrets are used, meaning kuttl does not require cert-manager at test runtime.