diff --git a/proposals/2027_opt_out_of_self_signed_certificate_authorities/README.md b/proposals/2027_opt_out_of_self_signed_certificate_authorities/README.md new file mode 100644 index 000000000..585c6b7de --- /dev/null +++ b/proposals/2027_opt_out_of_self_signed_certificate_authorities/README.md @@ -0,0 +1,261 @@ +--- +title: "Opt-Out of self-signed CAs (and their certs)" +authors: [ "@joshuatcasey" ] +status: "proposed" +sponsor: [ "@cfryanr", "@ashish-amarnath" ] +approval_date: "TBD" +--- + +*Disclaimer*: Proposals are point-in-time designs and decisions. +Once approved and implemented, they become historical documents. +If you are reading an old proposal, please be aware that the +features described herein might have continued to evolve since. + +# Opt-Out of self-signed CAs (and their certs) + +## Problem Statement + +Pinniped's Concierge, Supervisor, and Local-User-Authenticator today require the use of self-signed certificate +authorities (CAs). These self-signed CAs are used for a variety of purposes, but generally are used to sign leaf +certificates used to serve TLS or sign leaf certificates for client authentication (mTLS). + +Using self-signed certificates can mean that custom CA bundles need to be installed on client machines so that clients +can perform TLS verification with a Pinniped endpoint. As of Pinniped `v0.32.0`, any endpoints that are meant to be +visited by a client machine can use external certificates. In particular, each Supervisor `FederationDomain` serves +OIDC discovery and web endpoints that are meant to be visited by client machines, and each Concierge `ImpersonationProxy` +is an endpoint meant to be visited by client machines. This proposal will not change configuration for those endpoints. + +However, Pinniped does expose various endpoints for Kubernetes-internal use, such as to serve a `/healthz` endpoint +or an endpoint that backs an `APIService` (only the Kubernetes API Service will call an `APIService`), as well as +create certificates for client authentication (when the impersonation proxy is enabled). Today, Pinniped will generate +its own CA certificate and any leaf certificates that it needs. + +Some customers are unable to use self-signed CAs or certificates due to regulatory factors or some other security +specification that governs their environments. Other customers may want to have visibility into what CAs or leaf +certificates are used in their cluster, and relying on Pinniped to generate its own CAs and leaf certificates obscures +that visibility. Still other customers may want to have complete control over when certificates are rotated, even if +those certificates do not need to be trusted by client machines. + +## Terminology / Concepts + +* Self-signed certificate: A certificate not signed by a publicly-trusted CA authority. This is usually a CA certificate, +but could be a leaf certificate. +* Certificate Authority (CA): A certificate (with `isCA: true`) used to issue intermediate certificates or leaf +certificates, such as certificates for serving TLS or certificates for client authentication (mTLS). +* Generated certificate: A certificate generated by Pinniped, from its own CAs. By definition these generated certificates +are not themselves self-signed, since they are always signed by a CA (even if that CA is self-signed). +* External certificate: A certificate provided by something outside of Pinniped, such as `cert-manager`. Usually this +is a leaf certificate (to serve TLS, for example), but could be a CA certificate. +* Secret type `kubernetes.io/tls`: Secrets that must have keys `tls.crt` and `tls.key` that can contain either a CA, +intermediate, or leaf certificate. See https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets. +* `cert-manager`: A [tool](https://cert-manager.io/) to manage certificates used within a cluster. Generally outputs +secrets of type `kubernetes.io/tls`, and may populate an additional data field `ca.crt` with a higher-level certificate +if available. +* CA Bundle: A collection of certificates used to verify TLS or some other certificate signature. Usually should only +contain CA or intermediate certificates (not leaf certificates that should be rotated frequently), and should never +contain private keys. + +## Proposal + +Pinniped should accept valid, properly-formatted CAs and certificates in secrets of type `kubernetes.io/tls` in those +places where it currently generates its own CAs or certificates. Pinniped will perform validations to ensure that the +CAs and certificates are valid and unexpired, but will not perform lifecycle management (such as rotation) on those +CAs and certificates. Pinniped will also verify that the certificates it is presented are useable for purpose (such as +a CA certificate vs a leaf certificate) and have the appropriate key usage (such as `server signing`). + +Here is a list of all known locations where Pinniped will generate its own CA or certificate: + +### Supervisor +* Bootstrap CA certificate and leaf certificate for serving TLS. At the moment this appears to be held only in memory, +but Supervisor ignore this CA and leaf certificate if a secret (default name `pinniped-supervisor-default-tls-certificate`, +type `kubernetes.io/tls`) exists. We propose that Supervisor no longer generates its own CA and leaf certificate if the +secret is found. TODO: verify the claims above, also see https://pinniped.dev/docs/howto/supervisor/configure-supervisor/#configuring-tls-for-the-supervisor-oidc-endpoints + +_Note that this could require significant effort since it changes an extremely fundamental aspect of how the Supervisor works. +In implementation, we may decide that this is not worth it, since customers can provide a secret for the default TLS serving cert._ + +### Concierge + +_Note that the solution below is taken from the discussion on [this thread](https://github.com/vmware-tanzu/pinniped/issues/1238#issuecomment-2261503118)_ + +Concierge holds two CAs and one leaf certificate in the following secrets: +* `pinniped-concierge-api-tls-serving-certificate` +* `pinniped-concierge-impersonation-proxy-signer-ca-certificate` + +Here are the proposed changes for these secrets to be externally provided as secrets of type `kubernetes.io/tls` + +#### `pinniped-concierge-api-tls-serving-certificate` + +This secret (of type `Opaque`) is used to serve TLS for all Concierge endpoints. In practice, these endpoints are: + +* `/healthz` +* The backing endpoint for the `login` `APIService` +* The backing endpoint for the `identity` `APIService` + +This secret uses keys `caCertificate` and `caCertificatePrivateKey` (to hold a CA and its private key), `tlsCertificateChain`, and `tlsPrivateKey` (to hold a leaf certificate signed by the CA, used to serve TLS). + +This secret is given to five controllers: + +* `CertsManager`, responsible for creating the secret (with all keys and values) if it does not exist +* `CertsExpirer`, responsible for rotating the value in key `tlsCertificateChain` in the secret if it exceeds a hard-coded TTL +* `APIServiceUpdater` x2, once to update `spec.CABundle` for the `login` `APIService`, and once to update`spec.CABundle` for the `identity` `APIService`, using the value from key `caCertificate` +* `CertsObserver`, responsible for reading into memory the values from keys `tlsCertificateChain` and `tlsPrivateKey` so that they can be used to serve TLS + +##### Changes required to support `kubernetes.io/tls` + +`cert-manager` will give us something like the following `Secret`. + +```yaml +apiVersion: v1 +kind: Secret +type: kubernetes.io/tls +data: + ca.crt: LS0t... # this is optional + tls.crt: LS0t... + tls.key: LS0t... +``` + +The `APIServiceUpdater` controller should look for a certificate in the following order: `ca.crt`, `tls.crt`, and `caCertificate`. +To ensure no downtime for Concierge's aggregate API (`APIService`) services, we will document a preference for the `ca.crt` field, +so that the `APIService` is given a CA bundle that is resilient to certificate rotations. + +The `CertsObserver` controller should look for keys `tls.crt` and `tls.key` first, and then fall back to using `tlsCertificateChain` and `tlsPrivateKey`. + +The `CertsManager` controller will create this secret if it does not exist. If the secret exists, the `CertsManager` controller will take no action. (No change needed). + +The `CertsExpirer` controller should continue to look for keys `caCertificate` and `caCertificatePrivateKey` and if those do not exist, should continue to provide a log message and requeue itself (by returning an error). +We should confirm that this does not "thrash" the controller, but behaviorally should work. (No change needed). + +#### `pinniped-concierge-impersonation-proxy-signer-ca-certificate` + +This secret (of type `Opaque`) is used to issue mTLS certificates to clients that have made a `TokenCredentialRequest` when the impersonation proxy is used. +This secret only uses keys `caCertificate` and `caCertificatePrivateKey`. + +This secret is given to three controllers: + +* `ImpersonatorConfig`, responsible for reading the values from keys `caCertificate` and `caCertificatePrivateKey` into memory so that they can be used to issue client certificates for mTLS. +* `CertsManager`, responsible for creating the secret (with all keys and values) if it does not exist. +* `CertsExpirer`, responsible for rotating the CA in the secret if it exceeds a hard-coded TTL + +##### Changes required to support `kubernetes.io/tls` + +`cert-manager` will give us something like the following `Secret`. Note that Pinniped does appear to verify whether `tls.crt` has `isCA: true` set, so be sure that is configured! + +```yaml +apiVersion: v1 +kind: Secret +type: kubernetes.io/tls +data: + ca.crt: LS0t... # this is optional + tls.crt: LS0t... + tls.key: LS0t... +``` + +I think the only change here is that the `ImpersonatorConfig` controller should first check for existence of keys `tls.crt` and `tls.key` (instead of `caCertificate` and `caCertificatePrivateKey`) and then fall back to using `caCertificate` and `caCertificatePrivateKey`. + +The `CertsManager` controller will create this secret if it does not exist. If the secret exists, the `CertsManager` controller will take no action. (No change needed). + +The `CertsExpirer` controller should continue to look for keys `caCertificate` and `caCertificatePrivateKey` and if those do not exist, should continue to provide a log message and requeue itself (by returning an error). +We should confirm that this does not "thrash" the controller, but behaviorally should work. (No change needed). + +### Local-User-Authenticator +_Note that local-user-authenticator is not meant for production use. It is included in this proposal simply because +its code overlaps with Supervisor and Concierge, and excluding it would require significant effort._ + +* CA certificate and leaf certificate for serving TLS, currently held in secret `local-user-authenticator-tls-serving-certificate` +in its installed namespace, with type `Opaque`. This secret holds both a CA and a TLS serving cert (and their private keys). +This secret would be treated in the same way as the Concierge secret `pinniped-concierge-api-tls-serving-certificate`. + +### Goals and Non-goals + +Goal: +* Pinniped should provide enough configuration options for Pinniped administrators to completely avoid using self-signed CAs +* Pinniped should generally expect external certificates to be leaf certificates (to serve TLS, for example). Pinniped +will need a CA certificate when the impersonation proxy is enabled, in order to issue client certificates for mTLS. +* Pinniped should rely on Kubernetes standards such as secrets with type `kubernetes.io/tls`, instead of coupling itself +to any specific tool such as `cert-manager`. However, Pinniped will defer to a `ca.crt` field for CA bundles if that +field is available. + +Non-goals: +* It is not a goal of this proposal to remove Pinniped's self-signed CAs (and require the user to configure all CAs and +certificates). Pinniped should continue to generate any necessary CAs and certificates that are not externally provided. +* It is not a goal of this proposal to change how the Supervisor signs the ID tokens that it issues. Those tokens are +signed by ECDSA private keys specific to each `FederationDomain`. Clients that need to verify tokens issued by the +Pinniped Supervisor can trivially obtain the public keys through OIDC discovery, not through PKI distribution. +* It is not a goal of this proposal to change configuration for `FederationDomain` or `ImpersonationProxy` resources +that already accept external certificates as of `v0.32.0`. + +#### API Changes + +None. The `FederationDomain` and `ImpersonationProxy` custom resources will not be modified as a result of this proposal. + +#### Configuration Changes + +We may need to introduce a configuration variable for Concierge to indicate that certificates will be managed externally. +This is to prevent any potential race condition at startup if Concierge boots up and detects that it needs to create its +own CAs and certificates before external tooling has provided them. This also implies that Pinniped would not need to +manage the lifecycle of the listed secrets (such as creation and rotation of the certificates). Effectively, this would +disable the `CertsManager` and `CertsExpirer` controllers for secrets `pinniped-concierge-api-tls-serving-certificate` +and `pinniped-concierge-impersonation-proxy-signer-ca-certificate`. + +This new configuration option would not apply to the local user authenticator, which is not meant for production use. +It would also not apply to the Supervisor, which does manage the `default-tls-certificate` secret (although it does +manage its own in-memory bootstrap certificate). TBD: Should we support this configuration, only so that Supervisor does +not generate its own in-memory bootstrap certificate? + +#### Upgrades + +The upgrade process should be seamless for users that will continue to use Pinniped's self-signed CAs. +Some downtime may be required for users that choose to opt in to externally-provided certificates, since old versions +of Pinniped do not anticipate external management of the secrets and may attempt to manage their lifecycle. + +#### Tests + +Unit and integration tests will be written as necessary to achieve 100% test coverage. +Integration tests will inspect the certificates presented by Pinniped to ensure that Pinniped is using generated or +externally provided certificates as configured. + +#### New Dependencies + +No new dependencies within Pinniped. May require using external tools or tooling to manage certificates. + +#### Performance Considerations + +None. + +#### Observability Considerations + +None. + +#### Security Considerations + +None. + +#### Usability Considerations + +None, these changes are not user-facing. + +#### Documentation Considerations + +Pinniped configuration options are only lightly documented outside their own comments. +A blog post to accompany the release is probably the best external documentation for this feature. + +### Other Approaches Considered + +Leave as-is. + +## Open Questions + +TBD + +## Answered Questions + +TBD + +## Implementation Plan + +TBD + +## Implementation PRs + +* TBD diff --git a/proposals/2027_opt_out_of_self_signed_certificate_authorities/fail_test.go b/proposals/2027_opt_out_of_self_signed_certificate_authorities/fail_test.go new file mode 100644 index 000000000..843055fc0 --- /dev/null +++ b/proposals/2027_opt_out_of_self_signed_certificate_authorities/fail_test.go @@ -0,0 +1,16 @@ +// Copyright 2024 the Pinniped contributors. All Rights Reserved. +// SPDX-License-Identifier: Apache-2.0 + +package fail_test + +// TODO: remove this file before merging. + +import ( + "testing" + + "github.com/stretchr/testify/require" +) + +func Test(t *testing.T) { + require.Fail(t, "fail this test so that unit tests fail and integration tests do not run in CI") +}