ETCD encryption configuration

[oliverbaehler]

Feature Request

Hi, I was playing around with etcd encryption and Talos, or rather trying to embed a kmsv2 provider to encrypt all secrets and configmaps at REST. However i have noticed, that currently it's only possible to pass one key to the template here: https://github.com/siderolabs/talos/blob/main/internal/app/machined/pkg/controllers/k8s/templates/kube-system-encryption-config-template.yaml

If I understood correctly, it's also not possible to overwrite that file. This causes some problems with ETCD encryptions:

• no other resources than secrets can be encrypted
• key rotation is not possible
• other kms than static keys on the controlplanes are not possible.

I was wondering if we could improve this by either:

• Being able to supply a custom-path to a user-defined encryption-configuration
• Having the Encryption-Config inline as with the AuditPolicy

Wondering what your thoughts are about this. I can probably contribute the feature, if we agree on an implementation.

[smira]

I think customizing encryption configuration would be a great additional feature!

[frezbo]

ETCD specific machine config document would be great

[oliverbaehler]

@frezbo Just for clarification, i would propose something like this:

cluster:
    apiServer:
        encryptionProviderConfig:
            apiVersion: apiserver.config.k8s.io/v1
            kind: EncryptionConfiguration
            resources:
              - resources:
                  - secrets
                  - configmaps
                providers:
                  - aescbc:
                      keys:
                        - name: key1
                          secret: "${ETCD_ENCRYPTION_KEY_1}"
                        - name: key2
                          secret: "${ETCD_ENCRYPTION_KEY_2}"

I would place it under cluster.apiServer.encryptionProviderConfig because it should always be the same for all the cps. From the kubernetes-docs:

For cluster configurations with two or more control plane nodes, the encryption configuration should be identical across each control plane node. If there is a difference in the encryption provider configuration between control plane nodes, this difference may mean that the kube-apiserver can't decrypt data.

If that's alright I will try to get back with a PR soon. 🍀

[frezbo]

@frezbo Just for clarification, i would propose something like this:

cluster:
apiServer:
encryptionProviderConfig:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
- configmaps
providers:
- aescbc:
keys:
- name: key1
secret: "${ETCD_ENCRYPTION_KEY_1}"
- name: key2
secret: "${ETCD_ENCRYPTION_KEY_2}"

I would place it under cluster.apiServer.encryptionProviderConfig because it should always be the same for all the cps. From the kubernetes-docs:

For cluster configurations with two or more control plane nodes, the encryption configuration should be identical across each control plane node. If there is a difference in the encryption provider configuration between control plane nodes, this difference may mean that the kube-apiserver can't decrypt data.

If that's alright I will try to get back with a PR soon. 🍀

@smira should we do a new config document or use existing v1alpha1

[smira]

I'd say we should better do a new config document.

[oliverbaehler]

@frezbo could you help me getting started. How and where do i initialize the new API ?

[smira]

We probably need to create a new folder https://github.com/siderolabs/talos/tree/main/pkg/machinery/config/types here, e.g. kubernetes (or k8s to match resource folder).

The first config there will be this one, after that it might need more plumbing to get it to work properly, we could help with that.

[oliverbaehler]

#10905 would probably require the same approach then. I will look into it

[oliverbaehler]

@smira @frezbo circling back to this, do you still thing it makes sense to add this to the API of Talos? Eg. With the Authentication Policy its currently also just possible to map the file to the kubernetes machine and then pass the path via extraArgs to the kubernetes control-plane. Not a 100% sure if you really want to port Kubernetes config types to the talos api as well..

What I am trying to say is that we also could just allow to define a custom path to the encryption file and this issue would be resolved - Maybe changing to case MergeOverwrite instead of MergeDenied?

talos/internal/app/machined/pkg/controllers/k8s/control_plane_static_pod.go

Line 425 in 5117483

"encryption-provider-config": argsbuilder.MergeDenied,

[smira]

I would prefer to keep this Talos-managed, as it's managed by Talos, but we need to extend the configuration to support more flexibility.

[clementnuss]

I gave a shot at implementing a new k8s config type in here: #11641.

@smira is that what you had in mind?

[smira]

The more I think about it, I think it does make sense, I just want to take some time to consider all use-cases. We will circle back to this in 1-2 weeks, after 1.11.0 is out.