Skip to content

Latest commit

 

History

History
379 lines (304 loc) · 15.1 KB

20200220-cluster-resource-set.md

File metadata and controls

379 lines (304 loc) · 15.1 KB
Raw
title authors reviewers creation-date last-updated status
ClusterResourceSet
@sedefsavas
@vincepri
@detiber
@ncdc
@fabriziopandini
2020-02-20
2020-08-05
experimental

ClusterResourceSet

Table of Contents

Glossary

Refer to the Cluster API Book Glossary.

Summary

Provide a mechanism for applying resources in a cluster once it is created.

Motivation

Clusters created by Cluster API are minimally functional. For instance,they do not have a container networking interface (CNI), which is required for pod-to-pod networking, or any StorageClasses, which are required for dynamic persistent volume provisioning. Users today must manually add these components to every cluster they create.

Having a mechanism to apply an initial set of default resources after clusters are created makes clusters created with Cluster API functional and ready for workloads from the beginning, without requiring additional user intervention.

To achieve this, ClusterResourceSet CRD is introduced that will be responsible for applying a set resources defined by users to the matching clusters (label selectors will be used to select clusters that the ClusterResourceSet resources will applied to.)

Goals

  • Provide a means to specify a set of resources to apply automatically to newly-created and existing Clusters. Resources will be reapplied when their definition changes.
  • Support additions to the resource list by applying the new added resources to both new and existing matching clusters.
  • Provide a way to see which ClusterResourceSets are applied to a particular cluster using a new CRD, ClusterResourceSetBinding.
  • Support both json and yaml resources.

Non-Goals/Future Work

  • Replace or compete with the Cluster Addons subproject.
  • Support deletion of resources from clusters. Deleting a resource from a ClusterResourceSet or deleting a ClusterResourceSet does not result in deletion of those resources from clusters.
  • Lifecycle management of the installed resources (such as CNI).

Proposal

User Stories

Story 1

As someone creating multiple clusters, I want some/all my clusters to have a CNI provider of my choosing installed automatically, so I don’t have to manually repeat the installation for each new cluster.

Story 2

As someone creating multiple clusters, I want some/all my clusters to have a StorageClass installed automatically, so I don't have to manually repeat the installation for each new cluster.

Story 3

As someone creating multiple clusters and using ClusterResourceSet to install some resources, I want to see which resources are applied to my clusters, when they are applied, and if applied successfully.

Implementation Details/Notes/Constraints

Data model changes to existing API types

None. We are planning to implement this feature without modifying any of the existing structure to minimize the footprint of ClusterResourceSet Controller. This enhancement will follow Kubernetes’s feature-gate structure and will be under the experimental package with its APIs, and enabled/disabled with a feature gate.

ClusterResourceSet Object Definition

This is the CRD that has a set of components (resources) to be applied to clusters that match the label selector in it. The label selector cannot be empty.

The resources field is a list of Secrets/ConfigMaps which should be in the same namespace with ClusterResourceSet. The clusterSelector field is a Kubernetes label selector that matches against labels on clusters (only the clusters in the same namespace with the ClusterResourceSet resource). ClusterResourceSet is namespace-scoped, all resources and clusters needs to be in the same namespace as the ClusterResourceSet. Sample ClusterResourceSet YAML

---
apiVersion: addons.cluster.x-k8s.io/v1alpha3
kind: ClusterResourceSet
metadata:
 name: crs1
 namespace: default
spec:
 mode: "ApplyOnce"
 clusterSelector:
   matchLabels:
     cni: calico
 resources:
   - name: db-secret
     kind: Secret
   - name: calico-addon
     kind: ConfigMap

The supported modes will be:

  • ApplyOnce. This will be the default mode if no mode is provided. Resources are only applied once.
  • Reconcile. Resources are reapplied when the content of the ConfigMap or Secret that defines them changes.

If ClusterResourceSet resources will be managed by an operator after they are applied by ClusterResourceSet controller, "ApplyOnce" mode must be used so that reconciliation on those resources can be delegated to the operator.

Each item in the resources specifies a kind (must be either ConfigMap or Secret) and a name. Each referenced ConfigMap/Secret contains yaml/json content as value. ClusterResourceSet object will be added as owner to its resources.

Secrets as Resources

Both Secrets and ConfigMaps data fields can be a list of key-value pairs. Any key is acceptable, and as value, there can be multiple objects in yaml or json format.

For preventing all secrets to be reached by all clusters in a namespace, only secrets with type addons.cluster.x-k8s.io/resource-set can be accessed by ClusterResourceSet controller. Secrets are preferred if the data includes sensitive information.

An easy way to create resource Secrets is to have a yaml or json file with the components.

E.g., this is db.yaml that has multiple objects:

kind: Secret
apiVersion: v1
metadata:
 name: mysql-access
 namespace: system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: db-admin
 namespace: system

We can create a secret that has these components in its data field to be used in ClusterResourceSet:

#kubectl create secret generic db-secret --from-file=db.yaml --type=addons.cluster.x-k8s.io/resource-set

apiVersion: v1
kind: Secret
metadata:
  name: db-secret
type: addons.cluster.x-k8s.io/resource-set
stringData:
  db.yaml: |-
    kind: Secret
    apiVersion: v1
    metadata:
     name: mysql-access
     namespace: system
    ---
    kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
     name: db-admin
     namespace: system

ConfigMaps as Resources

Similar to Secrets, ConfigMaps can be created using a yaml/json file: kubectl create configmap calico-addon --from-file=calico1.yaml,calico2.yaml Multiple keys in the data field and then multiple objects in each value are supported.

apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-addon
data:
  calico1.yaml: |-
     kind: Secret
     apiVersion: v1
     metadata:
      name: calico-secret1
      namespace: mysecrets
      ---
     kind: Secret
     apiVersion: v1
     metadata:
      name: calico-secret2
      namespace: mysecrets
  calico2.yaml: |-
     kind: ConfigMap
     apiVersion: v1
     metadata:
      name: calico-configmap
      namespace: myconfigmaps

ClusterResourceSetBinding Object Definition

The resources in ClusterResourceSet will be applied to matching clusters. There is many-to-many mapping between Clusters and ClusterResourceSets: Multiple ClusterResourceSets can match with a cluster; and multiple clusters can match with a single ClusterResourceSet. To keep information on which resources applied to which clusters, a new CRD is used, ClusterResourceSetBinding will be created in the management cluster. There will be one ClusterResourceSetBinding per workload cluster. ClusterResourceBinding's name will be same with the Cluster name. Both Cluster and the matching ClusterResourceSets will be added as owners to the ClusterResourceBinding.

Example ClusterResourceBinding object:

apiVersion: v1
kind: ClusterResourceBinding
metadata:
  name: <cluster-name>
  namespace: <cluster-namespace>
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1alpha3
    kind: Cluster
    name: <cluster-name>
    uid: e3a503a8-9be1-4264-8fa2-d536532687f9
  - apiVersion: addons.cluster.x-k8s.io/v1alpha3
    blockOwnerDeletion: true
    controller: true
    kind: ClusterResourceSet
    name: crs1
    uid: 62c77639-92d8-46d2-ba21-a880f62f7719
spec:
  bindings:
  - clusterResourceSetName: crs1
    resources:
    - applied: true
      hash: sha256:a3473f4e92ee5a2277ff37d5c559666d61d24332a497b554e65ae18e82727245
      kind: Secret
      lastAppliedTime: "2020-07-02T05:47:38Z"
      name: db-secret
    - applied: true
      hash: sha256:c1d0dc7e51bb05945a2f99e6745dc4b1043f8a03f37ad21391fe92353a02066e
      kind: ConfigMap
      lastAppliedTime: "2020-07-02T05:47:39Z"
      name: calico-addon

When a cluster is deleted, the associated ClusterResourceBinding will also be cleaned up. When a ClusterResourceSet is deleted, it will be removed from the bindings list of all ClusterResourceBindings that it is listed.

ClusterResourceSet will use ClusterResourceSetBinding to decide to apply a new resource or retry to apply an old one. In ApplyOnce mode, if a resource/applied is true, that resource will never be reapplied. If applying a resource is failed, ClusterResourceSet controller will reconcile it and use the controller-runtime's exponential back-off to retry applying failed resources. In case of new resource addition to a ClusterResourceSet, that ClusterResourceSet will be reconciled immediately and the new resource will be applied to all matching clusters because the new resource does not exist in any ClusterResourceBinding lists.

ApplyOnce mode

When the same resource exist in multiple ClusterResourceSets, only the first one will be applied but the resource will appear as applied in all ClusterResourceSets in the ClusterResourceSetsBinding/bindings. Similarly, if a resource is manually created in the workload cluster, when a ClusterResourceSet is applied with that resource, it will not update the existing resource to avoid any overwrites but in ClusterResourceSetBinding, that resource will show as applied.

Reconcile mode

Detecting changes ClusterResourceBindings will contain consistent hash for the resource/s definitions. We will use this to detect changes, comparing the hash of the current resource/s definition with the one stored in the ClusterResourceBindings.

Note that this hash will change when any of the resources is updated, a resource is added or a resource is removed. This means that all resources in the same ConfigMap or Secret, and not only the one that changed, will be reapplied in any of these 3 cases. It also means that resources removed from ConfigMap or Secret won't be deleted from the target clusters.

In the next before-after example we can see that only one resource has changed (ConfigMap calico-configmap). However, all the 3 resources (calico-secret1, calico-secret2 and calico-configmap) will be reapplied.

Before:

apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-addon
data:
  calico1.yaml: |-
     kind: Secret
     apiVersion: v1
     metadata:
      name: calico-secret1
      namespace: mysecrets
      ---
     kind: Secret
     apiVersion: v1
     metadata:
      name: calico-secret2
      namespace: mysecrets
  calico2.yaml: |-
     kind: ConfigMap
     apiVersion: v1
     metadata:
      name: calico-configmap
      namespace: myconfigmaps
     data:
       key: "original value"

After:

apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-addon
data:
  calico1.yaml: |-
     kind: Secret
     apiVersion: v1
     metadata:
      name: calico-secret1
      namespace: mysecrets
      ---
     kind: Secret
     apiVersion: v1
     metadata:
      name: calico-secret2
      namespace: mysecrets
  calico2.yaml: |-
     kind: ConfigMap
     apiVersion: v1
     metadata:
      name: calico-configmap
      namespace: myconfigmaps
     data:
       key: "value that changed"

Risks and Mitigations

Drift

The proposed solution only deals with changes in the resources' definitions and not with changes in the real objects in the workload clusters. If those objects are modified or deleted in the workload clusters, the ClusterResourceSet's controller won't do anything and they will remain unchanged until their definition in the management cluster is updated.

This could potentially be mitigated by:

  • Implementing a "periodic" reconciliation mode where resources are reapplied with a certain frequency even if their hash hasn't changed.

Deletion

Resource deletion is not supported. If a resource is removed from a ConfigMap or Secret, the hash for the CRS will change so the resources that haven't been removed will be reapplied. However, the removed resourced won't be deleted.

Alternatives

The Alternatives section is used to highlight and record other possible approaches to delivering the value proposed by a proposal.

Upgrade Strategy

This is an experimental feature supported by a new CRD and controller so there is no need to handle upgrades for existing clusters.

Additional Details

Test Plan [optional]

Extensive unit testing for all the cases supported when applying ClusterResourceSet resources. e2e testing as part of the cluster-api e2e test suite.

Graduation Criteria [optional]

This proposal will follow all maturity stages (alpha, beta, GA) and then may be merged with cluster-api apis and controllers.

Implementation History