The CNCC is a controller/operator which interfaces with the cloud API endpoints
to execute and perform the set up required on behalf of the user creating /
updating / deleting the CRs this controller manages. The user in the OpenShift
use-case are the network plugins (openshift-sdn / OVN-Kubernetes) which
interface with this component using its CRDs. Currently, this is only
cloud.network.openshift.io/cloudprivateipconfigs which adds / removes private
IP addresses to VM instances associated with Kubernetes nodes.
A CR for this looks like:
apiVersion: cloud.network.openshift.io/v1
kind: CloudPrivateIPConfig
metadata:
name: 192.168.126.11
spec:
node: nodeX
status:
node: nodeX
conditions:
- message: ""
reason: ""
status: "True|False|Unknown"
type: Assigned
The name of the CR represent the IP address which is to be assigned / removed.
Due to a limitation in the Kube API-server, CR names need to be RFC 1123
compliant
which means that IPv6 addresses can't just be written as-is. The format this
controller thus expects for IPv6 names are a fully expanded IP addresses with
colons replaced by dots, ex: fc00:f853:ccd:e793::54 ->
fc00.f853.0ccd.e793.0000.0000.0000.0054. Any controller interfacing with this
controller for IPv6 assignments must thus be capable of performing a
bi-directional conversion to and from that format.
Any assignment of an IP address should only be considered successful when
.spec.node == .status.node and .status.conditions[0].status == True, this
applies to creation / updates of CRs.
This controller utilizes a finalizer which it sets on any CR which is created. The reason for doing so is to prevent an instance from being removed from the API until this controller has reacted, i.e: making sure that even if a delete is performed while our controller is not running: nothing actually gets deleted and we don't lose out on the event.
The control loop for this controller performs atomic add/deletes. This is to say that for an update there will be two syncs performed and two updates occurring: one removal of the IP address from the current node, upon which the CR is updated, and then a second add to the new node, upon which the CR is updated again.
This controller requires credentials to be able to talk to the cloud API. The
format for these credentials depend on the cloud and mode of deployment. The
controller accepts an argument -secret-name which is to be placed in the
namespace this controller is running in. The controller figures out what that
namespace is using the environment variable CONTROLLER_NAMESPACE which can be
passed to it through the deployment stanza and using the downward-API. The
controller will then start watching for secret data change as to get notified of
credential rotation, upon which it exits and forces a restart of the pod. The
secret also needs to be mounted for this controller to read at start-up, this
location is by default: /etc/secret/cloudprovider but can be specified and
overridden using the argument -secret-override.
The credentials are expected to look like the following (for vanilla Kubernetes clusters).
tree /etc/secret/cloudprovider
└── service_account.json
or as a Kubernetes secret defined as:
apiVersion: v1
data:
service_account.json: ewogICJ0eXBlIjogInNlcnZpY2VfYWNjb3VudCIsCiAgInByb2nmxSOIJSdijdplY3RfaWQiOiAib3BlbnNoaWZ0LWdjZS1kZXZlbC1jaSIsCiAgInByaXZhdGVfa2V5X2lkIjogImZkMjYzMTMxMjU0M2U5ZWYxMDhkY2JlODUwZGI3N2E4MDc2YWE2ZTMiLAogICJwcml2YXRlX2tleSI6ICItLS0tLUJFR0lOIFBSSVZBVEUgS0VZLS0tLS1cbk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRRGFXRktDZzV0Vm9OVjZcbkNKSVRVWW1maENzenhoVHNmS1c5Q0lMdUZ2NHBuVTlPc1JoNG5wSmNDQmk1NkhqOWtBdHRiR3E3ME1XTGhudmtcbjFJR0p0R3RQQ0pXbE0ybW9VSHN2L2h5WXh2NjczYlhTa00vYm9kRmZBVWpIcGhjUTRHY1ZwdkdndE5TRXNhU1FcbmpGWXNad3orZVRZTWtRcHN5RXdMUk5YSnRER05YZllmRGNDL2ZsNEcvUExsdElNd0JuSUFLQTZJT3JXTmdiOW9cbmppYzJVc09mT3B4T1huOFo1OHFKcTczZkNoYVd0VHZYRTIrUXM0VUhDQXIxZHFSckNrdVVqRndBTkZPS0JZdHVcbmJMU2d4NUFwWkNkUGY5cGFwSlVSU2lwSkVKa2w5VnNGTDgwZG9zV21WWi9TSWdFbzVaTW1DYVdrUFNQN2ZTdXBcbnVuMWVOVTdIQWdNQkFBRUNnZ0VBVEFFa3I1UijdijdidjidijdGFWWU5IN0ZQaGJXSWJzdkJTRFpLdEkrajcxSUtLK1EvOTUwU0Fcbkg5ZGJ1bGtRZjRLK0FMRGd0UHNZVHozSEpadTF1Q3pYSWIrclcrRDIzYXNTVkZCQ1Bqbk50OVlQNUVxWXo1S01cbndVQnhhblc0cVFhTWJCcnZ1b3N5dHdIRzZIY3A1d3JqU3dIZTJWUUIzTzhhbG1OQ3FyMUtZejNSNFlXZEhpVHJcbkUxZ1VEN1dOVldQQmFrN3J6VGZnL2V6Mkl0c21Vb0wwWi9SWGg1dGM0YlFxTzE1SHRyeVd3Vmsrc2JGWS9ZY3Bcbm5hY0RSL2JFOGZXWjE1YXRuZGJwVFJ2TVhkaUNoSDQ3WTJMSlA4RDFTUGk1Ynh0VXhyTjRiUjFscUptWDZ5TGdcbm5qb2lUTjZnRjZFZW41d0dYV2tUeE1lZ2tleDd5UnUrVUtBdTl1U3FrUUtCZ1FEOVVNS3EzWUdTMFltNmlKQ0JcbmRnaXdGalFiK3R3ODNhcHhBOWNWUVBDMXlzdUoxdjVCTWw3WXVIRkw4TVlBbSswR1NNL0JmK3BsVndJNnFSMVRcbmJEdEdHL1JBUFY1VDlYM2hDdGIwczhpOVo5MHdTbkYxQVI5Y3Jvb1NvVTZTa08xUUxkUTg3R3VPNHlWL0R3ckNcbmZZZkxNbmdLamxHZWdTUzFKbDU3SS83a21RS0JnUURjcUxBUWs4bTVOYnJJY0NzNGxuU3VzWDlKU1Z6TUlwRjhcblpGNjlLd3ZYK2V6S0RBQlJheklHK0ltdVNCVnc0R1h3ZlFnMHNUWEl4SDhDaS9oTmYrV1pDU3hPeDE3eGpaWW5cbmNwWWM1d2pNaEhzd1Y0bkVyV3dNY2tLODFwM2x0NGQwdWs2N0lGYlFoN2pUMmczbHZuWHRseTBpWGkvZFdTSjdcbllSSlU4UVNLWHdLQmdRRGFIV1hoRmVWeDg3Wnh4UkVZUi9mbkZ0YzdtRjkya1M4bkxMVlArYURLQjVvR1QvYVJcblVMdldROHBhSnpGMmFNeEljdjFna2JIVUhIMHc0Vmo4OGQ1LzJhWVFna2JzYUI5QlhNSUY1Wi9kWnNkUHcybTVcbjQ0T0xuRVlMRUpYRkljRVZIc0QyekdNNG4yRXo3RkhKY2FreFQzMkpLVTRoK3ZVT2ZjRXdxcWZaVVFLQmdGL1dcbkpDSVBEaDRTNS8wR09yOXBHV1NHVXZKUm1xeE9sMEdmbGtZeTNBSUIyb1lta0R6TWdmM2xGR2ROaDRKdTg5ZHZcbmRwRHNKcC84TisyelBUVHJ4NXlnRDA1bjZTU2dpZ2E0RGRxZnZZS1dSNnJIV2w3QnM1djBSR2dnRHBRbkVmM3dcbnJTRTQrbnUzZHQ4TVpkelN6QVZWRTVWSnN5QkFCbW52enpaMU43T2hBb0dBYzZrWGhRZ3dYbFl4SHBFTEw2cEZcbjRBV0J5VExSbWFIdGNaa1J3ZmJpc0JxNjhOMnpIbUZwMlAyb3JHSW0wSUVhVFJOT2lHM2xQSU8zb09zL200MDBcbjNpSEVhU1pwN2dLWndRUCtzSDYrdyt5K1FoaUwwQ2JlV25ud1YzYkJKN2RTMUxvT1krb0luY2RyZHVVaTY5YUNcblZHM1RJMHZpWW1MU1VPdUFkVURVeHMwPVxuLS0tLS1FTkQgUFJJVkFURSBLRVktLS0tLVxuIiwKICAiY2xpZW50X2VtYWlsIjogImNpLWxuLTVoeGswMC1vcGVuc2hpZnQtYy12dHEyN0BvcGVuc2hpZnQtZ2NlLWRldmVsLWNpLmlhbS5nc2VydmljZWFjY291bnQuY29tIiwKICAiY2xpZW50X2lkIjogIjEwMDQyMDc0OTQyMzgyMjY0NjI3MiIsCiAgImF1dGhfdXJpIjogImh0dHBzOi8vYWNjb3VudHMuZ29vZ2xlLmNvbS9vL29hdXRoMi9hdXRoIiwKICAidG9rZW5fdXJpIjogImh0dHBzOi8vb2F1dGgyLmdvb2dsZWFwaXMuY29tL3Rva2VuIiwKICAiYXV0aF9wcm92aWRlcl94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL29hdXRoMi92MS9jZXJ0cyIsCiAgImNsaWVudF94NTA5X2NlcnRfdXJsIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL3JvYm90L3YxL21ldGFkYXRhL3g1MDkvY2ktbG4tNWh4azAwLW9wZW5zaGlmdC1jLXZ0cTI3JTQwb3BlbnNoaWZ0LWdjZS1kZXZlbC1jaS5pYW0uZ3NlcnZpY2VhY2NvdW50LmNvbSIKfQo=
kind: Secret
metadata:
name: cloud-credentials
namespace: openshift-cloud-network-config-controller
type: Opaque
tree /etc/secret/cloudprovider
├── aws_access_key_id
└── aws_secret_access_key
or as a Kubernetes secret defined as:
apiVersion: v1
data:
aws_access_key_id: QUtJQVdXT1NCDIAaiaNVhGREJRRVA0N04=
aws_secret_access_key: MG0rV05WMASOKSo2VCdmR1RG9mV1pIcDh4bWJDinsaSSAbXWEx1bHQzekhJU2Jud1UyMw==
kind: Secret
metadata:
name: cloud-credentials
namespace: openshift-cloud-network-config-controller
type: Opaque
If -platform-aws-ca-override=<file name> is set then the CNCC expects to find
a valid CA certificate chain at that location. This CA certificate chain will be
used when talking to the AWS API.
The Cluster Network Operator will create a ConfigMap named kube-cloud-config
inside the CNCC's namespace and will mount this ConfigMap at location /kube-cloud-config.
It will then set -platform-aws-ca-override=/kube-cloud-config/ca-bundle.pem.
Additionally, if parameter -config-name=<name of ConfigMap> is set then the CNCC
will start monitoring that ConfigMap for update or delete operations. If such
an event gets triggered, the process will gracefully shutdown. Kubernetes will
subsequently restart the container, thus spawning a process with updated configuration.
The default value for -config-name is kube-cloud-config.
tree /kube-cloud-config
└── ca-bundle.pem
Or as a Kubernetes ConfigMap defined as:
apiVersion: v1
data:
ca-bundle.pem: <data>
kind: ConfigMap
metadata:
name: kube-cloud-config
namespace: openshift-cloud-network-config-controller
tree /etc/secret/cloudprovider
├── azure_client_id
├── azure_client_secret
├── azure_region
├── azure_resource_prefix
├── azure_resourcegroup
├── azure_subscription_id
└── azure_tenant_id
or as a Kubernetes secret defined as:
apiVersion: v1
data:
azure_client_id: NGYwM2JjYWItN2I2My00SSoaNjE3LTk2NDEtMGUyZWViOWNjNWVi
azure_client_secret: aEs3TVUzMaiajiDD2tfXy0uVGNYaDU1LVpOcm5FMHMtUndpskD4ueFBvQg==
azure_region: ZWFzdHVz
azure_resource_prefix: Y2ktbG4tNnNqMzMydCs0xZDA5ZC1kZHE2Mg==
azure_resourcegroup: Y2ktbG4tNnNqMzMydC0xZDA5ZC1kZHE2Mi1yZw==
azure_subscription_id: ZDM4ZjFlMzgsokPSAAHUDH
azure_tenant_id: NjA0DSJAaN2M3ZTktYjJhZsokAOKADH
kind: Secret
metadata:
name: cloud-credentials
namespace: openshift-cloud-network-config-controller
type: Opaque
tree /etc/secret/cloudprovider
└── cloud.yaml
or as a Kubernetes secret defined as:
apiVersion: v1
data:
clouds.yaml: <base64 string>
kind: Secret
metadata:
name: cloud-credentials
namespace: cloud-network-config-controller
type: Opaque
The Cluster Network Operator will create a ConfigMap named kube-cloud-config
inside the CNCC's namespace and will mount this ConfigMap at location /kube-cloud-config.
The CNCC will look for a file at location /kube-cloud-config/ca-bundle.pem. If the
content of that file is != "", then the operator will assume that this is a valid CA chain
and will use this data when talking to the OpenStack API.
If parameter -config-name=<name of ConfigMap> is set to anything other than "",
then the CNCC will start monitoring that ConfigMap for update or delete operations. If
such an event gets triggered, the process will gracefully shutdown. Kubernetes will
subsequently restart the container, thus spawning a process with updated configuration.
The default value for -config-name is kube-cloud-config.
tree /kube-cloud-config
└── ca-bundle.pem
Or as a Kubernetes ConfigMap defined as:
apiVersion: v1
data:
ca-bundle.pem: <data>
kind: ConfigMap
metadata:
name: kube-cloud-config
namespace: openshift-cloud-network-config-controller
Assigning private IP addresses to instances on the cloud comes with some limitations.
An IP address can only be assigned to subnets which can host it. In the cloud subnets vary depending on the availability zone an instances is located in. This controller thus also retrieves the subnet information from the cloud API and annotates the node object with it, see below for what this annotation looks like.
Clouds limit the amount of private IP addresses which can be associated with instances. Some have a variable capacity per instance type and IP family (AWS) others have a global and fixed capacity defined (GCP/Azure). This controller does not validate that if capacity is superseded or not, it is up to the client controller to initialize its state and track how many assignments are still possible.
Any CloudPrivateIPConfig currently is only added to the instances' first NIC in the order specified by the cloud. On Azure the notion of a "primary" instance exists, and is hence used, even if that NIC might not be defined first. As to account for future work where IP addresses might be assigned to other NICs besides the first one, the annotation reports an array of interface/subnet/capacity. For now this array is always of length 1, with the first/"primary" interface.
All of these attributes are placed on the node object as an annotation, which looks like:
cloud.network.openshift.io/egress-ipconfig: [{"interface": "$IFNAME/$IFID", "ifaddr": {"ipv4": "$IPv4_ADDRESS/$IPv4_SUBNET_MASK", "ipv6": "$IPv6_ADDRESS/$IPv6_SUBNET_MASK"}, "capacity": {"ip": "$IPv4_AND_IPv6_CAPACITY"}}]
if the capacity is IP family agnostic. If that is not the case, the annotation will look like:
cloud.network.openshift.io/egress-ipconfig: [{"interface": "$IFNAME/$IFID", "ifaddr": {"ipv4": "$IPv4_ADDRESS/$IPv4_SUBNET_MASK", "ipv6": "$IPv6_ADDRESS/$IPv6_SUBNET_MASK"}, "capacity": {"ipv4": "$IPv4_CAPACITY", "ipv6": "$IPv6_CAPACITY"}}]
When you want to debug this component, use one of the following methods.
If you want to run the operator directly on your computer, you can
run the ./hack/run-locally.sh, but make sure you have a KUBECONFIG
initialized in your environment variables. If running against an OpenShift
cluster: the credentials will be copied locally to your computer. That being
said, you probably won't be able to talk to the cloud API endpoints from your
local development environment, so you might need to comment out code which is
non-essential to your testing.
If you have issues talking directly to the cloud API endpoints or if you want to test how the operator runs inside the actual container, use this method.
Set up your own publicly accessible repository. For example, use a quay repository:
export CNCC_REPOSITORY="quay.io/akaris/cloud-network-config-controller"
If you prefer using docker instead of podman, run:
export CONTAINER_ENGINE="docker"
Run:
hack/run_in_cloud.sh
This will then build the container from the Dockerfile, tag the image as ${CNCC_REPOSITORY}:${UUID}
where ${UUID} is a unique ID, push the image to the registry and patch the clusterversion operator
and the network-operator so that this new image will be used.
In order to rollback, run oc edit clusterversion version and remove the configuration in
/spec/overrides. The clusterversion operator should then bring the cluster back to its
original state.
Note: If the cloud-network-config-controller pod runs into authentication problems after using this method, it may be necessary to delete the pod manually to "kick" it.