The User-Provisioned Infrastructure (UPI) process installs OpenShift in stages, providing opportunities for modifications or integrating with existing infrastructure.
It contrasts with the fully-automated Installer-Provisioned Infrastructure (IPI) which creates everything in one go.
With UPI, creating the cloud (OpenStack) resources (e.g. Nova servers, Neutron ports, security groups) is the responsibility of the person deploying OpenShift.
The installer is still used to generate the ignition files and monitor the installation process.
This provides a greater flexibility at the cost of a more explicit and interactive process.
Below is a step-by-step guide to a UPI installation that mimics an automated IPI installation; prerequisites and steps described below should be adapted to the constraints of the target infrastructure.
- Prerequisites
- Install Ansible
- OpenShift Configuration Directory
- Red Hat Enterprise Linux CoreOS (RHCOS)
- API and Ingress Floating IP Addresses
- Install Config
- Edit Manifests
- Ignition Config
- Network Topology
- Bootstrap
- Control Plane
- Compute Nodes
- Destroy the OpenShift Cluster
The file inventory.yaml contains the variables most likely to need customisation.
NOTE: some of the default pods (e.g. the openshift-router) require at least two nodes so that is the effective minimum.
The requirements for UPI are broadly similar to the ones for OpenStack IPI:
- OpenStack account with
clouds.yaml- input in the
openshift-installwizard
- input in the
- Nova flavors
- inventory:
os_flavor_masterandos_flavor_worker
- inventory:
- An external subnet you want to use for floating IP addresses
- inventory:
os_external_network
- inventory:
- The
openshift-installbinary - A subnet range for the Nova servers / OpenShift Nodes, that does not conflict with your existing network
- inventory:
os_subnet_range
- inventory:
- A cluster name you will want to use
- input in the
openshift-installwizard
- input in the
- A base domain
- input in the
openshift-installwizard
- input in the
- OpenShift Pull Secret
- input in the
openshift-installwizard
- input in the
- A DNS zone you can configure
- it must be the resolver for the base domain, for the installer and for the end-user machines
- it will host two records: for API and apps access
For an installation with Kuryr SDN on UPI, you should also check the requirements which are the same needed for OpenStack IPI with Kuryr. Please also note that RHEL 7 nodes are not supported on deployments configured with Kuryr. This is because Kuryr container images are based on RHEL 8 and may not work properly when run on RHEL 7.
This repository contains Ansible playbooks to deploy OpenShift on OpenStack.
Requirements:
- Python
- Ansible
- Python modules required in the playbooks. Namely:
- openstacksdk
- netaddr
- openstackclient
The file requirements.txt included in the playbook directory helps using pip for gathering the required dependencies in a Python virtual environment:
python -m venv venv
source venv/bin/activate
pip install -U pip
pip install -r requirements.txtAll the configuration files, logs and installation state are kept in a single directory:
$ mkdir -p openstack-upi
$ cd openstack-upiA proper RHCOS image in the OpenStack cluster or project is required for successful installation.
Get the RHCOS image for your OpenShift version here. You should download images with the highest version that is less than or equal to the OpenShift version that you install. Use the image versions that match your OpenShift version if they are available.
The OpenStack QCOW2 image is delivered in compressed format and therefore has the .gz extension. Unfortunately, compressed image support is not supported in OpenStack. So, you have to decompress the data before uploading it into Glance. The following command will unpack the image and create rhcos-${RHCOSVERSION}-openstack.qcow2 file without .gz extension.
$ gunzip rhcos-${RHCOSVERSION}-openstack.qcow2.gzNext step is to create a Glance image.
NOTE: This document will use rhcos as the Glance image name, but it's not mandatory.
$ openstack image create --container-format=bare --disk-format=qcow2 --file rhcos-${RHCOSVERSION}-openstack.qcow2 rhcosNOTE: Depending on your OpenStack environment you can upload the RHCOS image as raw or qcow2. See Disk and container formats for images for more information.
Finally validate that the image was successfully created:
$ openstack image show rhcosNOTE: throughout this document, we will use 203.0.113.23 as the public IP address for the OpenShift API endpoint and 203.0.113.19 as the public IP for the ingress (*.apps) endpoint.
$ openstack floating ip create --description "OpenShift API" <external>
=> 203.0.113.23
$ openstack floating ip create --description "OpenShift Ingress" <external>
=> 203.0.113.19The OpenShift API (for the OpenShift administrators and app developers) will be at api.<cluster name>.<cluster domain> and the Ingress (for the apps' end users) at *.apps.<cluster name>.<cluster domain>.
Create these two records in your DNS zone:
api.openshift.example.com. A 203.0.113.23
*.apps.openshift.example.com. A 203.0.113.19
They will need to be available to your developers, end users as well as the OpenShift installer process later in this guide.
Run the create install-config subcommand and fill in the desired entries:
$ openshift-install create install-config
? SSH Public Key </home/user/.ssh/id_rsa.pub>
? Platform <openstack>
? Cloud <openstack>
? ExternalNetwork <external>
? APIFloatingIPAddress <203.0.113.23>
? FlavorName <m1.xlarge>
? Base Domain <example.com>
? Cluster Name <openshift>Most of these are self-explanatory. Cloud is the cloud name in your clouds.yaml i.e. what's set as your OS_CLOUD environment variable.
Cluster Name and Base Domain will together form the fully qualified domain name which the API interface will expect to the called, and the default name with which OpenShift will expose newly created applications.
Given the values above, the OpenShift API will be available at:
https://api.openshift.example.com:6443/
Afterwards, you should have install-config.yaml in your current directory:
$ tree
.
└── install-config.yamlThe installer added a default IP range for the OpenShift nodes. It must match the range for the Neutron subnet you'll create later on.
We're going to use a custom subnet to illustrate how that can be done.
Our range will be 192.0.2.0/24 so we need to add that value to
install-config.yaml. Look under networking -> machineNetwork -> network -> cidr.
This command will do it for you:
$ python -c 'import yaml;
path = "install-config.yaml";
data = yaml.safe_load(open(path));
data["networking"]["machineNetwork"][0]["cidr"] = "192.0.2.0/24";
open(path, "w").write(yaml.dump(data, default_flow_style=False))'NOTE: All the scripts in this guide work with Python 3 as well as Python 2.
UPI will not rely on the Machine API for node creation. Instead, we will create the compute nodes ("workers") manually.
We will set their count to 0 in install-config.yaml. Look under compute -> (first entry) -> replicas.
This command will do it for you:
$ python -c '
import yaml;
path = "install-config.yaml";
data = yaml.safe_load(open(path));
data["compute"][0]["replicas"] = 0;
open(path, "w").write(yaml.dump(data, default_flow_style=False))'By default the networkType is set to OpenShiftSDN on the install-config.yaml.
If an installation with Kuryr is desired, you must modify the networkType field.
This command will do it for you:
$ python -c '
import yaml;
path = "install-config.yaml";
data = yaml.safe_load(open(path));
data["networking"]["networkType"] = "Kuryr";
open(path, "w").write(yaml.dump(data, default_flow_style=False))'Also set os_networking_type to Kuryr in inventory.yaml.
We are not relying on the Machine API so we can delete the control plane Machines and compute MachineSets from the manifests.
WARNING: The install-config.yaml file will be automatically deleted in the next section. If you want to keep it around, copy it elsewhere now!
First, let's turn the install config into manifests:
$ openshift-install create manifests
$ tree
.
├── manifests
│ ├── 04-openshift-machine-config-operator.yaml
│ ├── cloud-provider-config.yaml
│ ├── cluster-config.yaml
│ ├── cluster-dns-02-config.yml
│ ├── cluster-infrastructure-02-config.yml
│ ├── cluster-ingress-02-config.yml
│ ├── cluster-network-01-crd.yml
│ ├── cluster-network-02-config.yml
│ ├── cluster-proxy-01-config.yaml
│ ├── cluster-scheduler-02-config.yml
│ ├── cvo-overrides.yaml
│ ├── etcd-ca-bundle-configmap.yaml
│ ├── etcd-client-secret.yaml
│ ├── etcd-host-service-endpoints.yaml
│ ├── etcd-host-service.yaml
│ ├── etcd-metric-client-secret.yaml
│ ├── etcd-metric-serving-ca-configmap.yaml
│ ├── etcd-metric-signer-secret.yaml
│ ├── etcd-namespace.yaml
│ ├── etcd-service.yaml
│ ├── etcd-serving-ca-configmap.yaml
│ ├── etcd-signer-secret.yaml
│ ├── kube-cloud-config.yaml
│ ├── kube-system-configmap-root-ca.yaml
│ ├── machine-config-server-tls-secret.yaml
│ └── openshift-config-secret-pull-secret.yaml
└── openshift
├── 99_cloud-creds-secret.yaml
├── 99_kubeadmin-password-secret.yaml
├── 99_openshift-cluster-api_master-machines-0.yaml
├── 99_openshift-cluster-api_master-machines-1.yaml
├── 99_openshift-cluster-api_master-machines-2.yaml
├── 99_openshift-cluster-api_master-user-data-secret.yaml
├── 99_openshift-cluster-api_worker-machineset-0.yaml
├── 99_openshift-cluster-api_worker-user-data-secret.yaml
├── 99_openshift-machineconfig_master.yaml
├── 99_openshift-machineconfig_worker.yaml
├── 99_rolebinding-cloud-creds-secret-reader.yaml
└── 99_role-cloud-creds-secret-reader.yaml
2 directories, 38 filesRemove the control-plane Machines and compute MachineSets, because we'll be providing those ourselves and don't want to involve the machine-API operator:
$ rm -f openshift/99_openshift-cluster-api_master-machines-*.yaml openshift/99_openshift-cluster-api_worker-machineset-*.yamlYou are free to leave the compute MachineSets in if you want to create compute machines via the machine API, but if you do you may need to update the various references (subnet, etc.) to match your environment.
Currently emptying the compute pools makes control-plane nodes schedulable. But due to a Kubernetes limitation, router pods running on control-plane nodes will not be reachable by the ingress load balancer. Update the scheduler configuration to keep router pods and other workloads off the control-plane nodes:
$ python -c '
import yaml;
path = "manifests/cluster-scheduler-02-config.yml"
data = yaml.safe_load(open(path));
data["spec"]["mastersSchedulable"] = False;
open(path, "w").write(yaml.dump(data, default_flow_style=False))'Next, we will turn these manifests into Ignition files. These will be used to configure the Nova servers on boot (Ignition performs a similar function as cloud-init).
$ openshift-install create ignition-configs
$ tree
.
├── auth
│ ├── kubeadmin-password
│ └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ignThe OpenShift cluster has been assigned an identifier in the form of <cluster name>-<random string>. You do not need this for anything, but it is a good idea to keep it around.
You can see the various metadata about your future cluster in metadata.json.
The Infra ID is under the infraID key:
$ export INFRA_ID=$(jq -r .infraID metadata.json)
$ echo $INFRA_ID
openshift-qlvwvWe'll use the infraID as the prefix for all the OpenStack resources we'll create. That way, you'll be able to have multiple deployments in the same OpenStack project without name conflicts.
Make sure your shell session has the $INFRA_ID environment variable set when you run the commands later in this document.
The generated boostrap ignition file (bootstrap.ign) tends to be quite large (around 300KB -- it contains all the manifests, master and worker ignitions etc.). This is generally too big to be passed to the server directly (the OpenStack Nova user data limit is 64KB).
To boot it up, we will create a smaller Ignition file that will be passed to Nova as user data and that will download the main ignition file upon execution.
The main file needs to be uploaded to an HTTP(S) location the Bootstrap node will be able to access.
You are free to choose any storage you want. For example:
- Swift Object Storage
Create the
<container_name>container and upload thebootstrap.ignfile:Make the container accessible:$ swift upload <container_name> bootstrap.ign
Get the$ swift post <container_name> --read-acl ".r:*,.rlistings"
StorageURLfrom the output:$ swift stat -v
- Amazon S3
- Internal HTTP server inside your organisation
- A short-lived Nova server in
$INFRA_ID-nodeshosting the file for bootstrapping
In this guide, we will assume the file is at the following URL:
https://static.example.com/bootstrap.ign
NOTE: In case the Swift object storage option was chosen the URL will have the following format: <StorageURL>/<container_name>/bootstrap.ign
IMPORTANT: The bootstrap.ign contains sensitive information such as your clouds.yaml credentials and TLS certificates. It should not be accessible by the public! It will only be used once during the Nova boot of the Bootstrap server. We strongly recommend you restrict the access to that server only and delete the file afterwards.
As mentioned before due to Nova user data size limit, we will need to create a new Ignition file that will load the bulk of the Bootstrap node configuration. This will be similar to the existing master.ign and worker.ign files.
Create a file called $INFRA_ID-bootstrap-ignition.json (fill in your infraID) with the following contents:
{
"ignition": {
"config": {
"append": [
{
"source": "https://static.example.com/bootstrap.ign",
"verification": {}
}
]
},
"security": {},
"timeouts": {},
"version": "2.2.0"
},
"networkd": {},
"passwd": {},
"storage": {},
"systemd": {}
}Change the ignition.config.append.source field to the URL hosting the bootstrap.ign file you've uploaded previously.
We need to set the bootstrap hostname explicitly. The IPI installer does this automatically, but for now UPI does not.
We will update the ignition to create the following file:
/etc/hostname:
openshift-qlvwv-bootstrap
(using the infraID)
NOTE: We recommend you back up the Ignition files before making any changes!
You can edit the Ignition file manually or run this Python script:
import base64
import json
import os
with open('bootstrap.ign', 'r') as f:
ignition = json.load(f)
files = ignition['storage'].get('files', [])
infra_id = os.environ.get('INFRA_ID', 'openshift').encode()
hostname_b64 = base64.standard_b64encode(infra_id + b'-bootstrap\n').decode().strip()
files.append(
{
'path': '/etc/hostname',
'mode': 420,
'contents': {
'source': 'data:text/plain;charset=utf-8;base64,' + hostname_b64,
'verification': {}
},
'filesystem': 'root',
})
ignition['storage']['files'] = files;
with open('bootstrap.ign', 'w') as f:
json.dump(ignition, f)Feel free to make any other changes.
Similar to bootstrap, we need to make sure the hostname is set to the expected value (it must match the name of the Nova server exactly).
Since that value will be different for every master node, we will need to create multiple Ignition files: one for every node.
We will deploy three Control plane (master) nodes. Their Ignition configs can be create like so:
$ for index in $(seq 0 2); do
MASTER_HOSTNAME="$INFRA_ID-master-$index\n"
python -c "import base64, json, sys;
ignition = json.load(sys.stdin);
files = ignition['storage'].get('files', []);
files.append({'path': '/etc/hostname', 'mode': 420, 'contents': {'source': 'data:text/plain;charset=utf-8;base64,' + base64.standard_b64encode(b'$MASTER_HOSTNAME').decode().strip(), 'verification': {}}, 'filesystem': 'root'});
ignition['storage']['files'] = files;
json.dump(ignition, sys.stdout)" <master.ign >"$INFRA_ID-master-$index-ignition.json"
doneThis should create files openshift-qlvwv-master-0-ignition.json, openshift-qlvwv-master-1-ignition.json and openshift-qlvwv-master-2-ignition.json.
If you look inside, you will see that they contain very little. In fact, most of the master Ignition is served by the Machine Config Server running on the bootstrap node and the masters contain only enough to know where to look for the rest.
You can make your own changes here.
NOTE: The worker nodes do not require any changes to their Ignition, but you can make your own by editing worker.ign.
In this section we'll create all the networking pieces necessary to host the OpenShift cluster: security groups, network, subnet, router, ports.
$ ansible-playbook -i inventory.yaml 01_security-groups.yamlThe playbook creates one Security group for the Control Plane and one for the Compute nodes, then attaches rules for enabling communication between the nodes.
$ ansible-playbook -i inventory.yaml 02_network.yamlThe playbook creates a network and a subnet. The subnet obeys os_subnet_range; however the first ten IP addresses are removed from the allocation pool. These addresses will be used for the VRRP addresses managed by keepalived for high availability. For more information, read the networking infrastructure design document.
Outside connectivity will be provided by attaching the floating IP addresses (IPs in the inventory) to the corresponding routers.
NOTE: This step is optional and only necessary if you want to control the default resolvers your Nova servers will use.
During deployment, the OpenShift nodes will need to be able to resolve public name records to download the OpenShift images and so on. They will also need to resolve the OpenStack API endpoint.
The default resolvers are often set up by the OpenStack administrator in Neutron. However, some deployments do not have default DNS servers set, meaning the servers are not able to resolve any records when they boot.
If you are in this situation, you can add resolvers to your Neutron subnet (openshift-qlvwv-nodes). These will be put into /etc/resolv.conf on your servers post-boot.
For example, if you want to add the following nameservers: 198.51.100.86 and 198.51.100.87, you can run this command:
$ openstack subnet set --dns-nameserver <198.51.100.86> --dns-nameserver <198.51.100.87> "$INFRA_ID-nodes"$ ansible-playbook -i inventory.yaml 03_bootstrap.yamlThe playbook sets the allowed address pairs on each port attached to our OpenShift nodes.
Since the keepalived-managed IP addresses are not attached to any specific server, Neutron would block their traffic by default. By passing them to --allowed-address the traffic can flow freely through.
An additional Floating IP is also attached to the bootstrap port. This is not necessary for the deployment (and we will delete the bootstrap resources afterwards). However, if the bootstrapping phase fails for any reason, the installer will try to SSH in and download the bootstrap log. That will only succeed if the node is reachable (which in general means a floating IP).
After the bootstrap server is active, you can check the console log to see that it is getting the ignition correctly:
$ openstack console log show "$INFRA_ID-bootstrap"You can also SSH into the server (using its floating IP address) and check on the bootstrapping progress:
$ ssh [email protected]
[core@openshift-qlvwv-bootstrap ~]$ journalctl -b -f -u bootkube.service$ ansible-playbook -i inventory.yaml 04_control-plane.yamlOur control plane will consist of three nodes. The servers will be passed the master-?-ignition.json files prepared earlier.
The master nodes should load the initial Ignition and then keep waiting until the bootstrap node stands up the Machine Config Server which will provide the rest of the configuration.
If os_networking_type is set to Kuryr in the Ansible inventory, the playbook creates the Trunks for Kuryr to plug the containers into the OpenStack SDN.
When that happens, the masters will start running their own pods, run etcd and join the "bootstrap" cluster. Eventually, they will form a fully operational control plane.
You can monitor this via the following command:
$ openshift-install wait-for bootstrap-completeEventually, it should output the following:
INFO API v1.14.6+f9b5405 up
INFO Waiting up to 30m0s for bootstrapping to complete...
This means the masters have come up successfully and are joining the cluster.
Eventually, the wait-for command should end with:
INFO It is now safe to remove the bootstrap resources
You can use the oc or kubectl commands to talk to the OpenShift API. The admin credentials are in auth/kubeconfig:
$ export KUBECONFIG="$PWD/auth/kubeconfig"
$ oc get nodes
$ oc get pods -ANOTE: Only the API will be up at this point. The OpenShift UI will run on the compute nodes.
$ ansible-playbook -i inventory.yaml down-03_bootstrap.yamlThe teardown playbook deletes the bootstrap port, server and floating IP address.
If you haven't done so already, you should also disable the bootstrap Ignition URL.
$ ansible-playbook -i inventory.yaml 05_compute-nodes.yamlThis process is similar to the masters, but the workers need to be approved before they're allowed to join the cluster.
The workers need no ignition override.
If os_networking_type is set to Kuryr in the Ansible inventory, the playbook creates the Trunks for Kuryr to plug the containers into the OpenStack SDN.
Even after they've booted up, the workers will not show up in oc get nodes.
Instead, they will create certificate signing requests (CSRs) which need to be approved. You can watch for the CSRs here:
$ watch oc get csr -AEventually, you should see Pending entries looking like this
$ oc get csr -A
NAME AGE REQUESTOR CONDITION
csr-2scwb 16m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-5jwqf 16m system:node:openshift-qlvwv-master-0 Approved,Issued
csr-88jp8 116s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
csr-9dt8f 15m system:node:openshift-qlvwv-master-1 Approved,Issued
csr-bqkw5 16m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-dpprd 6s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
csr-dtcws 24s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Pending
csr-lj7f9 16m system:node:openshift-qlvwv-master-2 Approved,Issued
csr-lrtlk 15m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-wkm94 16m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,IssuedYou should inspect each pending CSR and verify that it comes from a node you recognise:
$ oc describe csr csr-88jp8
Name: csr-88jp8
Labels: <none>
Annotations: <none>
CreationTimestamp: Wed, 23 Oct 2019 13:22:51 +0200
Requesting User: system:serviceaccount:openshift-machine-config-operator:node-bootstrapper
Status: Pending
Subject:
Common Name: system:node:openshift-qlvwv-worker-0
Serial Number:
Organization: system:nodes
Events: <none>If it does (this one is for openshift-qlvwv-worker-0 which we've created earlier), you can approve it:
$ oc adm certificate approve csr-88jp8Approved nodes should now show up in oc get nodes, but they will be in the NotReady state. They will create a second CSR which you should also review:
$ oc get csr -A
NAME AGE REQUESTOR CONDITION
csr-2scwb 17m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-5jwqf 17m system:node:openshift-qlvwv-master-0 Approved,Issued
csr-7mv4d 13s system:node:openshift-qlvwv-worker-1 Pending
csr-88jp8 3m29s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-9dt8f 17m system:node:openshift-qlvwv-master-1 Approved,Issued
csr-bqkw5 18m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-bx7p4 28s system:node:openshift-qlvwv-worker-0 Pending
csr-dpprd 99s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-dtcws 117s system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-lj7f9 17m system:node:openshift-qlvwv-master-2 Approved,Issued
csr-lrtlk 17m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-wkm94 18m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-wqpfd 21s system:node:openshift-qlvwv-worker-2 Pending(we see the CSR approved earlier as well as a new Pending one for the same node: openshift-qlvwv-worker-0)
And approve:
$ oc adm certificate approve csr-bx7p4Once this CSR is approved, the node should switch to Ready and pods will be scheduled on it.
Run the following command to verify the OpenShift cluster is fully deployed:
$ openshift-install --log-level debug wait-for install-completeUpon success, it will print the URL to the OpenShift Console (the web UI) as well as admin username and password to log in.
$ ansible-playbook -i inventory.yaml \
down-06_load-balancers.yaml \
down-05_compute-nodes.yaml \
down-04_control-plane.yaml \
down-03_bootstrap.yaml \
down-02_network.yaml \
down-01_security-groups.yamlThe playbook down-06_load-balancers.yaml idempotently deletes the load balancers created by the Kuryr installation, if any.
Then, remove the api and *.apps DNS records.