diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index 6f85081cdc6..ee8c0ac8209 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -3626,6 +3626,8 @@ Topics: File: ztp-vdu-validating-cluster-tuning - Name: Advanced managed cluster configuration with ClusterInstance resources File: ztp-advanced-install-ztp +- Name: Replacing managed cluster nodes with ClusterInstance resources + File: ztp-replacing-cluster-nodes - Name: Managing cluster policies with PolicyGenerator resources Dir: policygenerator_for_ztp Distros: openshift-origin,openshift-enterprise diff --git a/edge_computing/ztp-advanced-install-ztp.adoc b/edge_computing/ztp-advanced-install-ztp.adoc index c49a017c30d..9f4f518ef55 100644 --- a/edge_computing/ztp-advanced-install-ztp.adoc +++ b/edge_computing/ztp-advanced-install-ztp.adoc @@ -19,3 +19,8 @@ include::modules/ztp-configuring-cluster-network-mtu.adoc[leveloffset=+1] * xref:../edge_computing/ztp-advanced-install-ztp.adoc#ztp-customizing-the-install-extra-manifests_ztp-advanced-install-ztp[Customizing extra installation manifests in the {ztp} pipeline] include::modules/ztp-deleting-node-using-siteconfig.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources + +* xref:../edge_computing/ztp-replacing-cluster-nodes.adoc#ztp-replacing-failed-control-plane-node_ztp-replacing-cluster-nodes[Replacing a failed control plane node by using the ClusterInstance CR] diff --git a/edge_computing/ztp-replacing-cluster-nodes.adoc b/edge_computing/ztp-replacing-cluster-nodes.adoc new file mode 100644 index 00000000000..2b41ce10e47 --- /dev/null +++ b/edge_computing/ztp-replacing-cluster-nodes.adoc @@ -0,0 +1,30 @@ +:_mod-docs-content-type: ASSEMBLY +[id="ztp-replacing-cluster-nodes"] += Replacing managed cluster nodes with ClusterInstance resources +include::_attributes/common-attributes.adoc[] +:context: ztp-replacing-cluster-nodes + +toc::[] + +[role="_abstract"] +You can replace failed control plane nodes and add compute nodes on a GitOps-managed cluster by editing the `ClusterInstance` custom resource (CR). +The SiteConfig Operator applies the change and the Assisted Installer reprovisions the host. + +include::modules/ztp-about-clusterinstance-node-replacement.adoc[leveloffset=+1] + +include::modules/ztp-replacing-failed-control-plane-node.adoc[leveloffset=+1] + +include::modules/ztp-deleting-node-using-siteconfig.adoc[leveloffset=+1] + +include::modules/ztp-adding-compute-node-clusterinstance.adoc[leveloffset=+1] + +include::modules/ztp-clusterinstance-node-fields-ref.adoc[leveloffset=+1] + +[role="_additional-resources"] +[id="additional-resources_ztp-replacing-cluster-nodes"] +== Additional resources + +* xref:../edge_computing/ztp-sno-additional-worker-node.adoc#ztp-sno-additional-worker-node[Expanding single-node OpenShift clusters with GitOps ZTP] +* xref:../edge_computing/ztp-migrate-clusterinstance.adoc#ztp-migrate-clusterinstance[Migrating from SiteConfig CRs to ClusterInstance CRs] +* link:https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/{rh-rhacm-version}/html/multicluster_engine_operator_with_red_hat_advanced_cluster_management/siteconfig-intro[SiteConfig Operator] +* link:https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/{rh-rhacm-version}/html-single/apis/index#clusterinstance-api[ClusterInstance API] diff --git a/modules/ztp-about-clusterinstance-node-replacement.adoc b/modules/ztp-about-clusterinstance-node-replacement.adoc new file mode 100644 index 00000000000..88903469bab --- /dev/null +++ b/modules/ztp-about-clusterinstance-node-replacement.adoc @@ -0,0 +1,44 @@ +// Module included in the following assemblies: +// +// * edge_computing/ztp-replacing-cluster-nodes.adoc + +:_mod-docs-content-type: CONCEPT +[id="ztp-about-clusterinstance-node-replacement_{context}"] += About control plane node replacement with the ClusterInstance CR + +[role="_abstract"] +You can replace a failed control plane node on a bare-metal multi-node cluster by removing the node from the `ClusterInstance` custom resource (CR) and then adding a replacement node. +The SiteConfig Operator applies the Git change and the Assisted Installer reprovisions the host. + +When a control plane node becomes unreachable, the cluster continues to serve the API if etcd still has a quorum of at least two members. +You restore a three-member etcd cluster by completing the following GitOps operations, one node at a time: + +Scale in:: +Remove the failed node from the `ClusterInstance` CR. +The SiteConfig Operator deletes the generated installation manifests for that node, including the `BareMetalHost` resource. +The Assisted Installer deprovisions the host and removes the node from etcd. +Scale out:: +Add a replacement node to the `ClusterInstance` CR. +The SiteConfig Operator creates the installation manifests again. +The Assisted Installer provisions the host and joins it to the cluster. + +[IMPORTANT] +==== +Replace only one control plane node at a time. +Replacing two control plane nodes in parallel can lose etcd quorum and make the cluster unrecoverable with this workflow. +==== + +Complete these operations by using the `ClusterInstance` CR. +The `SiteConfig` CR is deprecated from {product-title} 4.18 and is removed from {product-title} 4.21. + +Any multi-node cluster that you installed with the SiteConfig Operator and a `ClusterInstance` CR can use the same scale-in and scale-out steps. + +The following limitations apply: + +* The cluster must keep an etcd quorum of at least two healthy control plane nodes for the duration of the workflow. +If only one control plane node remains, this workflow cannot restore the cluster. +* Do not use this workflow when control plane nodes host {rh-storage-first} daemons such as OSDs or monitors. +Replacing those nodes requires the storage replacement workflow so that daemons drain before the node is removed. +* Deprovisioning a node whose baseboard management controller (BMC) is unreachable can take much longer than a graceful removal. +* After the `BareMetalHost` resource is deleted, wait until the corresponding `Agent` resource is also deleted before you add the replacement node. +Adding the node again while the `Agent` resource still exists can cause the scale-out to fail. diff --git a/modules/ztp-adding-compute-node-clusterinstance.adoc b/modules/ztp-adding-compute-node-clusterinstance.adoc new file mode 100644 index 00000000000..8487139c9f5 --- /dev/null +++ b/modules/ztp-adding-compute-node-clusterinstance.adoc @@ -0,0 +1,113 @@ +// Module included in the following assemblies: +// +// * edge_computing/ztp-replacing-cluster-nodes.adoc + +:_mod-docs-content-type: PROCEDURE +[id="ztp-adding-compute-node-clusterinstance_{context}"] += Adding a compute node by using the ClusterInstance CR + +[role="_abstract"] +You can add a compute node to a multi-node cluster by appending a node entry to the `ClusterInstance` custom resource (CR) and pushing the change to Git. +The SiteConfig Operator creates the installation manifests and the Assisted Installer provisions the host. + +This procedure does not change etcd membership. +To remove a compute node first, see "Deleting a node by using the ClusterInstance CR". +Wait until the `BareMetalHost` and `Agent` resources are deleted, and then add the node again. + +.Prerequisites + +* You have installed the SiteConfig Operator on the hub cluster. +* You have deployed the managed cluster by using a `ClusterInstance` CR that Argo CD synchronizes from a Git repository. +* You have baseboard management controller (BMC) credentials for the new compute host. +* You have the kubeconfig file for the hub cluster and the kubeconfig file for the managed cluster. + +.Procedure + +. Add the compute node to the `spec.nodes` list in the `ClusterInstance` CR: ++ +[source,yaml] +---- +apiVersion: siteconfig.open-cluster-management.io/v1alpha1 +kind: ClusterInstance +metadata: + name: "example-cluster" + namespace: "example-cluster" +spec: + # ... + nodes: + - hostName: "compute-3.example.com" + role: "worker" + bmcAddress: "redfish-virtualmedia+https://192.0.2.33/redfish/v1/Systems/System.Embedded.1" + bmcCredentialsName: + name: "compute-3-bmc-secret" + bootMACAddress: "AA:BB:CC:DD:EE:33" + bootMode: "UEFI" + nodeNetwork: + interfaces: + - name: eno1 + macAddress: "AA:BB:CC:DD:EE:33" + config: + interfaces: + - name: eno1 + type: ethernet + state: up + macAddress: "AA:BB:CC:DD:EE:33" + ipv4: + enabled: true + address: + - ip: 192.0.2.13 + prefix-length: 24 + ipv6: + enabled: false +# ... +---- + +. Create a BMC authentication secret for the new host in the same namespace as the `ClusterInstance` CR: ++ +[source,yaml] +---- +apiVersion: v1 +kind: Secret +metadata: + name: "compute-3-bmc-secret" + namespace: example-cluster +data: + username: "username" + password: "password" +type: Opaque +---- + +. Commit the `ClusterInstance` CR and the secret, and push the changes to the Git repository that Argo CD monitors. + +.Verification + +. Verify that the SiteConfig Operator created a `BareMetalHost` resource for the new node by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get bmh -n +---- ++ +.Example output +[source,terminal] +---- +NAME STATE CONSUMER ONLINE ERROR AGE +compute-1.example.com provisioned true 10d +compute-3.example.com provisioning true 4m +---- ++ +The new host reports a `provisioning` state while the Assisted Installer installs it. + +. Verify that the `Agent` resource is approved and reaches the `Done` stage by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get agent -n +---- + +. Verify that the compute node is `Ready` on the managed cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get nodes +---- diff --git a/modules/ztp-clusterinstance-node-fields-ref.adoc b/modules/ztp-clusterinstance-node-fields-ref.adoc new file mode 100644 index 00000000000..a12a6e6cedd --- /dev/null +++ b/modules/ztp-clusterinstance-node-fields-ref.adoc @@ -0,0 +1,58 @@ +// Module included in the following assemblies: +// +// * edge_computing/ztp-replacing-cluster-nodes.adoc + +:_mod-docs-content-type: REFERENCE +[id="ztp-clusterinstance-node-fields-ref_{context}"] += ClusterInstance node management fields + +[role="_abstract"] +The following `ClusterInstance` custom resource (CR) fields control how the SiteConfig Operator adds or removes a node. +Use these fields when you scale a node in or out of a managed cluster. + +[cols="1,1,3",options="header"] +|==== +| Field | Type | Description + +| `spec.nodes.hostName` +| string +| Node hostname. +This value must match the `BareMetalHost` name that the SiteConfig Operator generates. + +| `spec.nodes.role` +| string +| Node role in the installation manifests. +Use `master` for a control plane node and `worker` for a compute node. + +| `spec.nodes.bmcAddress` +| string +| BMC URL that the installer uses to power and provision the host, for example a Redfish virtual media address. + +| `spec.nodes.bmcCredentialsName.name` +| string +| Name of the BMC `Secret` resource in the `ClusterInstance` namespace. + +| `spec.nodes.bootMACAddress` +| string +| MAC address of the NIC that boots the installation ISO. + +| `spec.nodes.bootMode` +| string +| Firmware boot mode, for example `UEFI`. + +| `spec.nodes.nodeNetwork` +| object +| NMState configuration for the node, including interfaces, IP addresses, DNS, and routes. + +| `spec.nodes.extraAnnotations.BareMetalHost` +| object +| Annotations that the SiteConfig Operator copies onto the generated `BareMetalHost` resource. +Set `bmac.agent-install.openshift.io/remove-agent-and-node-on-delete` to `"true"` before you prune the node so that the Assisted Installer removes the `Agent` resource and the node object. + +| `spec.nodes.pruneManifests` +| array +| Resources that the SiteConfig Operator deletes for this node. +To scale a node in, include the `BareMetalHost` resource (`apiVersion: metal3.io/v1alpha1`, `kind: BareMetalHost`). +|==== + +For the complete schema, see the link:https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/{rh-rhacm-version}/html-single/apis/index#clusterinstance-api[ClusterInstance API]. diff --git a/modules/ztp-deleting-node-using-siteconfig.adoc b/modules/ztp-deleting-node-using-siteconfig.adoc index 458dbea53d5..33874c7a1b3 100644 --- a/modules/ztp-deleting-node-using-siteconfig.adoc +++ b/modules/ztp-deleting-node-using-siteconfig.adoc @@ -1,6 +1,7 @@ // Module included in the following assemblies: // // * edge_computing/ztp-advanced-install-ztp.adoc +// * edge_computing/ztp-replacing-cluster-nodes.adoc :_mod-docs-content-type: PROCEDURE [id="ztp-deleting-node-clusterinstance_{context}"] @@ -10,12 +11,18 @@ By using a `ClusterInstance` custom resource (CR), you can delete and reprovision a node. This method is more efficient than manually deleting the node. +[IMPORTANT] +==== +If you delete a control plane node, at least two control plane nodes must remain healthy so that etcd keeps a quorum. +Replace only one control plane node at a time. +For the full control plane replacement workflow, including etcd verification, see "Replacing a failed control plane node by using the ClusterInstance CR". +==== + .Prerequisites * You have configured the hub cluster to generate the required installation and policy CRs. - -* You have created a Git repository in which you can manage your custom site configuration data. The repository must be accessible from the hub cluster and be defined as the source repository for the Argo CD application. - +* You have created a Git repository in which you can manage your custom site configuration data. +The repository must be accessible from the hub cluster and be defined as the source repository for the Argo CD application. .Procedure @@ -43,7 +50,8 @@ spec: + [source,terminal] ---- -$ oc get bmh -n -ojsonpath='{.metadata}' | jq -r '.annotations["bmac.agent-install.openshift.io/remove-agent-and-node-on-delete"]' +$ oc get bmh -n \ + -o jsonpath='{.metadata.annotations.bmac\.agent-install\.openshift\.io/remove-agent-and-node-on-delete}{"\n"}' ---- + .Example output @@ -66,6 +74,9 @@ spec: nodes: - hostName: "worker-node2.example.com" role: "worker" + extraAnnotations: + BareMetalHost: + bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true" pruneManifests: - apiVersion: metal3.io/v1alpha1 kind: BareMetalHost @@ -73,23 +84,28 @@ spec: ---- . Push the changes to the Git repository and wait for deprovisioning to start. -The status of the `BareMetalHost` CR should change to `deprovisioning`. Wait for the `BareMetalHost` to finish deprovisioning, and be fully deleted. +The status of the `BareMetalHost` CR changes to `deprovisioning`. +Wait until the `BareMetalHost` resource is fully deleted. + +. After the `BareMetalHost` resource is deleted, remove the node definition from the `spec.nodes` section in the `ClusterInstance` CR and push the changes to the Git repository. .Verification -. Verify that the `BareMetalHost` and `Agent` CRs for the worker node have been deleted from the hub cluster by running the following commands: +. Verify that the `BareMetalHost` resource for the node is deleted from the hub cluster by running the following command: + [source,terminal] ---- $ oc get bmh -n ---- + +. Verify that the `Agent` resource for the node is deleted from the hub cluster by running the following command: + [source,terminal] ---- $ oc get agent -n ---- -. Verify that the node record has been deleted from the spoke cluster by running the following command: +. Verify that the node record has been deleted from the managed cluster by running the following command: + [source,terminal] ---- @@ -98,13 +114,12 @@ $ oc get nodes + [NOTE] ==== -If you are working with secrets, deleting a secret too early can cause an issue because ArgoCD needs the secret to complete resynchronization after deletion. -Delete the secret only after the node cleanup, when the current ArgoCD synchronization is complete. +If you are working with secrets, deleting a secret too early can cause an issue because Argo CD needs the secret to complete resynchronization after deletion. +Delete the secret only after the node cleanup, when the current Argo CD synchronization is complete. ==== -. After the `BareMetalHost` object is successfully deleted, remove the worker node definition from the `spec.nodes` section in the `ClusterInstance` CR and push the changes to the Git repository. - .Next steps +Do not add the node again until the `Agent` resource for that node is deleted. To reprovision a node, add the node definition back to the `spec.nodes` section in the `ClusterInstance` CR, push the changes to the Git repository, and wait for the synchronization to complete. -This regenerates the `BareMetalHost` CR of the worker node and triggers the re-install of the node. +This regenerates the `BareMetalHost` CR for the node and triggers reinstallation of the node. diff --git a/modules/ztp-replacing-failed-control-plane-node.adoc b/modules/ztp-replacing-failed-control-plane-node.adoc new file mode 100644 index 00000000000..667c439250b --- /dev/null +++ b/modules/ztp-replacing-failed-control-plane-node.adoc @@ -0,0 +1,284 @@ +// Module included in the following assemblies: +// +// * edge_computing/ztp-replacing-cluster-nodes.adoc + +:_mod-docs-content-type: PROCEDURE +[id="ztp-replacing-failed-control-plane-node_{context}"] += Replacing a failed control plane node by using the ClusterInstance CR + +[role="_abstract"] +You restore etcd quorum after a control plane node fails by removing the failed node from the `ClusterInstance` custom resource (CR) and adding a replacement node. +Complete the replacement before a second control plane failure can lose quorum. + +.Prerequisites + +* You have installed the SiteConfig Operator on the hub cluster. +* You have deployed the managed cluster by using a `ClusterInstance` CR that Argo CD synchronizes from a Git repository. +* At least two of the three control plane nodes are in the `Ready` state so that etcd still has a quorum. +* You have a replacement bare-metal host with baseboard management controller (BMC) credentials. +If you reuse the same hardware, you can keep the existing BMC secret. +* Control plane nodes do not host {rh-storage} OSDs or monitors. +* You have the kubeconfig file for the hub cluster and the kubeconfig file for the managed cluster. + +.Procedure + +. Identify the failed control plane node on the managed cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get nodes +---- ++ +.Example output +[source,terminal] +---- +NAME STATUS ROLES AGE VERSION +control-plane-0.example.com Ready control-plane,master 10d v1.33.0 +control-plane-1.example.com Ready control-plane,master 10d v1.33.0 +control-plane-2.example.com NotReady control-plane,master 10d v1.33.0 +compute-1.example.com Ready worker 10d v1.33.0 +---- ++ +The failed node reports `NotReady` or is absent from the list. +Note the `hostName` value that matches that node in the `ClusterInstance` CR. + +. Confirm that two control plane nodes remain `Ready` by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get nodes -l node-role.kubernetes.io/control-plane +---- + +. In the `ClusterInstance` CR, add the `bmac.agent-install.openshift.io/remove-agent-and-node-on-delete` annotation to the failed node, and push the change to the Git repository: ++ +[source,yaml] +---- +apiVersion: siteconfig.open-cluster-management.io/v1alpha1 +kind: ClusterInstance +metadata: + name: "example-cluster" + namespace: "example-cluster" +spec: + # ... + nodes: + - hostName: "control-plane-2.example.com" + role: "master" + extraAnnotations: + BareMetalHost: + bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true" +# ... +---- + +. Verify that the `BareMetalHost` resource on the hub cluster has the annotation by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get bmh -n \ + -o jsonpath='{.metadata.annotations.bmac\.agent-install\.openshift\.io/remove-agent-and-node-on-delete}{"\n"}' +---- ++ +.Example output +[source,terminal] +---- +true +---- ++ +where: ++ +``:: Specifies the kubeconfig file for the managed cluster. +``:: Specifies the kubeconfig file for the hub cluster. +``:: Specifies the namespace of the `ClusterInstance` CR. +``:: Specifies the name of the `BareMetalHost` resource, which matches the node `hostName` value. + +. Configure the `pruneManifests` field for the failed node so that the SiteConfig Operator deletes the `BareMetalHost` resource. +Keep the deletion annotation. +Push the change to the Git repository: ++ +[source,yaml] +---- +apiVersion: siteconfig.open-cluster-management.io/v1alpha1 +kind: ClusterInstance +metadata: + name: "example-cluster" + namespace: "example-cluster" +spec: + # ... + nodes: + - hostName: "control-plane-2.example.com" + role: "master" + extraAnnotations: + BareMetalHost: + bmac.agent-install.openshift.io/remove-agent-and-node-on-delete: "true" + pruneManifests: + - apiVersion: metal3.io/v1alpha1 + kind: BareMetalHost +# ... +---- ++ +The status of the `BareMetalHost` resource changes to `deprovisioning`. +If the BMC is unreachable, deprovisioning can take significantly longer than a graceful removal. +Wait until the `BareMetalHost` resource is fully deleted. + +. Verify that the `BareMetalHost` resource is deleted from the hub cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get bmh -n +---- + +. Verify that the `Agent` resource for the failed node is deleted from the hub cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get agent -n +---- ++ +[IMPORTANT] +==== +Do not add the replacement node until the `Agent` resource is gone. +==== + +. Verify that the failed node is no longer listed on the managed cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get nodes +---- + +. Verify that etcd has two healthy members by running the following command against a remaining control plane node: ++ +[source,terminal] +---- +$ oc --kubeconfig -n openshift-etcd rsh etcd- \ + etcdctl member list -w table +---- ++ +.Example output +[source,terminal] +---- ++------------------+---------+-----------------------------+--------------------------+--------------------------+------------+ +| ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | ++------------------+---------+-----------------------------+--------------------------+--------------------------+------------+ +| 1111111111111111 | started | control-plane-0.example.com | https://192.0.2.10:2380 | https://192.0.2.10:2379 | false | +| 2222222222222222 | started | control-plane-1.example.com | https://192.0.2.11:2380 | https://192.0.2.11:2379 | false | ++------------------+---------+-----------------------------+--------------------------+--------------------------+------------+ +---- ++ +where: ++ +``:: Specifies the hostname of a remaining `Ready` control plane node. ++ +The failed member is absent from the table. + +. Optional: If you want to permanently remove the failed node definition, delete that entry from `spec.nodes` and push the change. +If you intend to reuse the same `hostName` value for the replacement, you can keep the entry and update it in the next step. + +. Add the replacement control plane node to `spec.nodes`. +Remove `pruneManifests` and the deletion annotation. +If the hardware changed, update at least the BMC address, BMC secret, boot MAC address, and node network configuration: ++ +[source,yaml] +---- +apiVersion: siteconfig.open-cluster-management.io/v1alpha1 +kind: ClusterInstance +metadata: + name: "example-cluster" + namespace: "example-cluster" +spec: + # ... + nodes: + - hostName: "control-plane-2.example.com" + role: "master" + bmcAddress: "redfish-virtualmedia+https://192.0.2.22/redfish/v1/Systems/System.Embedded.1" + bmcCredentialsName: + name: "control-plane-2-bmc-secret" + bootMACAddress: "AA:BB:CC:DD:EE:22" + bootMode: "UEFI" + nodeNetwork: + interfaces: + - name: eno1 + macAddress: "AA:BB:CC:DD:EE:22" + config: + interfaces: + - name: eno1 + type: ethernet + state: up + macAddress: "AA:BB:CC:DD:EE:22" + ipv4: + enabled: true + address: + - ip: 192.0.2.12 + prefix-length: 24 + ipv6: + enabled: false +# ... +---- + +. If the replacement host uses a new BMC secret, create the secret in the same namespace as the `ClusterInstance` CR: ++ +[source,yaml] +---- +apiVersion: v1 +kind: Secret +metadata: + name: "control-plane-2-bmc-secret" + namespace: example-cluster +data: + username: "username" + password: "password" +type: Opaque +---- + +. Commit the `ClusterInstance` CR and any new secret, and push the changes to the Git repository that Argo CD monitors. +The SiteConfig Operator creates a new `BareMetalHost` resource. +The Assisted Installer provisions the host and joins it to the cluster. + +. Monitor provisioning on the hub cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get bmh -n +---- ++ +.Example output +[source,terminal] +---- +NAME STATE CONSUMER ONLINE ERROR AGE +control-plane-0.example.com provisioned true 10d +control-plane-1.example.com provisioned true 10d +control-plane-2.example.com provisioning true 5m +---- + +.Verification + +. Verify that the replacement node is `Ready` on the managed cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get nodes -l node-role.kubernetes.io/control-plane +---- + +. Verify that etcd has three healthy members by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig -n openshift-etcd rsh etcd- \ + etcdctl member list -w table +---- + +. Verify that every etcd endpoint is healthy by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig -n openshift-etcd rsh etcd- \ + etcdctl endpoint health --cluster +---- + +. Verify that the `etcd` and `kube-apiserver` cluster operators have finished reconciling by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig get clusteroperators etcd kube-apiserver +---- ++ +Wait until both operators report `True` in the `AVAILABLE` column and `False` in the `PROGRESSING` column.