From 07d282893f38bc341f12b073c53870b99bb2ef55 Mon Sep 17 00:00:00 2001 From: Daniel Lawton Date: Tue, 11 Aug 2026 19:19:51 +0100 Subject: [PATCH] Add OpenStack multi-AZ documentation for IPI installation (OCPSTRAT-3035) Draft AsciiDoc modules for OpenStack availability zone support as a working reference for the docs team (betamax). Includes a concept module, multi-AZ sample install-config, and a procedure for configuring AZ placement. Wired into the custom and restricted install assemblies. --- ...installing-openstack-installer-custom.adoc | 12 + ...alling-openstack-installer-restricted.adoc | 11 + modules/installation-initializing.adoc | 13 + .../installation-osp-availability-zones.adoc | 31 ++ ...on-osp-configuring-availability-zones.adoc | 100 +++++++ .../installation-osp-multiaz-config-yaml.adoc | 82 ++++++ ...machineset-osp-multi-az-anti-affinity.adoc | 278 ++++++++++++++++++ 7 files changed, 527 insertions(+) create mode 100644 modules/installation-osp-availability-zones.adoc create mode 100644 modules/installation-osp-configuring-availability-zones.adoc create mode 100644 modules/installation-osp-multiaz-config-yaml.adoc create mode 100644 modules/machineset-osp-multi-az-anti-affinity.adoc diff --git a/installing/installing_openstack/installing-openstack-installer-custom.adoc b/installing/installing_openstack/installing-openstack-installer-custom.adoc index bd63f90eec8..28560dc9fe8 100644 --- a/installing/installing_openstack/installing-openstack-installer-custom.adoc +++ b/installing/installing_openstack/installing-openstack-installer-custom.adoc @@ -40,6 +40,15 @@ include::modules/installation-osp-describing-cloud-parameters.adoc[leveloffset=+ include::modules/installation-osp-setting-cloud-provider-options.adoc[leveloffset=+1] +include::modules/installation-osp-availability-zones.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources + +* xref:../../installing/installing_openstack/installation-config-parameters-openstack.adoc#installation-config-parameters-openstack[Installation configuration parameters for {rh-openstack}] + +* xref:../../machine_management/control_plane_machine_management/cpmso_provider_configurations/cpmso-config-options-openstack.adoc#cpmso-yaml-failure-domain-openstack_cpmso-config-options-openstack[Sample {rh-openstack-first} failure domain configuration for the Control Plane Machine Set Operator] + include::modules/installation-obtaining-installer.adoc[leveloffset=+1] include::modules/installation-initializing.adoc[leveloffset=+1] @@ -69,6 +78,9 @@ After you deploy your cluster, you can attach pods to additional networks. For m include::modules/installation-osp-config-yaml.adoc[leveloffset=+2] +include::modules/installation-osp-multiaz-config-yaml.adoc[leveloffset=+2] + +include::modules/installation-osp-configuring-availability-zones.adoc[leveloffset=+2] //Dual-stack networking include::modules/install-osp-dualstack.adoc[leveloffset=+2] diff --git a/installing/installing_openstack/installing-openstack-installer-restricted.adoc b/installing/installing_openstack/installing-openstack-installer-restricted.adoc index 6db0dc3f97b..0722d92a2aa 100644 --- a/installing/installing_openstack/installing-openstack-installer-restricted.adoc +++ b/installing/installing_openstack/installing-openstack-installer-restricted.adoc @@ -39,6 +39,15 @@ include::modules/installation-osp-describing-cloud-parameters.adoc[leveloffset=+ include::modules/installation-osp-setting-cloud-provider-options.adoc[leveloffset=+1] +include::modules/installation-osp-availability-zones.adoc[leveloffset=+1] + +[role="_additional-resources"] +.Additional resources + +* xref:../../installing/installing_openstack/installation-config-parameters-openstack.adoc#installation-config-parameters-openstack[Installation configuration parameters for {rh-openstack}] + +* xref:../../machine_management/control_plane_machine_management/cpmso_provider_configurations/cpmso-config-options-openstack.adoc#cpmso-yaml-failure-domain-openstack_cpmso-config-options-openstack[Sample {rh-openstack-first} failure domain configuration for the Control Plane Machine Set Operator] + include::modules/installation-creating-image-restricted.adoc[leveloffset=+1] include::modules/installation-initializing.adoc[leveloffset=+1] @@ -51,6 +60,8 @@ include::modules/installation-configure-proxy.adoc[leveloffset=+2] include::modules/installation-osp-restricted-config-yaml.adoc[leveloffset=+2] +include::modules/installation-osp-configuring-availability-zones.adoc[leveloffset=+2] + // include::modules/installation-osp-setting-worker-affinity.adoc[leveloffset=+1] include::modules/ssh-agent-using.adoc[leveloffset=+1] diff --git a/modules/installation-initializing.adoc b/modules/installation-initializing.adoc index b33f0e9c4bc..e375e8842d8 100644 --- a/modules/installation-initializing.adoc +++ b/modules/installation-initializing.adoc @@ -365,6 +365,19 @@ ifdef::osp[] ... Specify a {rh-openstack} flavor with at least 16 GB RAM to use for control plane nodes and 8 GB RAM for compute nodes. + +ifeval::["{context}" == "installing-openstack-installer-custom"] +[NOTE] +==== +After you create the installation configuration file, you can modify the file to deploy machines across multiple {rh-openstack} availability zones. For more information, see "{rh-openstack} availability zone enablement". +==== +endif::[] +ifeval::["{context}" == "installing-openstack-installer-restricted"] +[NOTE] +==== +After you create the installation configuration file, you can modify the file to deploy machines across multiple {rh-openstack} availability zones. For more information, see "{rh-openstack} availability zone enablement". +==== +endif::[] ++ ... Select the base domain to deploy the cluster to. All DNS records will be sub-domains of this base and will also include the cluster name. endif::osp[] diff --git a/modules/installation-osp-availability-zones.adoc b/modules/installation-osp-availability-zones.adoc new file mode 100644 index 00000000000..d626024a783 --- /dev/null +++ b/modules/installation-osp-availability-zones.adoc @@ -0,0 +1,31 @@ +// Module included in the following assemblies: +// +// * installing/installing_openstack/installing-openstack-installer-custom.adoc +// * installing/installing_openstack/installing-openstack-installer-restricted.adoc + +:_mod-docs-content-type: CONCEPT +[id="installation-osp-availability-zones_{context}"] += {rh-openstack} availability zone enablement + +You can deploy an {product-title} cluster across multiple {rh-openstack-first} availability zones (AZs) by configuring Nova compute placement and Cinder root volume placement in the installation configuration file. Spreading control plane and compute machines across AZs reduces the risk that a failure in a single compute or storage failure domain takes down the cluster. + +[IMPORTANT] +==== +Availability zones in {rh-openstack} do not inherently guarantee physical fault isolation. The isolation boundary depends on how your {rh-openstack} administrator configured the zones. Confirm with your cloud provider which failure domains each AZ represents before you rely on them for high availability. +==== + +The default installation configuration deploys machines by using the default Nova and Cinder settings for your cloud. Those defaults often place the cluster in a single effective failure domain. To deploy a cluster across multiple availability zones, you must edit the `install-config.yaml` file so that each machine pool specifies Nova `zones` and matching Cinder `rootVolume.zones`. + +The `install-config.yaml` file includes the following fields for availability zone placement: + +* `controlPlane.platform.openstack.zones` and `compute.platform.openstack.zones`: Nova availability zones where the installation program creates machines. +* `controlPlane.platform.openstack.rootVolume.zones` and `compute.platform.openstack.rootVolume.zones`: Cinder availability zones where the installation program creates root volumes. + +[NOTE] +==== +If you set `zones` for a machine pool and define a `rootVolume` block, you must also set `rootVolume.zones` for that machine pool. Pair each Nova availability zone with a Cinder availability zone that can serve machines in that failure domain. Nova and Cinder zone names do not have to match. +==== + +When the number of replicas does not divide evenly across AZs, the installer round-robins placement. For strict high-availability requirements, match the replica count to the AZ count (for example, 3 replicas across 3 AZs). + +After installation, the installation program encodes these values into machine resources. Control plane failure domains appear in the `ControlPlaneMachineSet` custom resource as paired Nova and Cinder availability zones. Compute machines use the Nova and Cinder zones that you configured for the compute machine pool. diff --git a/modules/installation-osp-configuring-availability-zones.adoc b/modules/installation-osp-configuring-availability-zones.adoc new file mode 100644 index 00000000000..5afab942a18 --- /dev/null +++ b/modules/installation-osp-configuring-availability-zones.adoc @@ -0,0 +1,100 @@ +// Module included in the following assemblies: +// +// * installing/installing_openstack/installing-openstack-installer-custom.adoc +// * installing/installing_openstack/installing-openstack-installer-restricted.adoc + +:_mod-docs-content-type: PROCEDURE +[id="installation-osp-configuring-availability-zones_{context}"] += Configuring availability zones for {rh-openstack} + +You can modify the `install-config.yaml` file so that the installation program deploys control plane and compute machines across multiple {rh-openstack-first} availability zones. + +.Prerequisites + +* You have an existing `install-config.yaml` installation configuration file. +* Your {rh-openstack} cloud provides multiple Nova availability zones with schedulable compute capacity in each zone. +* Your {rh-openstack} cloud provides Cinder volume availability zones that you can pair with the Nova availability zones that you plan to use. +* You have installed and configured the OpenStack command-line interface (`openstack`). + +.Procedure + +. List the Nova and Cinder availability zones in your cloud: ++ +[source,terminal] +---- +$ openstack availability zone list --compute +$ openstack availability zone list --volume +---- + +. Edit your `install-config.yaml` file. For each machine pool that you want to spread across availability zones, set `platform.openstack.zones` to the Nova availability zones and `platform.openstack.rootVolume.zones` to the paired Cinder availability zones. ++ +The lists must contain the same number of entries. The installation program pairs zones by position in each list. ++ +[NOTE] +==== +You can optionally set `platform.openstack.serverGroupPolicy` to `soft-anti-affinity` to spread machines across different compute hosts within an availability zone. This setting does not replace availability zone placement. +==== ++ +.Sample `install-config.yaml` availability zone configuration +[%collapsible] +==== +[source,yaml] +---- +# ... +controlPlane: + name: master + replicas: 3 + platform: + openstack: + type: + zones: + - + - + - + rootVolume: + size: 25 + types: + - + zones: + - + - + - + serverGroupPolicy: soft-anti-affinity +compute: +- name: worker + replicas: 6 + platform: + openstack: + type: + zones: + - + - + - + rootVolume: + size: 25 + types: + - + zones: + - + - + - + serverGroupPolicy: soft-anti-affinity +# ... +---- +==== + +. Change to the directory that contains the installation program and generate manifests to validate the configuration: ++ +[source,terminal] +---- +$ ./openshift-install create manifests --dir +---- + +. Optional: Inspect the generated `ControlPlaneMachineSet` manifest in `/openshift/` and confirm that `failureDomains.openstack` lists the Nova and Cinder availability zone pairs that you configured. + +[role="_additional-resources"] +.Additional resources + +* xref:../../machine_management/control_plane_machine_management/cpmso_provider_configurations/cpmso-config-options-openstack.adoc#cpmso-yaml-failure-domain-openstack_cpmso-config-options-openstack[Sample {rh-openstack-first} failure domain configuration for the Control Plane Machine Set Operator] + +* xref:../../installing/installing_openstack/installation-config-parameters-openstack.adoc#installation-config-parameters-openstack[Installation configuration parameters for {rh-openstack}] diff --git a/modules/installation-osp-multiaz-config-yaml.adoc b/modules/installation-osp-multiaz-config-yaml.adoc new file mode 100644 index 00000000000..d1d3d7612c1 --- /dev/null +++ b/modules/installation-osp-multiaz-config-yaml.adoc @@ -0,0 +1,82 @@ +// Module included in the following assemblies: +// +// * installing/installing_openstack/installing-openstack-installer-custom.adoc + +:_mod-docs-content-type: REFERENCE +[id="installation-osp-multiaz-config-yaml_{context}"] += Sample multi-AZ install-config.yaml file for {rh-openstack} + +The following example `install-config.yaml` file deploys a highly available cluster across three {rh-openstack-first} availability zones. The example pairs Nova availability zones with Cinder root volume availability zones for the control plane and compute machine pools. + +[IMPORTANT] +This sample file is provided for reference only. You must obtain your `install-config.yaml` file by using the installation program. Replace availability zone names, flavors, volume types, and other values with settings from your cloud. + +.Example multi-AZ `install-config.yaml` file +[%collapsible] +==== +[source,yaml] +---- +apiVersion: v1 +baseDomain: example.com +controlPlane: + name: master + replicas: 3 + platform: + openstack: + type: m1.xlarge + zones: + - nova-az0 + - nova-az1 + - nova-az2 + rootVolume: + size: 25 + types: + - performance + zones: + - cinder-az0 + - cinder-az1 + - cinder-az2 + serverGroupPolicy: soft-anti-affinity +compute: +- name: worker + replicas: 6 + platform: + openstack: + type: m1.large + zones: + - nova-az0 + - nova-az1 + - nova-az2 + rootVolume: + size: 25 + types: + - performance + zones: + - cinder-az0 + - cinder-az1 + - cinder-az2 + serverGroupPolicy: soft-anti-affinity +metadata: + name: multiaz +networking: + clusterNetwork: + - cidr: 10.128.0.0/14 + hostPrefix: 23 + machineNetwork: + - cidr: 10.0.0.0/16 + serviceNetwork: + - 172.30.0.0/16 + networkType: OVNKubernetes +platform: + openstack: + cloud: mycloud + externalNetwork: external +ifndef::openshift-origin[] +fips: false +endif::openshift-origin[] +pullSecret: '{"auths": ...}' +sshKey: ssh-ed25519 AAAA... +---- +==== + +For more information about configuring availability zones, see xref:installation-osp-configuring-availability-zones_{context}[Configuring availability zones for {rh-openstack}]. diff --git a/modules/machineset-osp-multi-az-anti-affinity.adoc b/modules/machineset-osp-multi-az-anti-affinity.adoc new file mode 100644 index 00000000000..09654e37471 --- /dev/null +++ b/modules/machineset-osp-multi-az-anti-affinity.adoc @@ -0,0 +1,278 @@ +// Module included in the following assemblies: +// +// * machine_management/creating_machinesets/creating-machineset-osp.adoc + +:_mod-docs-content-type: PROCEDURE +[id="machineset-osp-multi-az-anti-affinity_{context}"] += Distributing worker nodes across OpenStack availability zones with anti-affinity + +You can distribute {product-title} worker nodes across multiple {rh-openstack-first} Nova availability zones and configure anti-affinity policies to ensure nodes are spread across different compute hosts for improved fault tolerance. + +[IMPORTANT] +==== +Availability zones in OpenStack do not inherently guarantee physical fault isolation. The actual fault tolerance depends on how your OpenStack administrator configured the zones. Consult your cloud provider to understand the isolation boundaries. +==== + +.Prerequisites + +* You have an {rh-openstack} cloud with multiple Nova availability zones configured. +* You have identified available zones by running `openstack availability zone list --compute`. +* You have obtained your cluster's infrastructure ID by running: ++ +[source,terminal] +---- +$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster +---- + +.Procedure + +. Identify the availability zones in your {rh-openstack} environment: ++ +[source,terminal] +---- +$ openstack availability zone list --compute +---- ++ +.Example output +[source,terminal] +---- ++-----------+----------+ +| Zone Name | Status | ++-----------+----------+ +| nova-az0 | available| +| nova-az1 | available| +| nova-az2 | available| ++-----------+----------+ +---- + +. Create a server group with an anti-affinity policy for each availability zone: ++ +[source,terminal] +---- +$ openstack server group create \ + --policy soft-anti-affinity \ + -worker- +---- ++ +where: ++ +``:: Specifies your cluster's infrastructure ID. +``:: Specifies the availability zone name, for example `az0`. ++ +.Example command +[source,terminal] +---- +$ openstack server group create \ + --policy soft-anti-affinity \ + cluster-abc-worker-az0 +---- ++ +[NOTE] +==== +Use `soft-anti-affinity` to allow instance creation even if anti-affinity cannot be satisfied. Use `anti-affinity` for strict enforcement, but note this requires additional compute hosts during maintenance and may prevent scaling if insufficient hosts are available. +==== + +. Record the server group ID from the output: ++ +[source,terminal] +---- +$ openstack server group show -worker- -c id -f value +---- ++ +.Example output +[source,terminal] +---- +7ee219f3-d2e9-48a1-96c2-e7429f1b0da7 +---- + +. List existing worker MachineSets to use as a template: ++ +[source,terminal] +---- +$ oc get machineset -n openshift-machine-api +---- ++ +.Example output +[source,terminal] +---- +NAME DESIRED CURRENT READY AVAILABLE +cluster-abc-worker-0 3 3 3 3 +---- + +. Export an existing worker MachineSet configuration: ++ +[source,terminal] +---- +$ oc get machineset -n openshift-machine-api -o yaml > machineset-az0.yaml +---- + +. Edit the file to configure the first availability zone: ++ +[source,yaml] +---- +apiVersion: machine.openshift.io/v1beta1 +kind: MachineSet +metadata: + labels: + machine.openshift.io/cluster-api-cluster: + machine.openshift.io/cluster-api-machine-role: worker + machine.openshift.io/cluster-api-machine-type: worker + name: -worker-az0 # <1> + namespace: openshift-machine-api +spec: + replicas: 2 # <2> + selector: + matchLabels: + machine.openshift.io/cluster-api-cluster: + machine.openshift.io/cluster-api-machineset: -worker-az0 + template: + metadata: + labels: + machine.openshift.io/cluster-api-cluster: + machine.openshift.io/cluster-api-machine-role: worker + machine.openshift.io/cluster-api-machine-type: worker + machine.openshift.io/cluster-api-machineset: -worker-az0 + spec: + providerSpec: + value: + apiVersion: machine.openshift.io/v1alpha1 + availabilityZone: nova-az0 # <3> + cloudName: openstack + cloudsSecret: + name: openstack-cloud-credentials + namespace: openshift-machine-api + flavor: + image: + serverGroupID: # <4> + kind: OpenstackProviderSpec + networks: + - filter: {} + subnets: + - filter: + name: + tags: openshiftClusterID= + primarySubnet: + rootVolume: # <5> + availabilityZone: cinder-az0 + size: 25 + volumeType: + securityGroups: + - filter: {} + name: -worker + serverMetadata: + Name: -worker + openshiftClusterID: + tags: + - openshiftClusterID= + trunk: true + userDataSecret: + name: worker-user-data +---- +<1> Unique MachineSet name including the availability zone identifier. +<2> Number of worker replicas for this zone. For even distribution across 3 zones with 6 total workers, use `2` replicas per zone. +<3> Nova availability zone where compute instances will be created. +<4> Server group ID from step 3 to enforce anti-affinity policy. +<5> Optional: Specify Cinder availability zone for root volumes. Aligning storage and compute zones improves reliability. + +. Remove the `status` section entirely from the YAML file. + +. Create the MachineSet for the first availability zone: ++ +[source,terminal] +---- +$ oc create -f machineset-az0.yaml +---- + +. Repeat steps 2-9 for each additional availability zone, modifying: ++ +* MachineSet name (for example, `-worker-az1`) +* `availabilityZone` value (for example, `nova-az1`) +* `rootVolume.availabilityZone` value (for example, `cinder-az1`) +* `serverGroupID` value (create separate server group for each zone) +* All label selectors to match the new MachineSet name + +.Verification + +. Verify that all MachineSets are created and scaling: ++ +[source,terminal] +---- +$ oc get machineset -n openshift-machine-api +---- ++ +.Example output +[source,terminal] +---- +NAME DESIRED CURRENT READY AVAILABLE AGE +cluster-abc-worker-az0 2 2 2 2 5m +cluster-abc-worker-az1 2 2 2 2 4m +cluster-abc-worker-az2 2 2 2 2 3m +---- + +. Verify that worker nodes are distributed across availability zones: ++ +[source,terminal] +---- +$ oc get nodes -L topology.kubernetes.io/zone +---- ++ +.Example output +[source,terminal] +---- +NAME STATUS ROLES AGE VERSION ZONE +worker-az0-1 Ready worker 5m15s v1.30.1+012d409 nova-az0 +worker-az0-2 Ready worker 5m10s v1.30.1+012d409 nova-az0 +worker-az1-1 Ready worker 4m20s v1.30.1+012d409 nova-az1 +worker-az1-2 Ready worker 4m18s v1.30.1+012d409 nova-az1 +worker-az2-1 Ready worker 3m25s v1.30.1+012d409 nova-az2 +worker-az2-2 Ready worker 3m22s v1.30.1+012d409 nova-az2 +---- ++ +The `topology.kubernetes.io/zone` label is automatically applied to nodes based on the availability zone specified in the MachineSet. + +. Verify anti-affinity by checking that instances are on different compute hosts: ++ +[source,terminal] +---- +$ openstack server list --name -worker -c Name -c Host -c Status +---- ++ +.Example output +[source,terminal] +---- ++---------------------------+------------------+---------+ +| Name | Host | Status | ++---------------------------+------------------+---------+ +| cluster-abc-worker-az0-1 | compute-host-01 | ACTIVE | +| cluster-abc-worker-az0-2 | compute-host-02 | ACTIVE | +| cluster-abc-worker-az1-1 | compute-host-03 | ACTIVE | +| cluster-abc-worker-az1-2 | compute-host-04 | ACTIVE | ++---------------------------+------------------+---------+ +---- ++ +Instances should be distributed across different compute hosts according to the anti-affinity policy. + +. Verify that instances are members of the server groups: ++ +[source,terminal] +---- +$ openstack server group show -worker-az0 +---- ++ +.Example output +[source,terminal] +---- ++----------+--------------------------------------+ +| Field | Value | ++----------+--------------------------------------+ +| id | 7ee219f3-d2e9-48a1-96c2-e7429f1b0da7 | +| members | instance-uuid-1, instance-uuid-2 | +| name | cluster-abc-worker-az0 | +| policies | soft-anti-affinity | ++----------+--------------------------------------+ +---- + +[NOTE] +==== +The machine-api-provider-openstack automatically applies the `machine.openshift.io/zone` label to Machine resources, which propagates to Node resources as `topology.kubernetes.io/zone`. This enables pod topology spread constraints for zone-aware workload placement. +====