Skip to content

OCPBUGS-51174: Updated the changing-cluster-network-mtu file with NIC… #89757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

OCPBUGS-51174: Updated the changing-cluster-network-mtu file with NIC… #89757

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions modules/aws-outposts-machine-set.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@ $ oc get machinesets.machine.openshift.io <original_machine_set_name_1> \
-n openshift-machine-api -o yaml
----
+
--
.Example output
[source,yaml]
----
Expand Down Expand Up @@ -90,11 +89,9 @@ spec:
<1> The cluster infrastructure ID.
<2> A default node label. For AWS Outposts, you use the `outposts` role.
<3> The omitted `providerSpec` section includes values that must be configured for your Outpost.
--

. Configure the new compute machine set to create edge compute machines in the Outpost by editing the `<new_machine_set_name_1>.yaml` file:
+
--
.Example compute machine set for AWS Outposts
[source,yaml]
----
Expand Down Expand Up @@ -166,7 +163,6 @@ spec:
<6> Specifies the AWS region in which the Outpost availability zone exists.
<7> Specifies the dedicated subnet for your Outpost.
<8> Specifies a taint to prevent workloads from being scheduled on nodes that have the `node-role.kubernetes.io/outposts` label. To schedule user workloads in the Outpost, you must specify a corresponding toleration in the `Deployment` resource for your application.
--

. Save your changes.

Expand Down
9 changes: 1 addition & 8 deletions modules/machineset-creating.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,6 @@ $ oc get machineset <machineset_name> \
-n openshift-machine-api -o yaml
----
+
--
.Example output
[source,yaml]
----
Expand Down Expand Up @@ -132,14 +131,8 @@ spec:
...
----
<1> The cluster infrastructure ID.
<2> A default node label.
+
[NOTE]
====
For clusters that have user-provisioned infrastructure, a compute machine set can only create `worker` and `infra` type machines.
====
<2> A default node label. For clusters that have user-provisioned infrastructure, a compute machine set can only create `worker` and `infra` type machines.
<3> The values in the `<providerSpec>` section of the compute machine set CR are platform-specific. For more information about `<providerSpec>` parameters in the CR, see the sample compute machine set CR configuration for your provider.
--

ifdef::vsphere[]
.. If you are creating a compute machine set for a cluster that has user-provisioned infrastructure, note the following important values:
Expand Down
7 changes: 2 additions & 5 deletions modules/nw-aws-load-balancer-with-outposts.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ You must annotate Ingress resources with the Outpost subnet or the VPC subnet, b

* Configure the `Ingress` resource to use a specified subnet:
+
--
.Example `Ingress` resource configuration
[source,yaml]
----
Expand All @@ -50,7 +49,5 @@ spec:
port:
number: 80
----
<1> Specifies the subnet to use.
* To use the Application Load Balancer in an Outpost, specify the Outpost subnet ID.
* To use the Application Load Balancer in the cloud, you must specify at least two subnets in different availability zones.
--
<1> Specifies the subnet to use. To use the Application Load Balancer in an Outpost, specify the Outpost subnet ID. To use the Application Load Balancer in the cloud, you must specify at least two subnets in different availability zones.

152 changes: 62 additions & 90 deletions modules/nw-cluster-mtu-change.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,7 @@ ifndef::outposts[= Changing the cluster network MTU]
ifdef::outposts[= Changing the cluster network MTU to support AWS Outposts]

ifdef::outposts[]
During installation, the maximum transmission unit (MTU) for the cluster network is detected automatically based on the MTU of the primary network interface of nodes in the cluster.
You might need to decrease the MTU value for the cluster network to support an AWS Outposts subnet.
During installation, the maximum transmission unit (MTU) for the cluster network is detected automatically based on the MTU of the primary network interface of nodes in the cluster. You might need to decrease the MTU value for the cluster network to support an AWS Outposts subnet.
endif::outposts[]

ifndef::outposts[As a cluster administrator, you can increase or decrease the maximum transmission unit (MTU) for your cluster.]
Expand Down Expand Up @@ -71,65 +70,65 @@ Status:
----

ifndef::local-zone,wavelength-zone,post-aws-zones,outposts[]
. Prepare your configuration for the hardware MTU:

** If your hardware MTU is specified with DHCP, update your DHCP configuration such as with the following dnsmasq configuration:
. Prepare your configuration for the hardware MTU by selecting one of the following methods:
+
.. If your hardware MTU is specified with DHCP, update your DHCP configuration similar to the following dnsmasq configuration:
+
[source,text]
----
dhcp-option-force=26,<mtu>
dhcp-option-force=26,<mtu> <1>
----
<1> Where `<mtu>` specifies the hardware MTU for the DHCP server to advertise.
+
.. If your hardware MTU is specified with a kernel command line with PXE, update that configuration accordingly.
+
.. If your hardware MTU is specified in a NetworkManager connection configuration, complete the following steps. This method is the default for {product-title} if you do not explicitly specify your network configuration with DHCP, a kernel command line, or some other method. Your cluster nodes must all use the same underlying network configuration for the following procedure to work unmodified.
+
--
where:

`<mtu>`:: Specifies the hardware MTU for the DHCP server to advertise.
--

** If your hardware MTU is specified with a kernel command line with PXE, update that configuration accordingly.

** If your hardware MTU is specified in a NetworkManager connection configuration, complete the following steps. This approach is the default for {product-title} if you do not explicitly specify your network configuration with DHCP, a kernel command line, or some other method. Your cluster nodes must all use the same underlying network configuration for the following procedure to work unmodified.

... Find the primary network interface by entering the following command:
+
[source,terminal]
----
$ oc debug node/<node_name> -- chroot /host nmcli -g connection.interface-name c show ovs-if-phys0
$ oc debug node/<node_name> -- chroot /host nmcli -g connection.interface-name c show ovs-if-phys0 <1> <2>
----
<1> Where `<node_name>` specifies the name of a node in your cluster.
<2> Where `ovs-if-phys0` is the primary network interface. For nodes that use multiple NIC bonds, append `bond-sub0` for the primary NIC bond interface and `bond-sub1` for the secondary NIC bond interface.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Append to what? Where does bond-sub0 and bond-sub1 come from?

+
--
where:

`<node_name>`:: Specifies the name of a node in your cluster.
--

... Create the following NetworkManager configuration in the `<interface>-mtu.conf` file:
... Create the following NetworkManager configuration in the `<interface>-mtu.conf` file.
+
.Example NetworkManager connection configuration
[source,ini]
----
[connection-<interface>-mtu]
match-device=interface-name:<interface>
ethernet.mtu=<mtu>
match-device=interface-name:<interface> <1>
ethernet.mtu=<mtu> <2>
----
<1> Where `<interface>` specifies the primary network interface name.
<2> Where `<mtu>` specifies the new hardware MTU value.
+
--
where:
[NOTE]
====
For nodes that use a network interface controller (NIC) bond interface, list the bond interface and any sub-interfaces in the `<bond-interface>-mtu.conf` file.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this information come from? The MTU of the sub-interfaces should automatically acquire the same MTU of the bond interface.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the <bond-interface>-mtu.conf mentioned before in this document.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jcaamano There are 2 things that come to mind with respect to the procedure as written:

  1. The docs does not address bonded NICs or VLANs at all. It is left as an exercise for the reader to extrapolate the documented steps, covering a single NIC, to multiple NICs under a bond

  2. The OpenShift agent based installer for bare metal systems allows for the MTU to be specified on each "interface"; sub NICs, bond, VLAN. If, during install, we set the MTU at all 3 levels, for some reason, and then we apply the MTU change procedure only to the sub NICs of the bond, will the bond and the vlan interfaces inherit the new sub NIC configuration, or, will the fact that I specifically set the MTU level at all 3 places during install mean that the sub NICs have the new MTU but the bond and vlan interfaces have the old MTU because there was an actual setting provided and not just inherited?

If on the other hand, changing the sub NIC interfaces' MTU would propagate then item 1 is really the primary concern. Leaving the extrapolation of the procedure as an exercise for the reader is risky and actually did impact one of our systems.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmc5179

  1. What do you feel this doc need to address about bonds that is missing? To my knowledge, the procedure would be exactly the same with bonds.
  2. You don't need to specify the MTU of the subordinates of the bond. They automatically acquire the MTU of the parent. Maybe this is what is being the problem but unknown without taking a deeper look. In any case, it would be impossible for this procedure to cover, or be aware of, all possible network configurations. I had already submitted a PR to add that disclaimer to this doc, I will dig that PR up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disclaimer Jira: https://issues.redhat.com/browse/OCPBUGS-17933


`<mtu>`:: Specifies the new hardware MTU value.
`<interface>`:: Specifies the primary network interface name.
--
.Example NetworkManager connection configuration
[source,ini]
----
[bond0-mtu]
match-device=interface-name:bond0
ethernet.mtu=9000

... Create two `MachineConfig` objects, one for the control plane nodes and another for the worker nodes in your cluster:
[connection-eth0-mtu]
match-device=interface-name:eth0
ethernet.mtu=9000

.... Create the following Butane config in the `control-plane-interface.bu` file:
+
[NOTE]
====
include::snippets/butane-version.adoc[]
[connection-eth1-mtu]
match-device=interface-name:eth1
ethernet.mtu=9000
----
====
+
[source,yaml, subs="attributes+"]
... Create the following Butane config in the `control-plane-interface.bu` file, which is the `MachineConfig` object for the control plane nodes:
+
[source,yaml,subs="attributes+"]
----
variant: openshift
version: {product-version}.0
Expand All @@ -145,16 +144,11 @@ storage:
mode: 0600
----
<1> Specify the NetworkManager connection name for the primary network interface.
<2> Specify the local filename for the updated NetworkManager configuration file from the previous step.

.... Create the following Butane config in the `worker-interface.bu` file:
<2> Specify the local filename for the updated NetworkManager configuration file from the previous step. For NIC bonds, specify the name for the `<bond-interface>-mtu.conf` file.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why <interface> would be different than <bond-interface>

+
[NOTE]
====
include::snippets/butane-version.adoc[]
====
... Create the following Butane config in the `worker-interface.bu` file, which is the `MachineConfig` object for the compute nodes:
+
[source,yaml, subs="attributes+"]
[source,yaml,subs="attributes+"]
----
variant: openshift
version: {product-version}.0
Expand All @@ -170,9 +164,9 @@ storage:
mode: 0600
----
<1> Specify the NetworkManager connection name for the primary network interface.
<2> Specify the local filename for the updated NetworkManager configuration file from the previous step.

.... Create `MachineConfig` objects from the Butane configs by running the following command:
<2> Specify the local filename for the updated NetworkManager configuration file from the previous step.
+
... Create `MachineConfig` objects from the Butane configs by running the following command:
+
[source,terminal]
----
Expand All @@ -192,23 +186,18 @@ endif::local-zone,wavelength-zone,post-aws-zones,outposts[]
[source,terminal]
----
$ oc patch Network.operator.openshift.io cluster --type=merge --patch \
'{"spec": { "migration": { "mtu": { "network": { "from": <overlay_from>, "to": <overlay_to> } , "machine": { "to" : <machine_to> } } } } }'
'{"spec": { "migration": { "mtu": { "network": { "from": <overlay_from>, "to": <overlay_to> } , "machine": { "to" : <machine_to> } } } } }' <1> <2> <3>
----
+
--
where:

`<overlay_from>`:: Specifies the current cluster network MTU value.
`<overlay_to>`:: Specifies the target MTU for the cluster network. This value is set relative to the value of `<machine_to>`. For OVN-Kubernetes, this value must be `100` less than the value of `<machine_to>`.
`<machine_to>`:: Specifies the MTU for the primary network interface on the underlying host network.
--
<1> Where `<overlay_from>` specifies the current cluster network MTU value.
<2> Where `<overlay_to>` specifies the target MTU for the cluster network.
<3> Where `<machine_to>` specifies the MTU for the primary network interface on the underlying host network. For OVN-Kubernetes, this value must be `100` less than the value of `<machine_to>`.
+
ifndef::outposts[]
.Example that increases the cluster MTU
[source,terminal]
----
$ oc patch Network.operator.openshift.io cluster --type=merge --patch \
'{"spec": { "migration": { "mtu": { "network": { "from": 1400, "to": 9000 } , "machine": { "to" : 9100} } } } }'
'{"spec": { "migration": { "mtu": { "network": { "from": 1400, "to": 8900 } , "machine": { "to" : 9000} } } } }'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change stemmed from https://issues.redhat.com/browse/OSDOCS-13689.

----
endif::outposts[]
ifdef::outposts[]
Expand Down Expand Up @@ -255,19 +244,16 @@ machineconfiguration.openshift.io/state: Done

.. Verify that the following statements are true:
+
--
* The value of `machineconfiguration.openshift.io/state` field is `Done`.
* The value of the `machineconfiguration.openshift.io/currentConfig` field is equal to the value of the `machineconfiguration.openshift.io/desiredConfig` field.
--

.. To confirm that the machine config is correct, enter the following command:
+
[source,terminal]
----
$ oc get machineconfig <config_name> -o yaml | grep ExecStart
$ oc get machineconfig <config_name> -o yaml | grep ExecStart <1>
----
+
where `<config_name>` is the name of the machine config from the `machineconfiguration.openshift.io/currentConfig` field.
<1> Where `<config_name>` is the name of the machine config from the `machineconfiguration.openshift.io/currentConfig` field.
+
The machine config must include the following update to the systemd configuration:
+
Expand All @@ -278,7 +264,7 @@ ExecStart=/usr/local/bin/mtu-migration.sh

ifndef::local-zone,wavelength-zone,post-aws-zones,outposts[]
. Update the underlying network interface MTU value:

+
** If you are specifying the new MTU with a NetworkManager connection configuration, enter the following command. The MachineConfig Operator automatically performs a rolling reboot of the nodes in your cluster.
+
[source,terminal]
Expand All @@ -287,7 +273,7 @@ $ for manifest in control-plane-interface worker-interface; do
oc create -f $manifest.yaml
done
----

+
** If you are specifying the new MTU with a DHCP server option or a kernel command line and PXE, make the necessary changes for your infrastructure.

. As the Machine Config Operator updates machines in each machine config pool, it reboots each node one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:
Expand Down Expand Up @@ -325,37 +311,28 @@ machineconfiguration.openshift.io/state: Done
+
Verify that the following statements are true:
+
--
* The value of `machineconfiguration.openshift.io/state` field is `Done`.
* The value of the `machineconfiguration.openshift.io/currentConfig` field is equal to the value of the `machineconfiguration.openshift.io/desiredConfig` field.
--
* The value of `machineconfiguration.openshift.io/state` field is `Done`.
* The value of the `machineconfiguration.openshift.io/currentConfig` field is equal to the value of the `machineconfiguration.openshift.io/desiredConfig` field.

.. To confirm that the machine config is correct, enter the following command:
+
[source,terminal]
----
$ oc get machineconfig <config_name> -o yaml | grep path:
$ oc get machineconfig <config_name> -o yaml | grep path: <1>
----
+
where `<config_name>` is the name of the machine config from the `machineconfiguration.openshift.io/currentConfig` field.
<1> Where `<config_name>` is the name of the machine config from the `machineconfiguration.openshift.io/currentConfig` field.
+
If the machine config is successfully deployed, the previous output contains the `/etc/NetworkManager/conf.d/99-<interface>-mtu.conf` file path and the `ExecStart=/usr/local/bin/mtu-migration.sh` line.
endif::local-zone,wavelength-zone,post-aws-zones,outposts[]

. To finalize the MTU migration, enter the following command for the OVN-Kubernetes network plugin:
. To finalize the MTU migration, enter the following command for the OVN-Kubernetes network plugin.
+
[source,terminal]
+
----
$ oc patch Network.operator.openshift.io cluster --type=merge --patch \
'{"spec": { "migration": null, "defaultNetwork":{ "ovnKubernetesConfig": { "mtu": <mtu> }}}}'
'{"spec": { "migration": null, "defaultNetwork":{ "ovnKubernetesConfig": { "mtu": <mtu> }}}}' <1>
----
+
--
where:

`<mtu>`:: Specifies the new cluster network MTU that you specified with `<overlay_to>`.
--
<1> Replace `<mtu>` with the new cluster network MTU that you specified with `<overlay_to>`.

. After finalizing the MTU migration, each machine config pool node is rebooted one by one. You must wait until all the nodes are updated. Check the machine config pool status by entering the following command:
+
Expand Down Expand Up @@ -398,15 +375,10 @@ $ oc get nodes
+
[source,terminal]
----
$ oc debug node/<node> -- chroot /host ip address show <interface>
$ oc debug node/<node> -- chroot /host ip address show <interface> <1> <2>
----
+
where:
+
--
`<node>`:: Specifies a node from the output from the previous step.
`<interface>`:: Specifies the primary network interface name for the node.
--
<1> Where `<node>` specifies a node from the output from the previous step.
<2> Where `<interface>` specifies the primary network interface name for the node.
+
.Example output
[source,text]
Expand Down
3 changes: 3 additions & 0 deletions networking/changing-cluster-network-mtu.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@ toc::[]
[role="_abstract"]
As a cluster administrator, you can change the MTU for the cluster network after cluster installation. This change is disruptive as cluster nodes must be rebooted to finalize the MTU change.

// About the cluster MTU
include::modules/nw-cluster-mtu-change-about.adoc[leveloffset=+1]

// Changing the cluster network MTU or Changing the cluster network MTU to support AWS Outposts
include::modules/nw-cluster-mtu-change.adoc[leveloffset=+1]

[role="_additional-resources"]
Expand Down