From 1e59cab19f3ac4d61e4307509a35d1964ad26318 Mon Sep 17 00:00:00 2001 From: Derek G Foster Date: Tue, 26 Sep 2023 13:28:51 -0700 Subject: [PATCH] Edit es2k-linux-networking document (#295) - Continue long command lines. - Simplify/repair indentation. Signed-off-by: Derek G Foster --- docs/apps/lnw/es2k/es2k-linux-networking.md | 104 ++++++++++++-------- 1 file changed, 65 insertions(+), 39 deletions(-) diff --git a/docs/apps/lnw/es2k/es2k-linux-networking.md b/docs/apps/lnw/es2k/es2k-linux-networking.md index 27730d4f..304cd09c 100644 --- a/docs/apps/lnw/es2k/es2k-linux-networking.md +++ b/docs/apps/lnw/es2k/es2k-linux-networking.md @@ -38,11 +38,11 @@ System under test will have above topology running the networking recipe. Link P ## Create P4 artifacts and start Infrap4d process - Use Linux networking p4 program present in the directory `/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking` for this scenario. -- Refer to [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d.md) for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on ACC. +- See [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d) for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on ACC. ## Creating the topology - *Note*: p4rt-ctl and ovs-vsctl utilities used in below steps can be found under $P4CP_INSTALL/bin +The p4rt-ctl and ovs-vsctl utilities can be found in $P4CP_INSTALL/bin. ### Set the forwarding pipeline @@ -50,20 +50,25 @@ Once the application is started, set the forwarding pipeline config using P4Runtime Client `p4rt-ctl` set-pipe command ```bash -$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin $OUTPUT_DIR/linux_networking.p4info.txt +$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin \ + $OUTPUT_DIR/linux_networking.p4info.txt ``` -Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt` along with other P4 artifacts are created as per the steps mentioned in previous section. +Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt` +along with other P4 artifacts are created as per the steps mentioned in previous section. ### Configure VSI Group and add a netdev -Use one of the IPDF netdevs on ACC to receive all control packets from overlay VM's by assigning to a VSI group. VSI group 3 is dedicated for this configuration, execute below devmem commands on IMC. +Use one of the IPDF netdevs on ACC to receive all control packets from overlay +VM's by assigning to a VSI group. VSI group 3 is dedicated for this configuration, +execute below devmem commands on IMC. ```bash # SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig devmem 0x20292002a0 64 0x8000050000000008 -# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3 +# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi +# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3. devmem 0x2029200388 64 0x3 # SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig @@ -89,8 +94,8 @@ ip addr add 99.0.0.2/24 dev ifconfig up ``` -Option 2: If we are unable to spawn VM's on top of the VF's, for this use case we can also leverage kernel network namespaces. -Move each VF to a network namespace and assign IP addresses +Option 2: If we are unable to spawn VM's on top of the VF's, we can leverage kernel network namespaces. +Move each VF to a network namespace and assign IP addresses: ```bash ip netns add VM0 @@ -115,7 +120,8 @@ rm -rf $RUN_OVS/var/run/openvswitch mkdir -p $RUN_OVS/etc/openvswitch/ mkdir -p $RUN_OVS/var/run/openvswitch -ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db /opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema +ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \ + /opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema ovsdb-server $RUN_OVS/etc/openvswitch/conf.db \ --remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \ @@ -140,7 +146,8 @@ ovs-vsctl show ### Create VLAN representers -For each VM that is spawned for overlay network we need to have a port representer. We create VLAN netdevs on top of the IPDF netdev which is assigned to VSI group 3 in step-2 mentioned above. +For each VM that is spawned for overlay network we need to have a port representer. +We create VLAN netdevs on top of the IPDF netdev which is assigned to VSI group 3 in step-2 mentioned above. ```bash ip link add link name vlan1 type vlan id 1 @@ -152,7 +159,7 @@ ifconfig vlan2 up Note: Here the assumption is, we have created 2 overlay VM's and creating 2 port representers for those VM's. Port representer should always be in the format: `lowercase string 'vlan'+'vlanID'` -### Create intergation bridge and add ports to the bridge +### Create integration bridge and add ports to the bridge Create OvS bridge, VxLAN tunnel and assign ports to the bridge. @@ -165,7 +172,8 @@ ovs-vsctl add-port br-int vlan2 ifconfig vlan1 up ifconfig vlan2 up -ovs-vsctl add-port br-int vxlan1 -- set interface vxlan1 type=vxlan options:local_ip=40.1.1.1 options:remote_ip=40.1.1.2 options:dst_port=4789 +ovs-vsctl add-port br-int vxlan1 -- set interface vxlan1 type=vxlan \ + options:local_ip=40.1.1.1 options:remote_ip=40.1.1.2 options:dst_port=4789 ``` Note: Here we are creating VxLAN tunnel with VNI 0, you can create any VNI for tunneling. @@ -187,30 +195,45 @@ These VSI values can be checked with `/usr/bin/cli_client -q -c` command on IMC. - Netdevs on IMC ```bash - -# Rules for control packets coming from overlay VF(VSI-14), IPU will add a VLAN tag 1 and send to HOST1(VSI-8) - - p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)" - p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" - p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)" - -# Rules for control packets coming from overlay VF(VSI-15), IPU will add a VLAN tag 2 and send to HOST1(VSI-8) - - p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)" - p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" - p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)" - -# Rules for control packets coming from HOST1(VSI-8), IPU will remove the VLAN tag 1 and send to overlay VF(VSI-14) - - p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)" - p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)" - p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop" - -# Rules for control packets coming from HOST1(VSI-8), IPU will remove the VLAN tag 2 and send to overlay VF(VSI-15) - - p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)" - p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)" - p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop" +# Rules for control packets coming from overlay VF (VSI-14). +# IPU will add a VLAN tag 1 and send to HOST1 (VSI-8). + +p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ + "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)" +p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ + "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" +p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ + "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)" + +# Rules for control packets coming from overlay VF (VSI-15). +# IPU will add a VLAN tag 2 and send to HOST1 (VSI-8). + +p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ + "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)" +p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \ + "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)" +p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \ + "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)" + +# Rules for control packets coming from HOST1 (VSI-8). +# IPU will remove the VLAN tag 1 and send to overlay VF (VSI-14). + +p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ + "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)" +p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ + "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)" +p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ + "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop" + +# Rules for control packets coming from HOST1 (VSI-8). +# IPU will remove the VLAN tag 2 and send to overlay VF (VSI-15). + +p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \ + "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)" +p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \ + "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)" +p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \ + "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop" ``` ### Configure rules for underlay control packets @@ -224,10 +247,12 @@ Below configuration assumes ```bash # Configuration for control packets between physical port 0 to underlay IDPF netdev VSI-10 - p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table "vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)" +p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \ + "vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)" # Configuration for control packets between underlay IDPF netdev VSI-10 to physical port 0 - p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)" +p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \ + "vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)" ``` ### Underlay configuration @@ -239,7 +264,8 @@ Below configuration assumes - Underlay IDPF netdev has a VSI value 10 ```bash -p4rt-ctl add-entry br0 linux_networking_control.ecmp_lpm_root_lut "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=2,action=linux_networking_control.ecmp_lpm_root_lut_action(0)" +p4rt-ctl add-entry br0 linux_networking_control.ecmp_lpm_root_lut \ + "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=2,action=linux_networking_control.ecmp_lpm_root_lut_action(0)" nmcli device set managed no ifconfig 40.1.1.1/24 up