Skip to content

Commit

Permalink
Edit es2k-linux-networking-ecmp document
Browse files Browse the repository at this point in the history 
- Continue long command lines.
- Simplify/repair indentation.

Signed-off-by: Derek G Foster <[email protected]>
  • Loading branch information
@ffoulkes
ffoulkes committed Sep 26, 2023
1 parent 24fda4e commit d66ef99
Showing 1 changed file with 81 additions and 47 deletions.
128 changes: 81 additions & 47 deletions docs/apps/lnw/es2k/es2k-linux-networking-ecmp.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ System under test will have above topology running the networking recipe. Link P
## Create P4 artifacts and start Infrap4d process

- Use Linux networking p4 program present in the directory `/opt/p4/p4sde/share/mev_reference_p4_files/linux_networking` for this scenario.
- Refer to [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d) for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on ACC.
- See [Running Infrap4d on Intel IPU E2100](/guides/es2k/running-infrap4d) for compiling `P4 artifacts`, `bringing up ACC` and running `infrap4d` on ACC.

## Creating the topology

Expand All @@ -50,27 +50,34 @@ Once the application is started, set the forwarding pipeline config using
P4Runtime Client `p4rt-ctl` set-pipe command

```bash
$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin $OUTPUT_DIR/linux_networking.p4info.txt
$P4CP_INSTALL/bin/p4rt-ctl set-pipe br0 $OUTPUT_DIR/linux_networking.pb.bin \
$OUTPUT_DIR/linux_networking.p4info.txt
```

Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt` along with other P4 artifacts are created as per the steps mentioned in previous section.
Note: Assuming `linux_networking.pb.bin` and `linux_networking.p4info.txt`
along with other P4 artifacts are created as per the steps mentioned in previous section.

### Configure VSI Group and add a netdev

Use one of the IPDF netdevs on ACC to receive all control packets from overlay VM's by assigning to a VSI group. VSI group 3 is dedicated for this configuration, execute below devmem commands on IMC.
Use one of the IPDF netdevs on ACC to receive all control packets from overlay
VM's by assigning to a VSI group. VSI group 3 is dedicated for this configuration,
execute below devmem commands on IMC.

```bash
# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig
devmem 0x20292002a0 64 0x8000050000000008

# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3
# SEM_DIRECT_MAP_PGEN_DATA_VSI_GROUP : This will set vsi
# (set in SEM_DIRECT_MAP_PGEN_CTRL register LSB) into VSIG-3
devmem 0x2029200388 64 0x3

# SEM_DIRECT_MAP_PGEN_CTRL: LSB 11-bit is for vsi which need to map into vsig
devmem 0x20292002a0 64 0xA000050000000008
```

Note: Here VSI 8 has been used for receiving all control packets and added to VSI group 3. This refers to HOST netdev VSIG 3 as per the topology diagram. Modify this VSI based on your configuration.
Note: Here VSI 8 has been used for receiving all control packets and added
to VSI group 3. This refers to HOST netdev VSIG 3 as per the topology
diagram. Modify this VSI based on your configuration.

### Create Overlay network

Expand Down Expand Up @@ -111,25 +118,26 @@ Legacy OvS is used as a control plane for source MAC learning of overlay VM's. O
```bash
export RUN_OVS=/tmp
rm -rf $RUN_OVS/etc/openvswitch
rm -rf $RUN_OVS/var/run/openvswitch
rm -rf $RUN_OVS/var/run/openvswitch
mkdir -p $RUN_OVS/etc/openvswitch/
mkdir -p $RUN_OVS/var/run/openvswitch

ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db /opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema

ovsdb-tool create $RUN_OVS/etc/openvswitch/conf.db \
/opt/p4/p4-cp-nws/share/openvswitch/vswitch.ovsschema

ovsdb-server $RUN_OVS/etc/openvswitch/conf.db \
--remote=punix:$RUN_OVS/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,Open_vSwitch,manager_options \
--pidfile=$RUN_OVS/var/run/openvswitch/ovsdb-server.pid \
--unixctl=$RUN_OVS/var/run/openvswitch/ovsdb-server.ctl \
--detach
--detach

ovs-vswitchd --detach \
--pidfile=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.pid \
--no-chdir unix:$RUN_OVS/var/run/openvswitch/db.sock \
--unixctl=$RUN_OVS/var/run/openvswitch/ovs-vswitchd.ctl \
--mlockall \
--log-file=/tmp/ovs-vswitchd.log
--log-file=/tmp/ovs-vswitchd.log

alias ovs-vsctl="ovs-vsctl --db unix:$RUN_OVS/var/run/openvswitch/db.sock"
ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1
Expand All @@ -140,7 +148,8 @@ ovs-vsctl show

### Create VLAN representers

For each VM that is spawned for overlay network we need to have a port representer. We create VLAN netdevs on top of the IPDF netdev which is assigned to VSI group 3 in step-2 mentioned above.
For each VM that is spawned for overlay network we need to have a port representer.
We create VLAN netdevs on top of the IPDF netdev which is assigned to VSI group 3 in step-2 mentioned above.

```bash
ip link add link <VSI 8> name vlan1 type vlan id 1
Expand All @@ -152,7 +161,7 @@ ifconfig vlan2 up
Note: Here the assumption is, we have created 2 overlay VM's and creating 2 port representers for those VM's.
Port representer should always be in the format: `lowercase string 'vlan'+'vlanID'`

### Create intergation bridge and add ports to the bridge
### Create integration bridge and add ports to the bridge

Create OvS bridge, VxLAN tunnel and assign ports to the bridge.

Expand All @@ -165,7 +174,8 @@ ovs-vsctl add-port br-int vlan2
ifconfig vlan1 up
ifconfig vlan2 up

ovs-vsctl add-port br-int vxlan1 -- set interface vxlan1 type=vxlan options:local_ip=30.1.1.1 options:remote_ip=40.1.1.1 options:dst_port=4789
ovs-vsctl add-port br-int vxlan1 -- set interface vxlan1 type=vxlan \
options:local_ip=30.1.1.1 options:remote_ip=40.1.1.1 options:dst_port=4789
```

Note: Here we are creating VxLAN tunnel with VNI 0, you can create any VNI for tunneling.
Expand All @@ -179,38 +189,55 @@ Below configuration assumes
- Overlay VF1 has a VSI value 14
- Overlay VF2 has a VSI value 15

These VSI values can be checked with `/usr/bin/cli_client -q -c` command on IMC. This command provides VSI ID, Vport ID, and corresponding MAC addresses for all
These VSI values can be checked with `/usr/bin/cli_client -q -c` command
on IMC. This command provides VSI ID, Vport ID, and corresponding MAC
addresses for all:

- IDPF netdevs on ACC
- VF's on HOST
- IDPF netdevs on HOST (if IDPF driver loaded by you on HOST)
- Netdevs on IMC

```bash

# Rules for control packets coming from overlay VF(VSI-14), IPU will add a VLAN tag 1 and send to HOST1(VSI-8)

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table "vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)"

# Rules for control packets coming from overlay VF(VSI-15), IPU will add a VLAN tag 2 and send to HOST1(VSI-8)

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table "vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)"

# Rules for control packets coming from HOST1(VSI-8), IPU will remove the VLAN tag 1 and send to overlay VF(VSI-14)

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table "vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop"

# Rules for control packets coming from HOST1(VSI-8), IPU will remove the VLAN tag 2 and send to overlay VF(VSI-15)

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table "vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table "vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table "vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop"
# Rules for control packets coming from overlay VF (VSI-14).
# IPU will add a VLAN tag 1 and send to HOST1 (VSI-8).

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \
"vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(1,24)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \
"vmeta.common.vsi=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \
"vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_push(1,0,1)"

# Rules for control packets coming from overlay VF (VSI-15).
# IPU will add a VLAN tag 2 and send to HOST1 (VSI-8).

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \
"vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.add_vlan_and_send_to_port(2,24)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_host_to_ovs_table \
"vmeta.common.vsi=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(24)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_push_mod_table \
"vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_push(1,0,2)"

# Rules for control packets coming from HOST1 (VSI-8).
# IPU will remove the VLAN tag 1 and send to overlay VF (VSI-14).

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \
"vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=1,action=linux_networking_control.remove_vlan_and_send_to_port(1,30)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \
"vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=14,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(30)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \
"vmeta.common.mod_blob_ptr=1,action=linux_networking_control.vlan_pop"

# Rules for control packets coming from HOST1 (VSI-8).
# IPU will remove the VLAN tag 2 and send to overlay VF (VSI-15).

p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_ovs_to_host_table \
"vmeta.common.vsi=8,hdrs.dot1q_tag[vmeta.common.depth].hdr.vid=2,action=linux_networking_control.remove_vlan_and_send_to_port(2,31)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_loopback_from_ovs_to_host_table \
"vmeta.misc_internal.vm_to_vm_or_port_to_port[27:17]=15,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(31)"
p4rt-ctl add-entry br0 linux_networking_control.vlan_pop_mod_table \
"vmeta.common.mod_blob_ptr=2,action=linux_networking_control.vlan_pop"
```

### Configure rules for underlay control packets
Expand All @@ -226,16 +253,20 @@ Below configuration assumes

```bash
# Configuration for control packets between physical port 0 to underlay IDPF netdev VSI-10
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table "vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \
"vmeta.common.port_id=0,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(26)"

# Configuration for control packets between underlay IDPF netdev VSI-10 to physical port 0
p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)"
p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \
"vmeta.common.vsi=10,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(0)"

# Configuration for control packets between physical port 1 to underlay IDPF netdev VSI-11
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table "vmeta.common.port_id=1,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(27)"
p4rt-ctl add-entry br0 linux_networking_control.handle_rx_from_wire_to_ovs_table \
"vmeta.common.port_id=1,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(27)"

# Configuration for control packets between underlay IDPF netdev VSI-11 to physical port 1
p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table "vmeta.common.vsi=11,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(1)"
p4rt-ctl add-entry br0 linux_networking_control.handle_tx_from_host_to_ovs_and_ovs_to_wire_table \
"vmeta.common.vsi=11,user_meta.cmeta.bit32_zeros=0,action=linux_networking_control.set_dest(1)"
```

### Underlay configuration
Expand All @@ -251,7 +282,8 @@ Below configuration assumes
- Underlay IDPF netdev has a VSI value 11 for second ECMP member

```bash
p4rt-ctl add-entry br0 linux_networking_control.ecmp_lpm_root_lut "user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=2,action=linux_networking_control.ecmp_lpm_root_lut_action(0)"
p4rt-ctl add-entry br0 linux_networking_control.ecmp_lpm_root_lut \
"user_meta.cmeta.bit32_zeros=4/255.255.255.255,priority=2,action=linux_networking_control.ecmp_lpm_root_lut_action(0)"

ip link add dev TEP0 type dummy

Expand All @@ -266,7 +298,8 @@ ifconfig TEP0 30.1.1.1/24 up
ifconfig <IDPF netdev for VSI 10> 50.1.1.1/24 up
ifconfig <IDPF netdev for VSI 11> 60.1.1.1/24 up

ip route add 40.1.1.1 nexthop via 50.1.1.2 dev <IDPF netdev for VSI 10> weight 1 nexthop via 60.1.1.2 dev <IDPF netdev for VSI 11> weight 1
ip route add 40.1.1.1 nexthop via 50.1.1.2 dev <IDPF netdev for VSI 10> \
weight 1 nexthop via 60.1.1.2 dev <IDPF netdev for VSI 11> weight 1
ip route show
```

Expand All @@ -278,15 +311,16 @@ ifconfig TEP0 40.1.1.1/24 up
ifconfig <underlay 1> 50.1.1.2/24 up
ifconfig <underlay 2> 60.1.1.2/24 up

ip route add 30.1.1.1 nexthop via 50.1.1.1 dev <underlay 1> weight 1 nexthop via 60.1.1.1 dev <underlay 1> weight 1
ip route add 30.1.1.1 nexthop via 50.1.1.1 dev <underlay 1> \
weight 1 nexthop via 60.1.1.1 dev <underlay 1> weight 1
ip route show

```

Option 2: FRR running configuration

```bash
# <Install FRR on ACC and enable BGP protocol>
# <Install FRR on ACC and enable BGP protocol>
# <FRR VTYSH running configuration>

interface TEP0
Expand Down

0 comments on commit d66ef99

Please sign in to comment.