Router could not be initialized - when using FromDPDKDevice/ToDPDKDevice elements with DelayShaper element

[cgautcher]

Hello,

I am working on a project to build a DPDK-based network bridge that will add delay to packets passing through it. I was looking at some of the elements that are included with FastClick, and DelayShaper seems like a logical choice.

I am working with a server with two 12-core CPU sockets (Intel Xeon Silver 4214 CPU @ 2.20GHz), with about 384GB of RAM balanced across each socket, with about 288GB of RAM allocated to 1GB HugePages, which is set via kernel boot time arguments (default_hugepagesz=1G hugepagesz=1G hugepages=288). The server has a dual-port 100Gb ConnectX-5 with the latest driver/firmware. DPDK version 23.11 and the latest main branch of FastClick.

$ sudo numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35
node 0 size: 192108 MB
node 0 free: 41762 MB
node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42 43 44 45 46 47
node 1 size: 193488 MB
node 1 free: 43975 MB
node distances:
node   0   1 
  0:  10  21 
  1:  21  10 

When trying a simple passthrough bridge using FromDPDKDevice and ToDPDKDevice, everything seems to work OK. I get good performance with very little dropped packets when running my iperf3 test between hosts on either end of the bridge.

$ cat bridge.click 
FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) -> ToDPDKDevice(1);
FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) -> ToDPDKDevice(0);

username@servername:~$ sudo ./fastclick-main/bin/click  --dpdk -a d8:00.0 -a d8:00.1 -c 0x0FFF000FFF0 -n 6 -- --file /home/username/bridge.click
[sudo] password for username:
EAL: Detected CPU lcores: 48
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.1 (socket 1)
FromDPDKDevice@1: 6 threads will be used to push packets downstream. 6 queues will be used meaning that each queue will be shared by 1 threads
FromDPDKDevice@1: using queues from 0 to 5
FromDPDKDevice@1: Queue 0 handled by th 12
FromDPDKDevice@1: Queue 1 handled by th 13
FromDPDKDevice@1: Queue 2 handled by th 14
FromDPDKDevice@1: Queue 3 handled by th 15
FromDPDKDevice@1: Queue 4 handled by th 16
FromDPDKDevice@1: Queue 5 handled by th 17
FromDPDKDevice@3: 6 threads will be used to push packets downstream. 6 queues will be used meaning that each queue will be shared by 1 threads
FromDPDKDevice@3: using queues from 0 to 5
FromDPDKDevice@3: Queue 0 handled by th 18
FromDPDKDevice@3: Queue 1 handled by th 19
FromDPDKDevice@3: Queue 2 handled by th 20
FromDPDKDevice@3: Queue 3 handled by th 21
FromDPDKDevice@3: Queue 4 handled by th 22
FromDPDKDevice@3: Queue 5 handled by th 23
Initializing DPDK

But when I try to add the DelayShaper element, it throws an error "Router could not be initialized!"

Config file and run attempt with DelayShaper:

$ cat bridge_delay_shaper.click 
FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) 
    -> DelayShaper(5ms) 
    -> ToDPDKDevice(1);

FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) 
    -> DelayShaper(5ms) 
    -> ToDPDKDevice(0);


$ sudo ./fastclick-main/bin/click  --dpdk -a d8:00.0 -a d8:00.1 -c 0x0FFF000FFF0 -n 6 -- --file /home/username/bridge_delay_shaper.click 
[sudo] password for username: 
EAL: Detected CPU lcores: 48
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.1 (socket 1)
‘FromDPDKDevice@1 :: FromDPDKDevice’ push output 0 connected to ‘DelayShaper@2 :: DelayShaper’ pull input 0
‘DelayShaper@2 :: DelayShaper’ pull output 0 connected to ‘ToDPDKDevice@3 :: ToDPDKDevice’ push input 0
‘FromDPDKDevice@4 :: FromDPDKDevice’ push output 0 connected to ‘DelayShaper@5 :: DelayShaper’ pull input 0
‘DelayShaper@5 :: DelayShaper’ pull output 0 connected to ‘ToDPDKDevice@6 :: ToDPDKDevice’ push input 0
Router could not be initialized!

This was my "./configure" line I used:

./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -O3" CXXFLAGS="-g -std=gnu++11 -O3" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-dpdk --enable-batch --with-netmap=no --enable-zerocopy --disable-dpdk-pool --disable-dpdk-packet --enable-local

My questions are:
- Is there a way to increase verbosity to determine why the router config did not initialize when using DelayShaper?
- Is DelayShaper meant for that type of use case of holding packets for an extra N milliseconds? Is there a better included element for that use case? Or would that require a custom element?

Thanks in advance

[ahenning]

***@***.*** :: FromDPDKDevice’ push output 0 connected to ***@***.*** :: DelayShaper’ pull input 0

It looks like a Queue element is needed between the FromDevice and the Shaper. From: cgautcher ***@***.*** Sent: 12 June 2024 04:18 AM To: tbarbette/fastclick ***@***.***> Cc: Subscribed ***@***.***> Subject: [tbarbette/fastclick] Router could not be initialized - when using FromDPDKDevice/ToDPDKDevice elements with DelayShaper element (Discussion #393) Hello, I am working on a project to build a DPDK-based network bridge that will add delay to packets passing through it. I was looking at some of the elements that are included with FastClick, and DelayShaper seems like a logical choice. I am working with a server with two 12-core CPU sockets (Intel Xeon Silver 4214 CPU @ 2.20GHz), with about 384GB of RAM balanced across each socket, with about 288GB of RAM allocated to 1GB HugePages, which is set via kernel boot time arguments (default_hugepagesz=1G hugepagesz=1G hugepages=288). The server has a dual-port 100Gb ConnectX-5 with the latest driver/firmware. DPDK version 23.11 and the latest main branch of FastClick. $ sudo numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 24 25 26 27 28 29 30 31 32 33 34 35 node 0 size: 192108 MB node 0 free: 41762 MB node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 36 37 38 39 40 41 42 43 44 45 46 47 node 1 size: 193488 MB node 1 free: 43975 MB node distances: node 0 1 0: 10 21 1: 21 10 When trying a simple passthrough bridge using FromDPDKDevice and ToDPDKDevice, everything seems to work OK. I get good performance with very little dropped packets when running my iperf3 test between hosts on either end of the bridge. $ cat bridge.click FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) -> ToDPDKDevice(1); FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) -> ToDPDKDevice(0); ***@***.***:~$ sudo ./fastclick-main/bin/click --dpdk -a d8:00.0 -a d8:00.1 -c 0x0FFF000FFF0 -n 6 -- --file /home/username/bridge.click [sudo] password for username: EAL: Detected CPU lcores: 48 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: VFIO support initialized EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.0 (socket 1) EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.1 (socket 1) ***@***.***: 6 threads will be used to push packets downstream. 6 queues will be used meaning that each queue will be shared by 1 threads ***@***.***: using queues from 0 to 5 ***@***.***: Queue 0 handled by th 12 ***@***.***: Queue 1 handled by th 13 ***@***.***: Queue 2 handled by th 14 ***@***.***: Queue 3 handled by th 15 ***@***.***: Queue 4 handled by th 16 ***@***.***: Queue 5 handled by th 17 ***@***.***: 6 threads will be used to push packets downstream. 6 queues will be used meaning that each queue will be shared by 1 threads ***@***.***: using queues from 0 to 5 ***@***.***: Queue 0 handled by th 18 ***@***.***: Queue 1 handled by th 19 ***@***.***: Queue 2 handled by th 20 ***@***.***: Queue 3 handled by th 21 ***@***.***: Queue 4 handled by th 22 ***@***.***: Queue 5 handled by th 23 Initializing DPDK But when I try to add the DelayShaper element, it throws an error "Router could not be initialized!" Config file and run attempt with DelayShaper: $ cat bridge_delay_shaper.click FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) -> DelayShaper(5ms) -> ToDPDKDevice(1); FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) -> DelayShaper(5ms) -> ToDPDKDevice(0); $ sudo ./fastclick-main/bin/click --dpdk -a d8:00.0 -a d8:00.1 -c 0x0FFF000FFF0 -n 6 -- --file /home/username/bridge_delay_shaper.click [sudo] password for username: EAL: Detected CPU lcores: 48 EAL: Detected NUMA nodes: 2 EAL: Detected static linkage of DPDK EAL: Multi-process socket /var/run/dpdk/rte/mp_socket EAL: Selected IOVA mode 'PA' EAL: VFIO support initialized EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.0 (socket 1) EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.1 (socket 1) ***@***.*** :: FromDPDKDevice’ push output 0 connected to ***@***.*** :: DelayShaper’ pull input 0 ***@***.*** :: DelayShaper’ pull output 0 connected to ***@***.*** :: ToDPDKDevice’ push input 0 ***@***.*** :: FromDPDKDevice’ push output 0 connected to ***@***.*** :: DelayShaper’ pull input 0 ***@***.*** :: DelayShaper’ pull output 0 connected to ***@***.*** :: ToDPDKDevice’ push input 0 Router could not be initialized! This was my "./configure" line I used: ./configure --enable-multithread --disable-linuxmodule --enable-intel-cpu --enable-user-multithread --verbose CFLAGS="-g -O3" CXXFLAGS="-g -std=gnu++11 -O3" --disable-dynamic-linking --enable-poll --enable-bound-port-transfer --enable-dpdk --enable-batch --with-netmap=no --enable-zerocopy --disable-dpdk-pool --disable-dpdk-packet --enable-local My questions are: - Is there a way to increase verbosity to determine why the router config did not initialize when using DelayShaper? - Is DelayShaper meant for that type of use case of holding packets for an extra N milliseconds? Is there a better included element for that use case? Or would that require a custom element? Thanks in advance — Reply to this email directly, view it on GitHub<#393>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABUOB2Y5K5OXE23MA6QMAWLZG6VURAVCNFSM6AAAAABJFNRKXCVHI2DSMVQWIX3LMV43ERDJONRXK43TNFXW4OZWHAYTAMJVGQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[cgautcher]

Thank you for your reply! Unfortunately I'm getting the same "Router could not be initialized!" message.

$ sudo ./fastclick-main/bin/click  --dpdk -a d8:00.0 -a d8:00.1 -c 0x0FFF000FFF0 -n 6  -- --file /home/spade/bridge_queue_delay_shaper.click 
EAL: Detected CPU lcores: 48
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.0 (socket 1)
EAL: Probe PCI driver: mlx5_pci (15b3:1019) device: 0000:d8:00.1 (socket 1)
‘DelayShaper@3 :: DelayShaper’ pull output 0 connected to ‘ToDPDKDevice@4 :: ToDPDKDevice’ push input 0
‘DelayShaper@7 :: DelayShaper’ pull output 0 connected to ‘ToDPDKDevice@8 :: ToDPDKDevice’ push input 0
Router could not be initialized!

$ cat bridge_queue_delay_shaper.click
FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) 
    -> Queue()
    -> DelayShaper(5ms)
    -> ToDPDKDevice(1);

FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) 
    -> Queue()
    -> DelayShaper(5ms) 
    -> ToDPDKDevice(0);

Is there a way to increase verbosity? Or any other recommended troubleshooting methods?

[tbarbette]

Hi @cgautcher !

The error is right above :
‘DelayShaper@7 :: DelayShaper’ pull output 0 connected to ‘ToDPDKDevice@8 :: ToDPDKDevice’ push input 0

DelayShaper is a pull element (see https://github.com/tbarbette/fastclick/wiki/DelayShaper) so it must be between two pull ports. Here Queue is push-to-pull but ToDPDKDevice is push.

What you want should be DelayUnqueue (https://github.com/tbarbette/fastclick/wiki/DelayUnqueue). However I never made it "batch-native", so performance won't be the best of the best as it can't use batching.

So you must do Queue->DelayUnqueue.

[cgautcher]

Hi Tom!

Thanks for the reply. I tried that, and it did start!

I'm not sure what to expect performance-wise with the non batch-native DelayUnqueue element. My simple FromDPDKDevice -> ToDPDKDevice gets about 10 Gb/s, but when I use FromDPDKDevice -> Queue-> DelayUnqueue -> ToDPDKDevice config, I'm only seeing 250Kb/s.

Then I did a 100-packet ping flood, where it sends all 100 packets at once, and I thought this was interesting:

# ping -l 100 -c 100 dest-ip
PING dest-ip (dest-ip) from source-ip: 56 data bytes
64 bytes from dest-ip: icmp_seq=0 ttl=64 time=1.152 ms
64 bytes from dest-ip: icmp_seq=1 ttl=64 time=2.117 ms
64 bytes from dest-ip: icmp_seq=2 ttl=64 time=3.112 ms
64 bytes from dest-ip: icmp_seq=3 ttl=64 time=4.101 ms
64 bytes from dest-ip: icmp_seq=4 ttl=64 time=5.101 ms
64 bytes from dest-ip: icmp_seq=5 ttl=64 time=6.105 ms
64 bytes from dest-ip: icmp_seq=6 ttl=64 time=7.101 ms
64 bytes from dest-ip: icmp_seq=7 ttl=64 time=8.102 ms
64 bytes from dest-ip: icmp_seq=8 ttl=64 time=9.096 ms
64 bytes from dest-ip: icmp_seq=9 ttl=64 time=10.101 ms
<...packets 10 through 96 snipped from this output...>
64 bytes from dest-ip: icmp_seq=97 ttl=64 time=97.989 ms
64 bytes from dest-ip: icmp_seq=98 ttl=64 time=98.983 ms
64 bytes from dest-ip: icmp_seq=99 ttl=64 time=99.990 ms

--- dest-ip ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.152/50.592/99.990/28.777 ms

If I just do one ping at a time, it works as expected:

# ping dest-ip
PING dest-ip (dest-ip) from source-ip: 56 data bytes
64 bytes from dest-ip: icmp_seq=0 ttl=64 time=2.158 ms
64 bytes from dest-ip: icmp_seq=1 ttl=64 time=2.215 ms
64 bytes from dest-ip: icmp_seq=2 ttl=64 time=2.191 ms
64 bytes from dest-ip: icmp_seq=3 ttl=64 time=2.110 ms
64 bytes from dest-ip: icmp_seq=4 ttl=64 time=2.232 ms
64 bytes from dest-ip: icmp_seq=5 ttl=64 time=2.253 ms
^C
--- dest-ip ping statistics ---
6 packets transmitted, 6 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.110/2.193/2.253/0.048 ms

I'm looking for functionality where if all 100 packets are sent at the same time, they should all take 2x the delay time (2ms in this case). Do you have any suggestions on how to achieve that?

Click config for the above ping test.

$ cat tom.click 
FromDPDKDevice(0, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) 
    -> Queue()
    -> DelayUnqueue(1ms)
    -> ToDPDKDevice(1);

FromDPDKDevice(1, PROMISC true, JUMBO true, MTU 9000, VERBOSE 9) 
    -> Queue()
    -> DelayUnqueue(1ms) 
    -> ToDPDKDevice(0);

[tbarbette]

Indeed, the DelayUnqueue element has only storage for one packet, so it's not what you want.

LinkUnqueue(LATENCY 1ms) might be what you want?