forked from mtcp-stack/mtcp
-
Notifications
You must be signed in to change notification settings - Fork 1
/
README.netmap
196 lines (166 loc) · 7.85 KB
/
README.netmap
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
- NETMAP VERSION -
------------------
-*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*-*-*-*-
-A- Tested with linux-3.13.0-121-generic & netmap commit: 8f3f79e4ca168c6d3220e7a5198a5778b030445e,
and,
-B- Tested with linux-3.13.0-121-generic & netmap commit: 21227d47ae995812412d1de22855c7fa8260f53a,
and,
-C- Tested with 4.11.12-200.fc25.x86_64+debug & netmap commit: 21227d47ae995812412d1de22855c7fa8260f53a
-*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*--*-*-*-*-*-*-
FOR -A-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding ixgbe-3.15.1/ driver
- please go through netmap's documentation for installation
instructions. We used the following command to set the compilation
scripts for netmap (for ixgbe-3.15.1 driver).
# ./configure --kernel-dir=/path/to/kernel/src --no-drivers=i40e,virtio_net.c
- To run mTCP clients correctly, you need to modify the RSS
seed in ixgbe_main.c:ixgbe_setup_mrqc() function. Our mTCP stack
uses a specific RSS seed (mentioned below).
- seed[10] should be reset to {
0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505
};
- Make sure that the underlying kernel module is correctly
working. You can use sample applications to validate your
setup.
# make
# sudo insmod ./netmap.ko
# sudo insmod ./ixgbe/ixgbe.ko
FOR -B-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding ixgbe-5.3.7/ driver
- please go through netmap's documentaiton for installation
instructions. We used the following command(s) to compile the driver
# cd LINUX; ./configure; make
- To run mTCP clients correctly, you need to modify the RSS seed in
ixgbe_main.c:ixgbe_init_rss_key() function. Our mTCP stack uses a specific
RSS seed (mentioned below).
static inline int ixgbe_init_rss_key(struct ixgbe_adapter *adapter)
{
#if 0
u32 *rss_key;
if (!adapter->rss_key) {
rss_key = kzalloc(IXGBE_RSS_KEY_SIZE, GFP_KERNEL);
if (unlikely(!rss_key))
return -ENOMEM;
netdev_rss_key_fill(rss_key, IXGBE_RSS_KEY_SIZE);
adapter->rss_key = rss_key;
}
#endif
static uint32_t sym_rss_key[10] = {
0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505, 0x05050505,
0x05050505, 0x05050505, 0x05050505
};
if (!adapter->rss_key)
adapter->rss_key = sym_rss_key;
return 0;
}
- Make sure that the underlying kernel module is correctly
working. You can use sample applications to validate your
setup.
# make
# sudo insmod ./netmap.ko
# sudo insmod ./ixgbe/ixgbe.ko
For -C-
--------------------------------------------------------------------------------------------------
1. Install the netmap driver and the corresponding i40e-2.4.6/ driver
- please go through netmap's documentation for installation
instructions. We used the following command(s) to compile the driver
# cd LINIX; ./configure; make
- To run mTCP clients correctly, you need to modify the RSS seed in
i40e_main.c:i40e_config_rss_aq() & i40e_main.c:i40e_config_rss_reg()
functions. Our mTCP stack uses a specific RSS seed (mentioned below).
static unsigned char sym_seed[] = {
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05, 0x05,
0x05, 0x05
};
static int i40e_config_rss_aq(struct i40e_vsi *vsi, const u8 *seed,
u8 *lut, u16 lut_size)
{
struct i40e_pf *pf = vsi->back;
struct i40e_hw *hw = &pf->hw;
int ret = 0;
seed = sym_seed;
...
}
static int i40e_config_rss_aq(struct i40e_vsi *vsi, const u8 *seed,
u8 *lut, u16 lut_size)
{
struct i40e_pf *pf = vsi->back;
struct i40e_hw *hw = &pf->hw;
int ret = 0;
seed = sym_seed;
...
}
- Make sure that the underlying kernel module is correctly
working. You can use sample applications to validate your
setup.
# make
# sudo insmod ./netmap.ko
# sudo insmod ./i40e/i40e.ko
FOR BOTH -A- & -B- & -C- (CONTINUED)
--------------------------------------------------------------------------------------------------
2. For optimum performance you are suggested to bind NICS IRQs to arbitrary
CPUs. Please use affinity-netmap.py script for this purpose. The current
script is setup for the netmap ixgbe-5.3.7 driver. Please use a variant
of this file for other cases (igb, i40e etc.).
# ./config/affinity-netmap.py ${IFACE}
- Disable flow control in Ethernet layer
# sudo ethtool -A ${IFACE} rx off
# sudo ethtool -A ${IFACE} tx off
- Disable lro (large receive offload) in Ethernet device. mTCP
does not support large packet sizes (> 1514B) yet)
# sudo ethtool -K ${IFACE} lro off
# sudo ethtool -K ${IFACE} gro off
- We used example/pktgen to test netmap raw network I/O
performance. Netmap's pktgen can be used not only for
packet generation but also packet reception. Since mTCP
relies on RSS-based NIC hardware queues, we recommend using
the following command-line arguments to test pkt-gen as a
sink before testing mTCP for netmap.
-SINK- (assuming the machine has 4 cpus)
# sudo ./pkt-gen -i ${IFACE}-0 -f rx -c 1 -a 0 -b 64 &
# sudo ./pkt-gen -i ${IFACE}-1 -f rx -c 1 -a 1 -b 64 &
# sudo ./pkt-gen -i ${IFACE}-2 -f rx -c 1 -a 2 -b 64 &
# sudo ./pkt-gen -i ${IFACE}-3 -f rx -c 1 -a 3 -b 64 &
where ${IFACE} is netmap-enabled interface.
The netmap README file shows a concise description on how to
use the driver. We reiterate some points that are essential in
understanding the command line arguments above. An interface
name post-appended with a number means that the process will
read traffic from the specified NIC hardware queue. `-a` argument
lets the program bind to a specific core.
3. Setup mtcp library:
# ./configure --enable-netmap
# make
- By default, mTCP assumes that there are 16 CPUs in your system.
You can set the CPU limit, e.g. on a 32-core system, by using the following command:
# ./configure --enable-netmap CFLAGS="-DMAX_CPUS=32"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case `./configure' script prints an error, run the
following command; and then re-do step-2 (configure again):
# autoreconf -ivf
- check libmtcp.a in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
4. Check the configurations in apps/example
- epserver.conf for server-side configuration
- epwget.conf for client-side configuration
- you may write your own configuration file for your application
5. Run the applications. *If you run the application with one thread,
mTCP core will assume that the multi-queues option is disabled. This
assumption is only valid for netmap version.*
6. Netmap module (mtcp/src/netmap_module.c) by default uses blocking
I/O by default. Most microbenchmarking applications (epserver/epwget)
shows best performance with this setup in our testbed. In case the
performance is sub-optimal in yours, we recommend that you try polling
mode (by enabling CONST_POLLING in line 24). You can also try tweaking
IDLE_POLL_WAIT/IDLE_POLL_COUNT macros while testing blocking mode I/O.