performance: upstream and downstream will never run concurrently

[howardjohn]

copy_bidirectional uses tokio::join for the copy from upstream->downstream and vis-versa. Join is not concurrent, so its impossible for these two to happen at the same time (utilizing threads).

Intuitively it seems like this should be helpful. On a simple call-response workload, no, but if there is continuous flow of data in both directions it should.

I put together a prototype, however, and do not see any benefits:

HBONE
Master
DEST           CLIENT  QPS   CONS  DUR  PAYLOAD  SUCCESS  THROUGHPUT   P50      P90      P99
fortio-server  fortio  0     1     5    0        96743    19348.29qps  0.048ms  0.066ms  0.113ms
fortio-server  fortio  0     1     5    1024     83755    16750.66qps  0.055ms  0.076ms  0.138ms
fortio-server  fortio  2000  1     5    0        9998     1999.53qps   0.094ms  0.143ms  0.277ms
fortio-server  fortio  2000  1     5    1024     10000    1999.89qps   0.106ms  0.156ms  0.289ms
fortio-server  fortio  0     2     5    0        146377   29274.81qps  0.061ms  0.093ms  0.180ms
fortio-server  fortio  0     2     5    1024     131639   26327.32qps  0.071ms  0.096ms  0.176ms
fortio-server  fortio  2000  2     5    0        10000    1999.50qps   0.118ms  0.175ms  0.331ms
fortio-server  fortio  2000  2     5    1024     10000    1999.59qps   0.134ms  0.198ms  0.358ms
fortio-server  fortio  0     4     5    0        222310   44460.78qps  0.085ms  0.117ms  0.192ms
fortio-server  fortio  0     4     5    1024     179137   35826.45qps  0.105ms  0.148ms  0.264ms
fortio-server  fortio  2000  4     5    0        9996     1998.58qps   0.125ms  0.183ms  0.304ms
fortio-server  fortio  2000  4     5    1024     9998     1998.93qps   0.150ms  0.236ms  0.606ms
fortio-server  fortio  0     64    5    0        407907   81573.11qps  0.744ms  1.430ms  1.965ms
fortio-server  fortio  0     64    5    1024     265418   53074.42qps  1.335ms  1.896ms  2.745ms
fortio-server  fortio  2000  64    5    0        9984     1987.19qps   0.142ms  0.200ms  0.296ms
fortio-server  fortio  2000  64    5    1024     9984     1987.05qps   0.177ms  0.258ms  0.385ms
ID   Interval          Transfer      Bitrate
[  0]   0.00..10.00 sec  10.46 GiB   8.99 Gbits/sec        sender
[  0]   0.00..10.00 sec  10.41 GiB   8.94 Gbits/sec        receiver

Spawning
DEST           CLIENT  QPS   CONS  DUR  PAYLOAD  SUCCESS  THROUGHPUT   P50      P90      P99
fortio-server  fortio  0     1     5    0        95146    19028.90qps  0.048ms  0.067ms  0.112ms
fortio-server  fortio  0     1     5    1024     81596    16318.86qps  0.057ms  0.078ms  0.137ms
fortio-server  fortio  2000  1     5    0        9998     1999.40qps   0.094ms  0.139ms  0.250ms
fortio-server  fortio  2000  1     5    1024     9998     1999.54qps   0.107ms  0.154ms  0.271ms
fortio-server  fortio  0     2     5    0        156544   31306.96qps  0.060ms  0.085ms  0.138ms
fortio-server  fortio  0     2     5    1024     128630   25725.32qps  0.073ms  0.099ms  0.186ms
fortio-server  fortio  2000  2     5    0        9998     1999.36qps   0.121ms  0.179ms  0.316ms
fortio-server  fortio  2000  2     5    1024     10000    1999.42qps   0.135ms  0.200ms  0.371ms
fortio-server  fortio  0     4     5    0        218797   43758.22qps  0.086ms  0.119ms  0.196ms
fortio-server  fortio  0     4     5    1024     182502   36499.22qps  0.102ms  0.145ms  0.268ms
fortio-server  fortio  2000  4     5    0        9998     1998.75qps   0.137ms  0.235ms  0.479ms
fortio-server  fortio  2000  4     5    1024     9998     1998.85qps   0.164ms  0.287ms  0.631ms
fortio-server  fortio  0     64    5    0        400404   80069.12qps  0.755ms  1.469ms  1.973ms
fortio-server  fortio  0     64    5    1024     286835   57358.42qps  1.231ms  1.860ms  2.070ms
fortio-server  fortio  2000  64    5    0        9984     1987.07qps   0.144ms  0.212ms  0.364ms
fortio-server  fortio  2000  64    5    1024     9984     1986.99qps   0.180ms  0.258ms  0.411ms
[  0]   0.00..10.00 sec  10.77 GiB   9.25 Gbits/sec        sender
[  0]   0.00..10.00 sec  10.71 GiB   9.20 Gbits/sec        receiver



TCP
Master
DEST           CLIENT  QPS   CONS  DUR  PAYLOAD  SUCCESS  THROUGHPUT    P50      P90      P99
fortio-server  fortio  0     1     3    0        93706    31234.58qps   0.029ms  0.040ms  0.075ms
fortio-server  fortio  0     1     3    64000    18341    6113.28qps    0.138ms  0.267ms  0.395ms
fortio-server  fortio  2000  1     3    0        5998     1998.92qps    0.062ms  0.103ms  0.245ms
fortio-server  fortio  2000  1     3    64000    5998     1999.05qps    0.181ms  0.364ms  0.595ms
fortio-server  fortio  0     64    3    0        368849   122926.89qps  0.507ms  0.682ms  1.397ms
fortio-server  fortio  0     64    3    64000    43652    14530.98qps   3.361ms  9.953ms  19.560ms
fortio-server  fortio  2000  64    3    0        5952     1978.43qps    0.098ms  0.144ms  0.316ms
fortio-server  fortio  2000  64    3    64000    5952     1978.35qps    0.284ms  0.645ms  1.627ms

[SUM]   0.00..10.00 sec  25.64 GiB   22.02 Gbits/sec        sender
[SUM]   0.00..9.97 sec  27.77 GiB   23.93 Gbits/sec        receiver

Spawning
DEST           CLIENT  QPS   CONS  DUR  PAYLOAD  SUCCESS  THROUGHPUT    P50      P90      P99
fortio-server  fortio  0     1     3    0        86532    28843.36qps   0.031ms  0.044ms  0.094ms
fortio-server  fortio  0     1     3    64000    14891    4963.41qps    0.164ms  0.330ms  0.640ms
fortio-server  fortio  2000  1     3    0        5998     1998.99qps    0.064ms  0.106ms  0.196ms
fortio-server  fortio  2000  1     3    64000    6000     1999.32qps    0.185ms  0.376ms  0.678ms
fortio-server  fortio  0     64    3    0        360397   120090.70qps  0.508ms  0.698ms  1.640ms
fortio-server  fortio  0     64    3    64000    44472    14802.99qps   3.186ms  9.915ms  19.528ms
fortio-server  fortio  2000  64    3    0        5952     1978.83qps    0.093ms  0.130ms  0.192ms
fortio-server  fortio  2000  64    3    64000    5952     1978.39qps    0.273ms  0.568ms  0.919ms

[SUM]   0.00..10.00 sec  25.04 GiB   21.51 Gbits/sec        sender
[SUM]   0.00..9.96 sec  28.12 GiB   24.25 Gbits/sec        receiver

We should investigate more

[ilrudie]

Did you push the prototype code?

[bleggett]

It shouldn't be terribly hard to use spawn and join on the handles instead, or whatever, I guess, if we want real parallelism.

A little more expensive for trivial cases, but probably worth it for all the others.

[ilrudie]

I think we can collect multiple handles from tokio::spawn and then await them. I presume something like that is what @howardjohn already tried but didn't see any benefit from.

[bleggett]

I think we can collect multiple handles from tokio::spawn and then await them. I presume something like that is what @howardjohn already tried but didn't see any benefit from.

Yeah - I misread. It probably doesn't make much difference because we already spawn the per-workload handler in a thread and are not remotely CPU bound even under load - distributing this specific operation across threads won't help much (and might make it easier for a greedy workload to starve other workloads on the node).

in general sticking with a one-thread-per-conn-handler-instance model seems best.