Lower throughput with MPTCP vs TCP

[roman-shuhov]

Pre-requisites

• A similar issue has not been reported before.
• mptcp.dev website does not cover my case.
• An up-to-date kernel is being used.
• This case is not fixed with the latest stable (or LTS) version listed on kernel.org

What did you do?

TL;DR Performance test of MPTCP on machine with 2 physical NICs (100Gb/s) shows noticeably worse throughput than single TCP connection (10.2 Gb/s vs. 43.2 Gb/s).

Servers setup

Both client and server machines are equal and connected to each other through Rack switch.

Kernel: v6.13.2
NICs: 2 Mellanox devices eth0 and eth1 with 100Gb/s speed

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 <server-eth0-prefix>::1  prefixlen 64  scopeid 0x0<global>
...
        RX errors 0  dropped 0  overruns 0  frame 0
...
        TX errors 0  dropped 4 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 <server-eth1-prefix>::1  prefixlen 64  scopeid 0x0<global>
...

MPTCP setup

Server side

# ip mptcp endpoint
<server-eth1-prefix>::1 id 1 signal dev eth1
# ip mptcp limit
add_addr_accepted 2 subflows 2

Client side

ip mptcp endpoint
<client-eth1-prefix>::1 id 1 subflow dev eth1
# ip mptcp limit
add_addr_accepted 2 subflows 2

Test setup

Using iperf3 compiled with mptcp support (esnet/iperf#1661).

MPTCP Test

[server] # iperf3 -m -s &
[server] # ip mptcp monitor
[LISTENER_CREATED] saddr6=:: sport=5201
[         CREATED] token=52bca52b remid=0 locid=0 saddr6=<server-eth0-prefix>::1 daddr6=<client-eth0-prefix>::1 sport=5201 dport=48573
[     ESTABLISHED] token=52bca52b remid=0 locid=0 saddr6=<server-eth0-prefix>::1 daddr6=<client-eth0-prefix>::1 sport=5201 dport=48573
[  SF_ESTABLISHED] token=52bca52b remid=1 locid=0 saddr6=<server-eth0-prefix>::1 daddr6=<client-eth1-prefix>::1 sport=5201 dport=58893 backup=0
[  SF_ESTABLISHED] token=52bca52b remid=2 locid=1 saddr6=<server-eth1-prefix>::1 daddr6=<client-eth1-prefix>::1 sport=5201 dport=63299 backup=0
[          CLOSED] token=52bca52b

# snapshot of NIC % util during test with `sar` tool

11:54:57 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
...
11:54:58 AM	 eth0 780104.95  15143.56 1143814.04   1598.74      0.00      0.00      0.99      9.37
11:54:58 AM	 eth1 344995.05    118.81 505897.96    160.35	   0.00      0.00      0.99	 4.14

[client] # iperf3 -c <server-eth0-prefix>::1 -t 30 -B <client-eth0-prefix>::1 -m
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  36.2 GBytes  10.4 Gbits/sec  1162            sender
[  5]   0.00-30.00  sec  36.2 GBytes  10.4 Gbits/sec                  receiver

TCP Test

[server]# iperf3 -s &

# sar tool snapshot during test run
11:54:27 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
...
11:54:28 AM	 eth0 3767797.00  29652.00 5569314.57   2644.75      0.00      0.00      0.00     45.62
11:54:28 AM	 eth1    120.00    120.00    146.75    161.95	   0.00      0.00      1.00	 0.00

[client]# iperf3 -c <server-eth0-prefix>::1 -t 30 -B <client-eth0-prefix>::1
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec   152 GBytes  43.6 Gbits/sec  3431            sender
[  5]   0.00-30.00  sec   152 GBytes  43.6 Gbits/sec                  receiver

Notes

There is another issue reported in #562 for MPTCP overhead, but might make sense to keep them separate.

What happened?

MPTCP Throughput is 10.4 Gb/s with 2 subflows and 2 NICs vs. 43.6 Gb/s TCP connection on single NIC

What did you expect to have?

Smaller overhead of MPTCP on throughput

System info: Client/Server

Stripped some info, let me know if it's necessary to add
v6.13.2
...
NAME="CentOS Stream"
VERSION="9"
...
net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.available_schedulers = default
net.mptcp.blackhole_timeout = 3600
net.mptcp.checksum_enabled = 0
net.mptcp.close_timeout = 60
net.mptcp.enabled = 1
net.mptcp.pm_type = 0
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4

Additional context

No response

[roman-shuhov]

Unable to try on latest kernel (nor with latest MPTCP features at this point), let me know if it's most likely a culprit.

Tried with 6.17 + mptcp-next on top kernel. Still observing significant difference ~30% in throughput between 2-flows MPTCP connection and simple TCP connection:

MPTCP

[server] # iperf3 -s -m &
[server] # ip mptcp monitor
[         CREATED] token=b7239a84 remid=0 locid=0 saddr6=<server-eth0-prefix>::1 daddr6=<client-eth0-prefix>::1 sport=5201 dport=49253 flags=2
[     ESTABLISHED] token=b7239a84 remid=0 locid=0 saddr6=<server-eth0-prefix>::1 daddr6=<client-eth0-prefix>::1 sport=5201 dport=49253 flags=2
[  SF_ESTABLISHED] token=b7239a84 remid=2 locid=2 saddr6=<server-eth1-prefix>::1 daddr6=<client-eth1-prefix>::1 sport=5201 dport=51233 backup=0
[          CLOSED] token=b7239a84

[server] # // sar command
01:14:42 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
01:14:43 PM        lo    631.00    631.00    484.24    484.24	   0.00      0.00      0.00	 0.00
01:14:43 PM	 eth0 1586710.00  56413.00 2329325.94   5580.58      0.00      0.00	 1.00     19.08
01:14:43 PM	 eth1 1203961.00    120.00 1765788.47    161.95      0.00      0.00	 1.00     14.47
01:14:43 PM	 eth2	   0.00      0.00      0.00      0.00	   0.00      0.00      0.00	 0.00
01:14:43 PM	 eth3    120.00    120.00    147.89    161.95	   0.00      0.00      0.00	 0.00

[client] # iperf3 -c <server-eth0-prefix>::1 -B <client-eth0-prefix>::1 -t 60 -m
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec   216 GBytes  31.0 Gbits/sec    0            sender
[  5]   0.00-60.00  sec   216 GBytes  31.0 Gbits/sec                  receiver

TCP

[server] # iperf3 -s &
[server] # //sar command during test execution
01:16:57 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
...
01:16:58 PM	 eth0 4532545.00  67942.00 6701330.77   5865.88      0.00      0.00      0.00     54.90
01:16:58 PM	 eth1    121.00    121.00    147.97    162.03      0.00      0.00      0.00      0.00
...

[client] # iperf3 -c <server-eth0-prefix>::1 -B <client-eth0-prefix>::1 -t 60
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec   311 GBytes  44.5 Gbits/sec   45            sender
[  5]   0.00-60.00  sec   311 GBytes  44.5 Gbits/sec                  receiver

[matttbe]

Hello,

Thank you for reporting this.

It would be interesting to analyse what's the limitation for both TCP and MPTCP: CPU (any visible bottlenecks?), buffers (MPTCP might require more space), NIC RX queue?

Could it be because some HW accelerations are used with TCP, but not with MPTCP? (e.g. hardware GRO?)

When multiple connections are used in parallel (e.g. iperf3 -P 48), is the difference still quite visible?

Note that there is certainly some room for improvement regarding the performances, especially in Data Centre use-case. There are different optimisations that could be applied, some probably "easy" and other more complex ones. I know some people who are interested in improving this situation. If you are too, it might be good to discuss together. Personally, I'm currently not really equipped to work on that :-/

[roman-shuhov]

Thank you for the pointers, doesn't look like CPU (no high usage visible during test), no NIC RX drops were observed.

So far something in the field of gso/gro/tso/lso looks like a suspect, but haven't caught this yet.

I also tried to abstract this experiment from hardware and create 2 network namespaces connected with each other over 2 veth devices.

here is a script to reproduce this https://gist.github.com/roman-shuhov/efbf5fd70abb466e387cb980e2cb59e4. Executed it on 6.13 kernel.

For experimentation I can also control number of flows for MPTCP with add_addr_accepted limit, if i set it to 0, no additional subflows are created, so performance should be comparable to pure TCP.

In the test i executed the following scenarios:

• single pure TCP connection
• single MPTCP conn. with no subflows
• single MPTCP conn. with 1 subflow
• single MPTCP conn. with 2 subflows
• 16 TCP conn-s
• 16 MPTCP conn-s with no subflows

In all experiments throughput with MPTCP was worse than pure TCP, which is a bit unexpected in case of multiple subflows and several veth devices.

	1 conn (Gb/s)	% diff from TCP	16 conns (Gb/s)
TCP	50.96	-	607.2
MPTCP (no subflows)	47.96	-6%	547
MPTCP (1 subflow)	46.42	-9%	515.8
MPTCP (2 subflows)	44.96	-12%	482.75

Multiple flows for TCP and MPTCP with regards to veth devices can be ignored, afau, as they most likely just bounded by CPU and total number of connections(flows) plays negative role here.

@matttbe do you think this are reasonable perf numbers for 1 connection comparison between TCP and MPTCP? I would expect them to be much closer to each other (especially for no subflow MPTCP), as there is no much overhead.

Some raw values https://gist.github.com/roman-shuhov/755babc59a750b38ee35a6887985f7c7

[cpaasch-oai]

Hi @roman-shuhov !

Is TCP in your case not CPU-limited ?

[roman-shuhov]

hey @cpaasch-oai , no CPU limits for TCP connections

[matttbe]

@roman-shuhov thank you for the experimentations!

@matttbe do you think this are reasonable perf numbers for 1 connection comparison between TCP and MPTCP? I would expect them to be much closer to each other (especially for no subflow MPTCP), as there is no much overhead.

It is hard to tell without knowing where the bottleneck is. If you are not CPU limited, could you maybe be simply limited by the congestion windows? (sysctl net.ipv4.tcp_*mem). If you can identify where the bottleneck is, that would be great to better understand the differences.

[cpaasch-oai]

hey @cpaasch-oai , no CPU limits for TCP connections

And even MPTCP is not CPU-limited, right ? In that case, can you show output of ss -itm on the sender to see what is actually limiting throughput ?

[roman-shuhov]

good point, i'm trying to understand where this bottleneck is coming from. One note, only limiting experiments to IPv6 protocol at this point.

GRO/GSO

There is gro/gso settings on the NIC with 64KB max_size. I bumped both of them to 512KB (hard limit) and it improved throughput proportionally for both TCP and MPTCP. Disabling gro makes things worse, so it doesn't look like a culprit in this case.

Memory

tried to increase net.core.[wr]mem_max all the way to 256MB, but no difference.

ss -itm

TCP

// around start of the test
ESTAB 0      2837436           [<client>]:62843                 [<server>]:5201  rto:0.2 cwnd:498 ssthresh:453 reordering:0
// around finish of the test
ESTAB 0      2215984           [<client>]:62843                 [<server>]:5201  rto:0.2 cwnd:490 ssthresh:477 reordering:0
...
[  5]   0.00-20.00  sec  99.0 GBytes  42.5 Gbits/sec                  receiver

MPTCP (no subflows)

// around start of the test
ESTAB 0      1773568           [<client>]:50085                 [<server>]:5201  rto:0.2 cwnd:447 ssthresh:439 reordering:0
// around finish of the test
ESTAB 0      2049896           [<client>]:50085                 [<server>]:5201  rto:0.2 cwnd:493 ssthresh:456 reordering:0
...
[  5]   0.00-20.00  sec  62.5 GBytes  26.9 Gbits/sec                  receiver

[matttbe]

@roman-shuhov Thank you for the experiment.

The output from ss -itm is very "light". Did you strip some info? Or do you use an old version? (I remember the v6.7 version had a bug if -p was not used, but I guess it is not that ; there was also another issues, and you needed to execute ss as root to get all info)

e.g. on my side, I have:

$ ss -itm
State        Recv-Q   Send-Q                               Local Address:Port                                             Peer Address:Port
ESTAB        0        0                                        (...):39286                                                   (...):https
	 skmem:(r0,rb131072,t0,tb87040,f0,w0,o0,bl0,d0) cubic wscale:10,10 rto:314 rtt:113.051/12.578 ato:40 mss:1424 pmtu:1500 rcvmss:1424 advmss:1448 cwnd:10 bytes_sent:9733 bytes_acked:9734 bytes_received:5213 segs_out:40 segs_in:47 data_segs_out:27 data_segs_in:27 send 1.01Mbps lastsnd:29757 lastrcv:29758 lastack:29652 pacing_rate 2.02Mbps delivery_rate 485kbps delivered:28 app_limited busy:2287ms rcv_space:5632 rcv_ssthresh:5632 minrtt:105.469 snd_wnd:100352 rcv_wnd:62464

Here, we can see: app_limited.

If you can easily reproduce the issue, can you:

• add the -M parameter (ss -itmnM)
• get the output from both the sender and receiver
• before the test, run nstat -n, then get the output of nstat | grep Tcp after the test, on both sides
• (if you have installed mptcpize / mptcpd, you might have /usr/libexec/mptcp-get-debug -r that can help you to get all that)

One last thing: do you use hardware GRO? What's the output of ethtool -k <IFACE>?

[cpaasch-oai]

GRO/GSO
There is gro/gso settings on the NIC with 64KB max_size. I bumped both of them to 512KB (hard limit) and it improved throughput proportionally for both TCP and MPTCP. Disabling gro makes things worse, so it doesn't look like a culprit in this case.

That's weird - I don't see how max_size increasing beyond 64KB can improve MPTCP, because the MPTCP data-len is 64KB.

[matttbe]

GRO/GSO
There is gro/gso settings on the NIC with 64KB max_size. I bumped both of them to 512KB (hard limit) and it improved throughput proportionally for both TCP and MPTCP. Disabling gro makes things worse, so it doesn't look like a culprit in this case.

That's weird - I don't see how max_size increasing beyond 64KB can improve MPTCP, because the MPTCP data-len is 64KB.

@roman-shuhov talking about that ↑, the limitation is in the protocol, and could be uplift in the future (see #461). But then, to be fair with MPTCP, it might be interesting to limit TCP GSO max size to 64KB:

ip link set ${IFACE} gso_max_size 65279

(on both sides)