in-kernel PM: Server is only using single IP to establish subflows

[roman-shuhov]

Pre-requisites

• A similar issue has not been reported before.
• mptcp.dev website does not cover my case.
• An up-to-date kernel is being used.
• This case is not fixed with the latest stable (or LTS) version listed on kernel.org

What did you do?

I'm trying to make a simple MPTCP connection between 2 hosts, each of these machines has 4 physical network cards (eth0..eth3) with unique IPv6/128 subnets.

# zcat /proc/config.gz | grep -i mptcp
CONFIG_MPTCP=y
CONFIG_INET_MPTCP_DIAG=m
CONFIG_MPTCP_IPV6=y

# sysctl -a | grep mptcp
net.ipv4.tcp_available_ulp = mptcp tls
net.mptcp.add_addr_timeout = 120
net.mptcp.allow_join_initial_addr_port = 1
net.mptcp.checksum_enabled = 0
net.mptcp.close_timeout = 60
net.mptcp.enabled = 1
net.mptcp.pm_type = 0
net.mptcp.scheduler = default
net.mptcp.stale_loss_cnt = 4

To make a connection I'm using a simple python client/server scripts. Server is listening on :8080 (from eth3) with MPTCP socket and client connect from :8086 (ports are arbitrary in this case). Each host also has configured list of endpoints (as a side note, I tried different configurations of subflows with and without fullmesh, but final behavior was the same):

[server] # ip mptcp endpoint
<srv-ip1> id 1 signal subflow dev eth0 
<srv-ip2> id 2 subflow fullmesh dev eth1 
<srv-ip3> id 3 subflow fullmesh dev eth2 
<srv-ip4> id 4 subflow fullmesh dev eth3 

[client] # ip mptcp endpoint
<cl-ip1> id 1 signal subflow dev eth0 
<cl-ip2> id 2 subflow fullmesh dev eth1 
<cl-ip3> id 3 subflow fullmesh dev eth2 
<cl-ip4> id 4 subflow fullmesh dev eth3

ss shows the following connections/subflows:

[server] # ss -tunlap | grep python tcp   
LISTEN     0      5   [<srv-ip4>]:8080  [::]:*     users:(("python3.12",pid=1678010,fd=3)) tcp   
ESTAB      0      0   [<srv-ip4>]:8080  [<cl-ip2>]:8086  users:(("python3.12",pid=1678010,fd=4)) tcp   
ESTAB      0      0   [<srv-ip4>]:8080  [<cl-ip1>]:43913 users:(("python3.12",pid=1678010,fd=4)) tcp  
ESTAB      0      0   [<srv-ip4>]:8080  [<cl-ip4]:34685 users:(("python3.12",pid=1678010,fd=4))
# ss -Mn
State       Recv-Q        Send-Q                          Local Address:Port                           Peer Address:Port
ESTAB       0             0                    [<srv-ip4>]:8080               [<cli-ip4>]:8087

The odd thing in this case, is that server is using the same <srv-ip4> in all subflows, even as fullmesh and several endpoints are configured. From my understanding server should use different endpoints in this case.

From the mptcp monitor, on server it shows that single IP was used for server side:

[server] # ip mptcp monitor
[LISTENER_CREATED] saddr6=<srv-ip4> sport=8080
[         CREATED] token=f2516375 remid=0 locid=0 saddr6=<srv-ip4> daddr6=<cl-ip4> sport=8080 dport=8087
[     ESTABLISHED] token=f2516375 remid=0 locid=0 saddr6=<srv-ip4> daddr6=<cl-ip4> sport=8080 dport=8087
[       ANNOUNCED] token=f2516375 remid=1 daddr6=<cl-ip1> dport=8087
[  SF_ESTABLISHED] token=f2516375 remid=4 locid=0 saddr6=<srv-ip4> daddr6=<cl-ip2> sport=8080 dport=42225 backup=0
[  SF_ESTABLISHED] token=f2516375 remid=2 locid=0 saddr6=<srv-ip4> daddr6=<cl-ip3> sport=8080 dport=37795 backup=0
[          CLOSED] token=f2516375
[ LISTENER_CLOSED] saddr6=<srv-ip4> sport=8080

[client] # ip mptcp monitor 
[         CREATED] token=8e8cde69 remid=0 locid=0 saddr6=<cl-ip4> daddr6=<srv-ip4> sport=8087 dport=8080
[     ESTABLISHED] token=8e8cde69 remid=0 locid=0 saddr6=<cl-ip4> daddr6=<srv-ip4> sport=8087 dport=8080
[       ANNOUNCED] token=8e8cde69 remid=1 daddr6=<srv-ip1> dport=8080
[       SF_CLOSED] token=8e8cde69 remid=1 locid=4 saddr6=<cl-ip1> daddr6=<srv-ip1> sport=38413 dport=8080 backup=0 ifindex=3
[  SF_ESTABLISHED] token=8e8cde69 remid=0 locid=4 saddr6=<cl-ip1> daddr6=<srv-ip4> sport=42225 dport=8080 backup=0 ifindex=3
[  SF_ESTABLISHED] token=8e8cde69 remid=0 locid=2 saddr6=<cl-ip2> daddr6=<srv-ip4> sport=37795 dport=8080 backup=0 ifindex=10
[       SF_CLOSED] token=8e8cde69 remid=1 locid=2 saddr6=<cl-ip2> daddr6=<srv-ip1> sport=60757 dport=8080 backup=0 ifindex=10
[       SF_CLOSED] token=8e8cde69 remid=1 locid=3 saddr6=<cl-ip4> daddr6=<srv-ip1> sport=45561 dport=8080 backup=0 ifindex=4
[          CLOSED] token=8e8cde69

But on closer look with bpftrace, I found that server is actually trying to use "local" ip addresses, all 3 and get EINVAL error, coming from this line https://elixir.bootlin.com/linux/v6.9/source/net/mptcp/subflow.c#L1663 (i'm using 6.9 kernel):

[server] # 
14:59:04.557572 -> 14:59:04.557585 TID/PID 837141/837141 (kworker/23:1/kworker/23:1):

                    ret_from_fork_asm+0x11                        (arch/x86/entry/entry_64.S:257)
                    ret_from_fork+0x2f                            (arch/x86/kernel/process.c:153)
                    kthread+0xae                                  (kernel/kthread.c:389)
                    worker_thread+0xc6                            (kernel/workqueue.c:3430)
                    . keep_working                                (kernel/workqueue.c:955)
                    . list_empty                                  (include/linux/list.h:373)
                    process_scheduled_works+0x184                 (kernel/workqueue.c:3348)
                    . process_one_work                            (kernel/workqueue.c:3272)
                    mptcp_worker+0x4d                             (net/mptcp/protocol.c:2745)
                    mptcp_pm_nl_work+0x1ed                        (net/mptcp/pm_netlink.c:874)
                    mptcp_pm_create_subflow_or_signal_addr+0x2b0  (net/mptcp/pm_netlink.c:600)
                    __mptcp_subflow_connect+0x55                  (net/mptcp/subflow.c:1537)
!    3us [-EINVAL]  mptcp_subflow_create_socket

Do you think it's the right expectation that server should use different source IP addresses? Do you think there any obvious errors with my setup?

Any help/advice is highly appreciated!

What happened?

all subflows are established with the same server-ip, even if I configure multiple subflow endpoints on server (tried with and without fullmesh). Server also reports EINVAL errors on attempt to use local addresses.

What did you expect to have?

expect server to pick multiple addresses from configured endpoints

System info: Client

6.9

System info: Server

6.9

Additional context

[server] #
14:59:04.557572 -> 14:59:04.557585 TID/PID 837141/837141 (kworker/23:1/kworker/23:1):

                ret_from_fork_asm+0x11                        (arch/x86/entry/entry_64.S:257)
                ret_from_fork+0x2f                            (arch/x86/kernel/process.c:153)
                kthread+0xae                                  (kernel/kthread.c:389)
                worker_thread+0xc6                            (kernel/workqueue.c:3430)
                . keep_working                                (kernel/workqueue.c:955)
                . list_empty                                  (include/linux/list.h:373)
                process_scheduled_works+0x184                 (kernel/workqueue.c:3348)
                . process_one_work                            (kernel/workqueue.c:3272)
                mptcp_worker+0x4d                             (net/mptcp/protocol.c:2745)
                mptcp_pm_nl_work+0x1ed                        (net/mptcp/pm_netlink.c:874)
                mptcp_pm_create_subflow_or_signal_addr+0x2b0  (net/mptcp/pm_netlink.c:600)
                __mptcp_subflow_connect+0x55                  (net/mptcp/subflow.c:1537)

! 3us [-EINVAL] mptcp_subflow_create_socket

see this error for each local endpoint server tries to use and it still somehow fallbacks to single IP address.

[matttbe]

Hi @roman-shuhov,

Thank you for this bug report!

I had a very quick look (I will try to check in more details later on, but it might be in a week or two), and I have a couple of comments:

• In your setup with 4 endpoints on each side, which paths do you expect to have?
• Typically, it is the client who initiates the subflows, and the server who announces/signals extra addresses. In this case, the client endpoints will all have the subflow flag, and the server ones will all have the signal flag. (See our website for more details.) In your case, you have a mix of both. Is it intentional?
• What are the MPTCP limits on your side? ip mptcp limit
• And the routing rules (ip route and maybe ip rule and ip route show table X if you are using rules).
• Could you eventually try with a more recent kernel? I sent quite a few fixes related to the PM last year, around June-July. They have landed in the "vanilla" stable versions in August. I see that the last v6.9.x version (v6.9.12) has been released in July: I don't know if the version you use have all the fixes.
• If your goal is to have one path per endpoint (so 4 paths in total, one per interface), you might want to have the server signalling all addresses (signal flag), and have specific routes per pairs of interfaces, e.g. having specific routing rules to reach the server's eth3 interface from the client's eth3 one instead of using the default route via eth0 I suppose (or have the latest dev kernel with in-kernel PM: use endpoints only to create subflows with received ADD_ADDR (laminar flag) #503)

[roman-shuhov]

Adding answers inline:

• In your setup with 4 endpoints on each side, which paths do you expect to have?
  overall, the goal is to have 4 subflows established between client and server in any combination, e.g.:

C(a) <-> S(d)
C(b) <-> S(c)
C(c) <-> S(b)
C(d) <-> S(a)

• Typically, it is the client who initiates the subflows, and the server who announces/signals extra addresses .... In your case, you have a mix of both. Is it intentional?
  yes, it's intentional, as each of the machines can take role of client or server
• What are the MPTCP limits on your side? ip mptcp limit
  add_addr_accepted 8 subflows 8
• And the routing rules (ip route and maybe ip rule and ip route show table X if you are using rules).
  there is basically <ip-1..4> dev eth{0..3} and default from <ip-1..4> via fe80::1 dev eth{0..3} and all interfaces are accesible between each other.
• Could you eventually try with a more recent kernel?
  going to try it today or tomorrow, will comment with results.
• If your goal is to have one path per endpoint (so 4 paths in total, one per interface), you might want to have the server signalling all addresses (signal flag), and have specific routes per pairs of interfaces...
  in this case, all interfaces (and also server/client roles) are interchangeable, will try dev kernel with these changes

Hope i added necessary info, let me know if some details are still confusing. I'll add separate comment about latest kernel and changes in dev kernel.

[matttbe]

Adding answers inline:

Thanks!

* **In your setup with 4 endpoints on each side, which paths do you expect to have?**
  overall, the goal is to have 4 subflows established between client and server in any combination, e.g.:

C(a) <-> S(d)
C(b) <-> S(c)
C(c) <-> S(b)
C(d) <-> S(a)

OK, one path per endpoint. (Then #503 would help, but it will only be available from the future v6.18)

* **Typically, it is the client who initiates the subflows, and the server who announces/signals extra addresses .... In your case, you have a mix of both. Is it intentional?**
  yes, it's intentional, as each of the machines can take role of client or server

OK, I see. Would it be practical for you to use different net namespaces? One the client apps, and one for the server apps? Or is it the same application that takes the role of client and server?

For the moment, if you set both subflow and signal flags, it means the same endpoint will be used to send an ADD_ADDR (announcing an extra address), and establish a new subflow from this endpoint to the other peer's initial IP address and port (e.g. C([b-d]) <-> S(a)). That's maybe not what you want?

Note: it might be good to add an option to use signal or subflow depending on the type of application (listener or connector). See: #590.

* **What are the MPTCP limits on your side? ip mptcp limit**
  `add_addr_accepted 8 subflows 8`

OK, that's enough (not for fullmesh in 4x4, but that's not what you want)

* **And the routing rules (ip route and maybe ip rule and ip route show table X if you are using rules).**
  there is basically `<ip-1..4> dev eth{0..3}` and `default from <ip-1..4> via fe80::1 dev eth{0..3}` and all interfaces are accesible between each other.

In your case, without #503, it might help to:

• set no endpoint on one side (e.g. client side), and set the add_addr_accepted limit to >= 3.
• set signal endpoints on the other side (e.g. server side)
• add specific routes, e.g. to reach S(b), the client will use C(b), etc.

With this, one peer (server) will announce its extra addresses (S[b-d]), and the client will react by creating paths to these extra addresses. The last point is needed, because (without #503), the in-kernel PM will rely on the routing tables to pick the source IP address. With a specific route, when the client will receive S(b) address, it will use C(b) as source address instead of C(a). A bit of a workaround, but that might be OK for you if you cannot use #503.

With #503, one peer can simply have endpoints with the new laminar flag, while the other one has endpoints with the signal one.

* **Could you eventually try with a more recent kernel?**
  going to try it today or tomorrow, will comment with results.

If what I suggested above doesn't work, it might help to try with a more recent kernel (or at least one including the fixes I sent in July last year).

* **If your goal is to have one path per endpoint (so 4 paths in total, one per interface), you might want to have the server signalling all addresses (signal flag), and have specific routes per pairs of interfaces...**
  in this case, all interfaces (and also server/client roles) are interchangeable, will try dev kernel with these changes

Yes, but it seems still better to have only one side announcing, and one side reacting, not to create the same path twice from both sides.

Hope i added necessary info, let me know if some details are still confusing. I'll add separate comment about latest kernel and changes in dev kernel.

All good!

[roman-shuhov]

unfortunately, not an option in our case to "add specific routes" or control application in any way. In this case it's a set of ephemeral machines with multiple network interfaces and an idea to provide transparent way to use full bandwidth for the applications running inside. Still have latest kernel and pointed #503 to try, will comment once i have an update.

wanted to discuss this point in more detail: "in-kernel PM will rely on the routing tables to pick the source IP address". Per my understanding (please feel free to point any mistakes) PM tries to collect local addresses [1] (e.g. configured endpoints with subflow/fullmesh tags) and tries to use them as a source IP, before fall back to a default address. And one of the observations was a stacktrace with a failure to use them (on this line [2]), do you think it indicates some issue with that logic?

[1] https://elixir.bootlin.com/linux/v6.9/source/net/mptcp/pm_netlink.c#L581
[2] https://elixir.bootlin.com/linux/v6.9/source/net/mptcp/subflow.c#L1663

[matttbe]

unfortunately, not an option in our case to "add specific routes" or control application in any way. In this case it's a set of ephemeral machines with multiple network interfaces and an idea to provide transparent way to use full bandwidth for the applications running inside. Still have latest kernel and pointed #503 to try, will comment once i have an update.

I understand. As a workaround, you could load balance the default routes by setting multiple interfaces as nexthops with the same weight, or rely on netfilter (or similar).

Or use the userspace PM. (We are working on a BPF version, but not ready yet)

wanted to discuss this point in more detail: "in-kernel PM will rely on the routing tables to pick the source IP address". Per my understanding (please feel free to point any mistakes) PM tries to collect local addresses [1] (e.g. configured endpoints with subflow/fullmesh tags) and tries to use them as a source IP, before fall back to a default address. And one of the observations was a stacktrace with a failure to use them (on this line [2]), do you think it indicates some issue with that logic?

[1] https://elixir.bootlin.com/linux/v6.9/source/net/mptcp/pm_netlink.c#L581
[2] https://elixir.bootlin.com/linux/v6.9/source/net/mptcp/subflow.c#L1663

Maybe better to check the latest version of the code, I think it is easier to understand the logic, which has not fundamentally changed since to v6.9.

There are 3 interesting main parts: announcing addresses (signal), the creation of subflows (subflow), and reacting when ADD_ADDR are received.

• Signalling: In mptcp_pm_create_subflow_or_signal_addr(), we have this code:
  

  mptcp_net-next/net/mptcp/pm_kernel.c

  Lines 337 to 370 in 1660e82

  	/* check first for announce */
  	if (msk->pm.add_addr_signaled < endp_signal_max) {
  	/* due to racing events on both ends we can reach here while
  	* previous add address is still running: if we invoke now
  	* mptcp_pm_announce_addr(), that will fail and the
  	* corresponding id will be marked as used.
  	* Instead let the PM machinery reschedule us when the
  	* current address announce will be completed.
  	*/
  	if (msk->pm.addr_signal & BIT(MPTCP_ADD_ADDR_SIGNAL))
  	return;
  	if (!select_signal_address(pernet, msk, &local))
  	goto subflow;
  	/* If the alloc fails, we are on memory pressure, not worth
  	* continuing, and trying to create subflows.
  	*/
  	if (!mptcp_pm_alloc_anno_list(msk, &local.addr))
  	return;
  	__clear_bit(local.addr.id, msk->pm.id_avail_bitmap);
  	msk->pm.add_addr_signaled++;
  	/* Special case for ID0: set the correct ID */
  	if (local.addr.id == msk->mpc_endpoint_id)
  	local.addr.id = 0;
  	mptcp_pm_announce_addr(msk, &local.addr, false);
  	mptcp_pm_addr_send_ack(msk);
  	if (local.flags & MPTCP_PM_ADDR_FLAG_SUBFLOW)
  	signal_and_subflow = true;
  	}

  
  It will select one endpoint with the signal flag in select_signal_address() (and check the other ones later)
• Subflow: Also in mptcp_pm_create_subflow_or_signal_addr(), we have this code:
  

  mptcp_net-next/net/mptcp/pm_kernel.c

  Lines 373 to 403 in 1660e82

  	/* check if should create a new subflow */
  	while (msk->pm.local_addr_used < endp_subflow_max &&
  	msk->pm.extra_subflows < limit_extra_subflows) {
  	struct mptcp_addr_info addrs[MPTCP_PM_ADDR_MAX];
  	bool fullmesh;
  	int i, nr;
  	if (signal_and_subflow)
  	signal_and_subflow = false;
  	else if (!select_local_address(pernet, msk, &local))
  	break;
  	fullmesh = !!(local.flags & MPTCP_PM_ADDR_FLAG_FULLMESH);
  	__clear_bit(local.addr.id, msk->pm.id_avail_bitmap);
  	/* Special case for ID0: set the correct ID */
  	if (local.addr.id == msk->mpc_endpoint_id)
  	local.addr.id = 0;
  	else /* local_addr_used is not decr for ID 0 */
  	msk->pm.local_addr_used++;
  	nr = fill_remote_addresses_vec(msk, &local.addr, fullmesh, addrs);
  	if (nr == 0)
  	continue;
  	spin_unlock_bh(&msk->pm.lock);
  	for (i = 0; i < nr; i++)
  	__mptcp_subflow_connect(sk, &local, &addrs[i]);
  	spin_lock_bh(&msk->pm.lock);
  	}

  
  Similarly, it will select one endpoint with the subflow flag in select_local_address() (and check the other ones later). If the fullmesh flag is not set, it will pick the other host's initial IP address and port as destination (not what you want). If fullmesh is set, it will create subflows from this endpoint to all known and used remote addresses (too much in your case):
  

  mptcp_net-next/net/mptcp/pm_kernel.c

  Lines 240 to 252 in 1660e82

  	static unsigned int
  	fill_remote_addresses_vec(struct mptcp_sock *msk, struct mptcp_addr_info *local,
  	bool fullmesh, struct mptcp_addr_info *addrs)
  	{
  	/* Non-fullmesh: fill in the single entry corresponding to the primary
  	* MPC subflow remote address, and return 1, corresponding to 1 entry.
  	*/
  	if (!fullmesh)
  	return fill_remote_addr(msk, local, addrs);
  	/* Fullmesh endpoint: fill all possible remote addresses */
  	return fill_remote_addresses_fullmesh(msk, local, addrs);
  	}

• received ADD_ADDR: the logic is in fill_local_addresses_vec():
  

  mptcp_net-next/net/mptcp/pm_kernel.c

  Lines 589 to 616 in 1660e82

  	/* Fill all the local addresses into the array addrs[],
  	* and return the array size.
  	*/
  	static unsigned int
  	fill_local_addresses_vec(struct mptcp_sock *msk, struct mptcp_addr_info *remote,
  	struct mptcp_pm_local *locals)
  	{
  	bool c_flag_case = remote->id && mptcp_pm_add_addr_c_flag_case(msk);
  	int i;
  	/* If there is at least one MPTCP endpoint with a fullmesh flag */
  	i = fill_local_addresses_vec_fullmesh(msk, remote, locals, c_flag_case);
  	if (i)
  	return i;
  	/* If there is at least one MPTCP endpoint with a laminar flag */
  	if (mptcp_pm_get_endp_laminar_max(msk))
  	return fill_local_laminar_endp(msk, remote, locals);
  	/* Special case: peer sets the C flag, accept one ADD_ADDR if default
  	* limits are used -- accepting no ADD_ADDR -- and use subflow endpoints
  	*/
  	if (c_flag_case)
  	return fill_local_addresses_vec_c_flag(msk, remote, locals);
  	/* No special case: fill in the single 'IPADDRANY' local address */
  	return fill_local_address_any(msk, remote, &locals[0]);
  	}

  
  If there are MPTCP endpoints with a fullmesh flag, they will all be picked to create subflows to the remote address and port announced by the other peer. If there are no fullmesh, but laminar endpoints, only one will be used (what you want I guess). Then there is a C-flag exception (not interesting for you, I guess), then relying on the routing rules.

Does it reply to your questions? :)

[roman-shuhov]

So far i tried latest kernel 6.17 and also laminar endpoints (merged mptcp_net-next on top of "vanilla" 6.17), but in both cases still see the same behavior. Maybe I'm missing something, let me provide a detailed report for test with laminar endpoints, feel free to point any mistakes.

Test Setup

2 machines (S1 and S2) with single NIC and 4 IPv6 addresses, all addresses are reachable (only had single NIC machine in this case, let me know if it makes sense to run with multiple NICs). Both machines has 6.17.0-08633-g82ebea5d2683 kernel with latest mptcp changes and the next settings are same on both machines:

# ifconfig eth0
flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
...
        inet6 <ip1>  prefixlen 128  scopeid 0x0<global>
        inet6 <ip2>  prefixlen 64  scopeid 0x0<global>
        inet6 <ip3>  prefixlen 128  scopeid 0x0<global>
        inet6 <ip4>  prefixlen 128  scopeid 0x0<global>
...

Routing table looks like this (with some unrelated routes removed):

<ip2> dev eth0 proto kernel metric 256 pref medium
<ip3> dev eth0 proto kernel metric 256 pref medium
<ip4> dev eth0 proto kernel metric 256 pref medium
<server-64-major-bits>::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::face:b00c dev eth0 proto static metric 1 pref medium

Setup 4 endpoints on each machine, use laminar+subflow+signal labels for each of them (let me know if it should make any difference to separate them):

# ip mptcp endpoint
<ip1> id 1 signal subflow rawflags 20 dev eth0
<ip2> id 2 signal subflow rawflags 20 dev eth0
<ip3> id 3 signal subflow rawflags 20 dev eth0
<ip4> id 4 signal subflow rawflags 20 dev eth0

Limits are set with "8":

# ip mptcp limit
add_addr_accepted 8 subflows 8

Compiled iperf3 with mptcp support (generally available since esnet/iperf#1661)

# iperf3 -v
iperf 3.19.1+ (cJSON 1.7.15)

Both machines also running mptcp monitor command

Test execution

On S1 running iperf server with (-m uses MPTCP):

# iperf3 -m -s -B <s1-ip1>
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
...

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-60.00  sec   159 GBytes  22.8 Gbits/sec                  receiver

On S2 running client with:

# iperf3 -m -B <s2-ip1> -c <s1-ip1> -t 60
...
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-60.00  sec   159 GBytes  22.8 Gbits/sec  114            sender
[  5]   0.00-60.00  sec   159 GBytes  22.8 Gbits/sec                  receiver

MPTCP Monitor on S1 (iperf3 server):

[LISTENER_CREATED] saddr6=<s1-ip1> sport=5201
[         CREATED] token=f2532ae9 remid=0 locid=0 saddr6=<s1-ip1> daddr6=<s2-ip1> sport=5201 dport=44761 flags=2
[     ESTABLISHED] token=f2532ae9 remid=0 locid=0 saddr6=<s1-ip1> daddr6=<s2-ip1> sport=5201 dport=44761 flags=2
[       ANNOUNCED] token=f2532ae9 remid=3 daddr6=<s2-ip2> dport=44761
[  SF_ESTABLISHED] token=f2532ae9 remid=4 locid=0 saddr6=<s1-ip1> daddr6=<s2-ip3> sport=5201 dport=46265 backup=0
[  SF_ESTABLISHED] token=f2532ae9 remid=3 locid=0 saddr6=<s1-ip1> daddr6=<s2-ip2> sport=5201 dport=44629 backup=0
[       SF_CLOSED] token=f2532ae9 remid=3 locid=2 saddr6=<s1-ip2> (??) daddr6=<s2-ip2> sport=63141 dport=44761 backup=0 ifindex=2
[          CLOSED] token=f2532ae9
[ LISTENER_CLOSED] saddr6=<s1-ip1> sport=5201

MPTCP Monitor on S2 (iperf client):

[         CREATED] token=a7c4f7f4 remid=0 locid=0 saddr6=<s2-ip1> daddr6=<s1-ip1> sport=44761 dport=5201
[     ESTABLISHED] token=a7c4f7f4 remid=0 locid=0 saddr6=<s2-ip1> daddr6=<s1-ip1> sport=44761 dport=5201
[       ANNOUNCED] token=a7c4f7f4 remid=2 daddr6=<s1-ip2> dport=5201
[       SF_CLOSED] token=a7c4f7f4 remid=2 locid=1 saddr6=<s2-ip4> daddr6=<s1-ip2> sport=42679 dport=5201 backup=0 ifindex=2
[  SF_ESTABLISHED] token=a7c4f7f4 remid=0 locid=4 saddr6=<s2-ip3> daddr6=<s1-ip1> sport=46265 dport=5201 backup=0 ifindex=2
[  SF_ESTABLISHED] token=a7c4f7f4 remid=0 locid=3 saddr6=<s2-ip2> daddr6=<s1-ip1> sport=44629 dport=5201 backup=0 ifindex=2

tried to use different ips, e.g. s1-ip2 with s2-ip2, but still the same behavior.

So somehow, even as laminar endpoints are set, server is still only using s1-ip1 to create subflows. The expected behavior per my understanding is for server to use separate s1-ip* addresses for subflows. Does it sound right for your, do you see any mistakes with this setup?

P.S. From the later execution, here is also "ss ..." output of active connections during iperf3 execution

On S1

# ss -Mtunlap | grep iperf
tcp   LISTEN    0      4096             [<s1-ip1>]:5201                                      [::]:*     users:(("iperf3",pid=119705,fd=3))
tcp   ESTAB     0      0                [<s1-ip1>]:5201                  [<s2-ip3>]:64511 users:(("iperf3",pid=119705,fd=5))
tcp   ESTAB     0      0                [<s1-ip1>]:5201                  [<s2-ip1>]:38009 users:(("iperf3",pid=119705,fd=5))
tcp   ESTAB     0      0                [<s1-ip1>]:5201                  [<s2-ip1>]:44945 users:(("iperf3",pid=119705,fd=4))
tcp   ESTAB     0      0                [<s1-ip1>]:5201                  [<s2-ip4>]:62551 users:(("iperf3",pid=119705,fd=5))
tcp   ESTAB     0      0                [<s1-ip1>]:5201                  [<s2-ip2>]:37615 users:(("iperf3",pid=119705,fd=5))
mptcp ESTAB     679488 0                [<s1-ip1>]:5201                  [<s2-ip1>]:38009 users:(("iperf3",pid=119705,fd=5))
mptcp LISTEN    0      4096             [<s1-ip1>]:5201                                         *:*     users:(("iperf3",pid=119705,fd=3))

On S2

# ss -Mtunlap | grep iperf
tcp   ESTAB     0      342836     [<s2-ip3>]%eth0:64511                 [<s1-ip1>]:5201  users:(("iperf3",pid=2256452,fd=5))
tcp   ESTAB     0      471504     [<s2-ip4>]%eth0:62551                 [<s1-ip1>]:5201  users:(("iperf3",pid=2256452,fd=5))
tcp   ESTAB     0      751680     [<s2-ip2>]%eth0:37615                 [<s1-ip1>]:5201  users:(("iperf3",pid=2256452,fd=5))
tcp   ESTAB     0      781036          [<s2-ip1>]:38009                 [<s1-ip1>]:5201  users:(("iperf3",pid=2256452,fd=5))
tcp   ESTAB     0      0               [<s2-ip1>]:44945                 [<s1-ip1>]:5201  users:(("iperf3",pid=2256452,fd=4))
mptcp ESTAB     0      0               [<s2-ip1>]:38009                 [<s1-ip1>]:5201  users:(("iperf3",pid=2256452,fd=5))

[matttbe]

So far i tried latest kernel 6.17 and also laminar endpoints (merged mptcp_net-next on top of "vanilla" 6.17), but in both cases still see the same behavior. Maybe I'm missing something, let me provide a detailed report for test with laminar endpoints, feel free to point any mistakes.

@roman-shuhov: thank you for having tried with v6.17

(only had single NIC machine in this case, let me know if it makes sense to run with multiple NICs)

It is fine to use a single NIC and multiple IP addresses in this case.

Setup 4 endpoints on each machine, use laminar+subflow+signal labels for each of them (let me know if it should make any difference to separate them):

Yes, it should make a difference to separate them.

subflow will create subflows from the linked endpoint to the initial IP address of the other peer. That's not what you want.

Ideally, it would be clearer to use only signal on one side, and only laminar and the other side. If I'm not mistaken, laminar,signal on both sides should work, but I didn't try.

Compiled iperf3 with mptcp support (generally available since esnet/iperf#1661)

(note that with older versions, you could always use mptcpize run iperf3 (...))

On S1 running iperf server with (-m uses MPTCP):

# iperf3 -m -s -B <s1-ip1>

Do you need this bind on a specific IP? I have some doubts now, but I think it will affect the creation of extra subflows, because it means the subflows created to <s1-ip[2-4]> will not be allowed as there is no listening sockets accepting them.

As a workaround, it should be fine to open other listening sockets on the other IPs with the same port (or execute iperf3 without the -B, or one instance per interface).

MPTCP Monitor on S1 (iperf3 server):

[LISTENER_CREATED] saddr6=<s1-ip1> sport=5201
(...)
[       SF_CLOSED] token=f2532ae9 remid=3 locid=2 saddr6=<s1-ip2> (??) daddr6=<s2-ip2> sport=63141 dport=44761 backup=0 ifindex=2

MPTCP Monitor on S2 (iperf client):

[       SF_CLOSED] token=a7c4f7f4 remid=2 locid=1 saddr6=<s2-ip4> daddr6=<s1-ip2> sport=42679 dport=5201 backup=0 ifindex=2

Linked to what I mentioned above, it looks like the client cannot open a connection to <s1-ip2> because there is only a bind on <s1-ip1>. Is it the issue?

[roman-shuhov]

good catch, that looks like a reason, on re-run it was using different server endpoints (even without latest 6.17 kernel nor laminar endpoints).

Although, i noticed that number of subflows is not even across all listed endpoints, MPTCP consistently creates more subflows with initial serverIP, which client used to connect, e.g.:

Server setup:

<srv-ip0> dev eth0 signal
<srv-ip1> dev eth1 signal
<srv-ip2> dev eth2 signal
<srv-ip3> dev eth3 signal

There is also <srv-ip4> address setup, client using it for iperf connection to avoid any collisions with endpoint addresses.

Client setup

<cl-ip0> dev eth0 subflow
<cl-ip1> dev eth1 subflow
<cl-ip2> dev eth2 subflow
<cl-ip3> dev eth3 subflow

Start iperf server on one end: iperf3 -m -s (similar to prev example but without -B). Client is connecting as iperf3 -m -c <srv-ip4> -t 60
After running perf test, server shows the following distribution of endpoints, used as a source address for subflows

For 1 connections

# ss -tunlap | grep iperf | grep "\]:5201" | awk '{print $5}' | sort | uniq -c
      1 [<srv-ip0>]:5201
      1 [<srv-ip1>]:5201
      1 [<srv-ip2>]:5201
      1 [<srv-ip3>]:5201
      6 [<srv-ip4>]:5201

For 100 connections (-P 100 in iperf)

// first run
     98 [<srv-ip0>]:5201
    100 [<srv-ip1>]:5201
     79 [<srv-ip2>]:5201
     39 [<srv-ip3>]:5201
    500 [<srv-ip4>]:5201
// second run
     95 [<srv-ip0>]:5201
     98 [<srv-ip1>]:5201
     72 [<srv-ip2>]:5201
     40 [<srv-ip3>]:5201
    495 [<srv-ip4>]:5201

Also re-run this test with 2 endpoints on server and client <srv-ip{0.1}> and <cl-ip{0.1}> and got similar results:

For 1 connection

      1 [<srv-ip0>]:5201
      1 [<srv-ip1>]:5201
      4 [<srv-ip4>]:5201

For 100 connections

    100 [<srv-ip0>]:5201
    100 [<srv-ip1>]:5201
    301 [<srv-ip4>]:5201

Do you think this is expected behavior for MPTCP connection?

[matttbe]

good catch, that looks like a reason, on re-run it was using different server endpoints (even without latest 6.17 kernel nor laminar endpoints).

Although, i noticed that number of subflows is not even across all listed endpoints, MPTCP consistently creates more subflows with initial serverIP, which client used to connect, e.g.:

Server setup:

<srv-ip0> dev eth0 signal
<srv-ip1> dev eth1 signal
<srv-ip2> dev eth2 signal
<srv-ip3> dev eth3 signal

There is also <srv-ip4> address setup, client using it for iperf connection to avoid any collisions with endpoint addresses.

Client setup

<cl-ip0> dev eth0 subflow
<cl-ip1> dev eth1 subflow
<cl-ip2> dev eth2 subflow
<cl-ip3> dev eth3 subflow

Yes, the extra number of subflows is probably linked to what I mentioned earlier:

subflow will create subflows from the linked endpoint to the initial IP address of the other peer. That's not what you want.

Ideally, it would be clearer to use only signal on one side, and only laminar and the other side. If I'm not mistaken, laminar,signal on both sides should work, but I didn't try.

During these tests, were you using an older kernel where laminar is not available?

[roman-shuhov]

yes, right, it was a test on older kernel, i tried to configure laminar endpoints as you described and it evenly distributed subflows (iperf with 128 connections and 4 endpoints on each side):

    128 saddr6=<srv-ip1>
    128 saddr6=<srv-ip1>
    128 saddr6=<srv-ip1>
    128 saddr6=<srv-ip1>

Do you think listening on a wildcard is an expected behavior for MPTCP in this case? I understand that this is happening because there is no listening socket available for signal address, but on the other hand mptcp has all the details (source/destination ip/port) to establish it.

[matttbe]

yes, right, it was a test on older kernel, i tried to configure laminar endpoints as you described and it evenly distributed subflows (iperf with 128 connections and 4 endpoints on each side):

Nice!

On older kernels, the only way to achieve that is not to use any endpoints on that side, and to rely on routing rules. Maybe using a load balancing with nexthops having the same weight works good enough.

Do you think listening on a wildcard is an expected behavior for MPTCP in this case? I understand that this is happening because there is no listening socket available for signal address, but on the other hand mptcp has all the details (source/destination ip/port) to establish it.

Technically, a listening socket needs to accept the SYN+MP_JOIN for the MPTCP stack to get notified. You can have additional listening sockets. Did you try the suggested workarounds?

As a workaround, it should be fine to open other listening sockets on the other IPs with the same port (or execute iperf3 without the -B, or one instance per interface).

A simple Python script doing this might be enough (to be tested):

import socket
s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM, socket.IPPROTO_MPTCP)
s.bind(('::1', 5201))  ## change the IP address or bind to an interface
s.listen(0)