feat: switch to QUIC multipath #3381

dignifiedquire · 2025-07-07T11:02:25Z

Work on integrating n0-computer/quinn#28 into the iroh magic

TODOs

Merge feat: add n0's version of quic nat hole punching quinn#177

github-actions · 2025-07-07T13:29:30Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/3381/docs/iroh/

Last updated: 2025-12-02T20:47:23Z

github-actions · 2025-07-07T13:37:37Z

Netsim report & logs for this PR have been generated and is available at: LOGS
This report will remain available for 3 days.

Last updated for commit: 66a8565

## Description For some odd reason AuthenticationError was given a branch that had quinn::ConnectionError inside it. But logically the connection error has nothing to do with authentication. That should have been a red flag. AuthenticationError itself is almost always wrapped in ConnectingError, which does correctly have a quinn::ConnectionError branch. And the few places where it was directly returned to the user it arguably **should** have been wrapped in a ConnectingError. The result of this is that before this fix you would get a very confusing authentication error if the remote client closed the connection right at the same time as the handshake completed for it (yes, this is difficult to do at the right time, and it only happens for the client since that completes the handshake one network hop before the server). But this was no authentication error, it is simply a closed connection. The new error structure captures this correctly. Similarly the InternalConsistencyError belongs on the ConnectingError. Though that one should be impossible to produce since it's supposed to be an invariant. ## Breaking Changes - `AuthenticationError` loses the `ConnectionError` and `InternalConsistencyError` branches. Both are on the `ConnectingError` instead. - `OutgoingZeroRttConnection::handshake_completed` and `IncomingZeroRttConnection` now return a `ConnectingError` instead of `AuthenticationError`. ## Notes & open questions While I've managed to trigger this error somewhat occasionally using a program that races the closing with the completed handshake in a very tight loop **before** applying this fix. I'm completely failing to trigger it since applying this fix, so I can admire the beautiful new error reporting this fix should give. It's a bit confusing. **edit**: it **is** confusing. But it is correct. Because my flaky failure does always close the connection *after* it completes the handshake. So ALPN and EndpointId are always available. And then you yield a valid `Connection` when awaiting an `Incoming`, it just is already closed. This fix could also be made against main. I believe the same commit should be able to be cherry-picked and will probably apply fairly clean. Do you think I should make it against main? ## Change checklist  - [x] Self-review. - [x] All breaking changes documented. - [x] List all breaking changes in the above "Breaking Changes" section.

There are issues for these already

…3664) ## Description This means these tests also work when nextest run locally. ## Breaking Changes  ## Notes & open questions I'm not sure why this wasn't done when the ci profile override was chosen. What am I missing? ## Change checklist  - [x] Self-review.

## Description Bumps netwatch and netdev, to remove duplicate dependency on both netdev@0.38 and netdev@0.39. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

This is a next step into the world of configurable transports. We now allow disabling the IP based transports entirely. Internally this starts to prepare for a world where the user can configure multiple different transports, IP, relay and others in the future. Closes #2957

## Description Remove the test-only `Endpoint::path_selection` API and instead use `Endpoint::clear_ip_transports` for `PathSelection::RelayOnly `, now that this public API was added in #3651. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

…moteState (#3673) ## Description Renames: * renamed `endpoint_map` -> `remote_map`, `EndpointMap` -> `RemoteMap`, `endpoint_state` -> `remote_state`, `EndpointStateActor` -> `RemoteStateActor` Moved: * moved `path_state` module under `remote_state` (prev `endpoint_state`), its items are used only there and nowhere else ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: Floris Bruynooghe <flub@n0.computer>

## Description Merges main and adapts for the changes from #3619 ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: Rüdiger Klaehn <rklaehn@protonmail.com> Co-authored-by: Friedel Ziegelmayer <me@dignifiedquire.com>

## Description Avoid potentially busy looping in a tokio task. I think this blocking leads to tokio not being able to close the runtime properly.

## Description Fixes #3642 This moves discovery handling fully into the `EndpointStateActor`. The pub(crate) interface to trigger discovery and get a EndpointMappedAddr is now `Magicsock::resolve_remote`, which sends the provided addresses to the EndpointStateActor. The actor starts discovery if it does not have a selected path and if discovery is not running. It returns either immediately if there are any known paths, or waits for discovery to produce at least one result or an error. Once this returns, `resolve_remote` returns either with a EndpointMappedAddr or with the discovery error. This means the current behavior is kept: We only start `quinn::Endpoint::connect` once we have at least one transport address for the remote. If not, we return the discovery error immediately from `iroh::Endpoint::connect`. This opens the door for us to easily tune when to run discovery in other siutations, e.g. when all available paths to a remote are closed. However, for now this PR still only starts discovery when `Endpoint::connect` is called and no path is selected at the moment. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

## Description * fix idle timeout clear condition (previously it would hot loop) * fix hot loop when local_addrs watchable becomes disconnected during shutdown * when sending a datagram fails in the transports sender, include the dst address in the error message * do not break the RemoteStateActor when sending a datagram fails ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: Philipp Krüger <philipp.krueger1@gmail.com>

## Description This reverts a change from this PR: #3384 I originally thought I could make this test more reliable by pausing the tokio time across the `tokio::time::timeout` calls, but it turns out that actually makes the test *more* flaky: - When time is paused, the timeout will immediately fire once the tokio runtime has no more CPU work to do. - It's possible that there's no CPU work to do anymore, while there's something else that is actually still doing work, e.g. networking. - Before the `ActiveRelayActor` finishes its `run_connected` loop, it will call `client_sink.close().await`, which will do actual I/O. When the tokio runtime is paused at that moment, it'll immediately trigger the test's timeout. ## Notes & open questions I couldn't reproduce this problem even across a couple thousand runs of the test locally. I'm not super confident that this fixes things, but I've analyzed the logs and this seems to be the most likely thing that's happening to me. Closes #3613 ## Change checklist  - [x] Self-review.

merge main

## Description This switches from the old DISCO to the so-new-it-doesnt-exit-yet QUIC NAT Traversal. ## Breaking Changes Nothing visible? Maybe? ## Notes & open questions The QUIC NAT Traversal API doesn't exist yet, so this won't even build on any machine that's not mine. I've locally patched in the dummy methods that I use. --------- Co-authored-by: dignifiedquire <me@dignifiedquire.com> Co-authored-by: Frando <frando@unbiskant.org>

## Description This adds the conecpt of hooks to the iroh endpoint. `Hooks` are structs implementing the `EndpointHooks` trait and are used to intercept the establishment of connections. Multiple hooks can be added to the endpoint, and they will be invoked for each hook in the order they have been added to the endpoint. Currently there's two methods on the `EndpointHooks` trait: * `before_connect` is invoked before an outgoing connection is started. * `after_handshake` is invoked for incoming and outgoing connections once the TLS handshake has completed Both methods return an `Outcome`, which can either be `Reject` or `Accept`. If any hook returns `Reject`, the connection or connection attempt will be rejected. The PR also adds `ConnectionInfo`, which is a struct that has information about a connection, but does not keep the connection itself alive. It allows to inspect stats and paths, and there's a `closed` method that returns a future which completes once the connection closes (without keeping the connection alive). The PR includes two examples: * `auth-hook` implements authentication for iroh protocols through a middleware and a separate authentication protocol. Individual protocols don't need to be aware of authentication at all. * `monitor-connnections` monitors incoming and outgoing connections and prints connection stats once a connection closes. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: dignifiedquire <me@dignifiedquire.com> Co-authored-by: ramfox <kasey@n0.computer>

removes the need for two `send` impls

Closes #3641

## Description Since the server was actively closing the connection it is possible that the client would not have read the response yet by the time the connection is closed. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

## Description Based on #3593 This was always just a placeholder, and can now be collected using `EndpointHooks`. ## Breaking Changes - remove `Endpoint::latency` --------- Co-authored-by: varun-doshi <doshivarun202@gmail.com>

## Description Bumps quinn to latest `main-iroh` and netwatch/portmapper to n0-computer/net-tools#72 (the latter is needed because the quinn-udp version changed to 0.6 on `main-iroh`). ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

## Description Whenever we insert a new path, trigger pruning paths. We currently only prune IP paths, and pruning paths only occurs if we have more than 30 IP paths. We will prune any paths that did not successfully holepunch. If there are still over 30 IP paths left, then we order the "inactive" paths (paths that have been closed, but at one point holepunched), and prune the paths that were closed earliest. ## Notes and Questions - Added constants: - `MAX_IP_PATHS` = 30 - maximum IP paths per endpoint - `MAX_INACTIVE_IP_PATHS` = 10 - maximum inactive IP paths to keep - New `PathState` field: - `status` - tracks the `PathStatus` of the path - New `PathStatus` enum: - `PathStatus::Open` - is an open path - `PathStatus::Inactive(Instant)` - was opened once, but currently inactive - `PathStatus::Unusable` - we attempted to use it, but it never connected - `PathStatus::Unknown` - we don't know the status yet - New methods on `RemotePathState`: - `abandoned_path` - marks a path as abandoned with timestamp, triggered when we get the `PathEvent::Abandoned` event - `prune_paths` - triggers path pruning, occurs whenever we insert a path to the `RemotePathState` - changed `insert` to `insert_open_path` - New `prune_ip_paths` function with all the prune logic: - Only prunes if IP paths exceed `MAX_IP_PATHS` - Never prunes active paths or paths of unknown status - Always prunes failed holepunch attempts (PathStatus::Unusable) - Keeps 10 most recently inactive paths that were previously successful - Special case: if all paths failed, keeps `MAX_IP_PATHS` instead of pruning everything - Added tests for edge cases and the typical case

n0bot bot added this to iroh Jul 7, 2025

github-project-automation bot moved this to 🏗 In progress in iroh Jul 7, 2025

dignifiedquire changed the title ~~[WIP] feat: use quinn multipath~~ [WIP] feat: use quic multipath Jul 8, 2025

dignifiedquire force-pushed the feat-multipath branch 3 times, most recently from 72cb071 to db712c0 Compare July 18, 2025 14:39

dignifiedquire force-pushed the feat-multipath branch 2 times, most recently from 4827e62 to 946f71c Compare July 28, 2025 20:29

dignifiedquire added 20 commits August 1, 2025 11:13

feat: use quinn multipath

c17a870

update iroh-quinn

7fe570d

start opening paths

2f469ac

add more paths

870716f

set keep alive and idle timeouts for new paths

346a7c2

insert relay path

68b1769

set relay path as backup

0eb3fde

start removing ping logic from the node_map

79ec17f

start tracking path events

c4baca8

start figuring out more details

549adee

wip

9ef5765

get some stuff to work again

75d5525

remove ip_mapped_addresses

4f78898

use correct relay addr on recv

130710f

ensure connection registration

dba89df

remove rtt_actor, this is now done inside quinn on a per path basis

aab083d

open additional paths after the initial connection

d4484da

ensure path open

6cb94e4

some debugging

998e283

update quinn branch

99cee61

flub and others added 30 commits November 14, 2025 10:46

bump quinn

3745e7e

Cleanup some broken tests

c24c5d4

There are issues for these already

fix

103e3c5

fix test by calling stream.finish() (#3665)

1f2db9f

Merge remote-tracking branch 'origin/main' into feat-multipath

7052392

test(iroh): Fix test_two_devices_setup_teardown hanging (#3675)

59a7e85

## Description Avoid potentially busy looping in a tokio task. I think this blocking leads to tokio not being able to close the runtime properly.

Merge remote-tracking branch 'origin/main' into feat-multipath-merge

c85bbbd

test: mark test_active_relay_inactive as non flaky

e620b5e

Merge pull request #3696 from n0-computer/feat-multipath-merge

d243702

merge main

chore: fixup wasm test

a28d1fb

refactor(iroh): simplify internal transports sending (#3708)

8d56889

removes the need for two `send` impls

refactor(iroh): avoid storing the TransportsSender (#3712)

3f4d365

Closes #3641

chore: update to quinn@main-iroh (#3716)

e22c001

refactor(iroh)!: remove Endpoint::latency (#3717)

783e2ef

## Description Based on #3593 This was always just a placeholder, and can now be collected using `EndpointHooks`. ## Breaking Changes - remove `Endpoint::latency` --------- Co-authored-by: varun-doshi <doshivarun202@gmail.com>

chore: update iroh-quinn (#3718)

2cf93a5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: switch to QUIC multipath #3381

feat: switch to QUIC multipath #3381

dignifiedquire commented Jul 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

feat: switch to QUIC multipath #3381

Are you sure you want to change the base?

feat: switch to QUIC multipath #3381

Conversation

dignifiedquire commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs

Uh oh!

github-actions bot commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dignifiedquire commented Jul 7, 2025 •

edited

Loading

github-actions bot commented Jul 7, 2025 •

edited

Loading

github-actions bot commented Jul 7, 2025 •

edited

Loading