Add dual-stack IP config for internal NICs #9389

bnaecker · 2025-11-12T04:49:47Z

Add dual-stack VPC private address configuration type and include it in the shared NetworkInterface type.
Add database model support for reading / writing the dual-stack NIC type to the database, handling serialization into optional fields.
Update all the callsites to handle the new dual-stack-aware type.
Add a bunch of conversions for the APIs which rely on that new type, of which there are many. This also adds the conversions and older types into the sled-agent-types crate, so they can be used in a few places that don't directly depend on the sled-agent-api crate itself, notably reconfigurator and nexus-inventory.
Update the sled-agent reconciler to deserialize previous versions of its sled-configuration ledgers, convert them, and write them out again as the new versions.
Updates the OPTE PortManager type with the new dual-stack support. This is only for private IP addresses, though. We still need some work to support OPTE ports with dual stack external addresses. This is about half of Enable dual-stack OPTE ports in the sled-agent #9247, the VPC-private part.
Closes Want dual-stack support for NICs between Nexus and sled-agent #9246

bnaecker · 2025-11-12T20:48:43Z

Thanks in advance to the reviewers. It turns out we use the NetworkInterface type basically everywhere, and so this change is uncomfortably large.

- Add dual-stack VPC private address configuration type and include it in the shared NetworkInterface type. - Add database model support for reading / writing the dual-stack NIC type to the database, handling serialization into optional fields. - Update all the callsites to handle the new dual-stack-aware type. - Add a bunch of conversions for the APIs which rely on that new type, of which there are many. This also adds the conversions and older types into the `sled-agent-types` crate, so they can be used in a few places that don't directly depend on the `sled-agent-api` crate itself, notably reconfigurator and `nexus-inventory`. - Update the sled-agent reconciler to deserialize previous versions of its sled-configuration ledgers, convert them, and write them out again as the new versions. - Updates the OPTE `PortManager` type with the new dual-stack support. This is only for private IP addresses, though. We still need some work to support OPTE ports with dual stack external addresses. This is about half of #9247, the VPC-private part. - Closes #9246

…ic-types

bnaecker · 2025-11-15T23:20:38Z

I'll figure out how to resolve the conflicts with the multicast work on Monday.

bnaecker · 2025-11-17T17:52:55Z

I'm going to wait to fix the conflicts until we address some changes in the way the conflicting code was structured. I'll still have conflicts to resolve that point, but that will be entirely mechanical. The PR is still ready for review as it stands.

rcgoodfellow

Sorry for the delay in getting this posted @bnaecker. Review comments follow. However, this touches on a lot of Omicron systems I'm not very familiar with like inventory, blueprints, ledgers and reconfigurator. It would be good to have someone who works in those systems to take a look.

common/src/api/internal/shared/network_interface/v2.rs

dev-tools/reconfigurator-cli/src/lib.rs

illumos-utils/src/opte/port_manager.rs

nexus/db-model/src/network_interface.rs

nexus/db-queries/src/db/datastore/deployment/external_networking.rs

rcgoodfellow · 2025-11-18T06:00:33Z

sled-agent/api/src/lib.rs

+        path = "/vmms/{propolis_id}",
+        versions = ..VERSION_ADD_DUAL_STACK_SHARED_NETWORK_INTERFACES,
+    }]
+    async fn v6_vmm_register(


This was very confusing to me at first as i was reading v6 as IPv6 and did not realize this was the old method being moved to API version 6.

Luckily once this catches up with main it'll be somewhere between v8 and v10 instead 😅

sled-agent/src/services.rs

common/src/api/internal/shared/network_interface/v2.rs

common/src/api/internal/shared/network_interface/mod.rs

illumos-utils/src/opte/mod.rs

jgallagher · 2025-11-18T16:45:12Z

sled-agent/api/src/lib.rs

+        path = "/vmms/{propolis_id}",
+        versions = ..VERSION_ADD_DUAL_STACK_SHARED_NETWORK_INTERFACES,
+    }]
+    async fn v6_vmm_register(


Luckily once this catches up with main it'll be somewhere between v8 and v10 instead 😅

sled-agent/config-reconciler/src/ledger.rs

sled-agent/config-reconciler/src/ledger/legacy_configs.rs

illumos-utils/src/opte/port_manager.rs

- Avoid panics in DB conversion and Omicron-to-OPTE IP configuration conversion - Clean up some error messages - Add helpers to return any IP address for an OPTE port / gateway - Remove explicit v2 module for current version of NIC type

…ic-types

This adds only a v8 into the public sled-agent-types crate with the previous version. It adds infallible conversions from older API types to and from v8, rather than changing those conversions to be directly to the latest type. Also call the "next" version of the actual API handler too, rather than jumping to the latest.

bnaecker · 2025-11-19T19:15:26Z

Ok @jgallagher @rcgoodfellow I believe I've addressed your comments and fixed up all the merge conflicts with main. This should be ready for another look when you get a chance. Thanks!

jgallagher

LGTM!

jgallagher · 2025-11-19T23:18:23Z

illumos-utils/src/opte/port_manager.rs

+        .subnet()
+        .iter()
+        // The 0th address is the network address, and OPTE's
+        // gateway sits at the next. See


Can this call v6.opte_gateway() instead?

Doh! Yeah, I added those methods and never used them. Thanks!

jgallagher · 2025-11-19T23:18:40Z

illumos-utils/src/opte/port_manager.rs

+    floating_ips: &[IpAddr],
+    is_ipv4_only: bool,
+) -> Result<Ipv4Cfg, Error> {
+    let gateway_ip = v4.subnet().first_host().into();


Can this call v4.opte_gateway() instead?

jgallagher · 2025-11-19T23:27:13Z

sled-agent/api/src/lib.rs

+        versions =
+            VERSION_ADD_PROBE_PUT_ENDPOINT..VERSION_ADD_DUAL_STACK_SHARED_NETWORK_INTERFACES,
+    }]
+    async fn v6_probes_put(


Should this be v9 instead of v6?

It should, yes. Thanks.

…ic-types

bnaecker marked this pull request as draft November 12, 2025 04:49

bnaecker force-pushed the dual-stack-internal-nic-types branch from 9d04b7a to 83a21bb Compare November 12, 2025 20:41

bnaecker requested review from jgallagher, rcgoodfellow and smklein November 12, 2025 20:41

bnaecker marked this pull request as ready for review November 12, 2025 20:42

bnaecker force-pushed the dual-stack-internal-nic-types branch from 83a21bb to 2033231 Compare November 12, 2025 20:48

bnaecker force-pushed the dual-stack-internal-nic-types branch from 2033231 to fe87906 Compare November 12, 2025 20:49

bnaecker added 5 commits November 14, 2025 21:46

Add more OPTE tests, fixup gateway IP computation

307abbd

Merge remote-tracking branch 'origin/main' into dual-stack-internal-n…

f21331c

…ic-types

Fix mismerge

bd60e20

Fixup expectorate test

d82e117

small api fixes

95a7f8f

rcgoodfellow reviewed Nov 18, 2025

View reviewed changes

jgallagher reviewed Nov 18, 2025

View reviewed changes

bnaecker added 6 commits November 19, 2025 04:00

Review feedback

dfedcbd

- Avoid panics in DB conversion and Omicron-to-OPTE IP configuration conversion - Clean up some error messages - Add helpers to return any IP address for an OPTE port / gateway - Remove explicit v2 module for current version of NIC type

New version of sled-agent API

01bcde4

Merge remote-tracking branch 'origin/main' into dual-stack-internal-n…

60936ed

…ic-types

Merge branch 'main' into dual-stack-internal-nic-types

0c96612

Cleanup after another merge conflict

cb289fb

doc nit

b241955

jgallagher approved these changes Nov 19, 2025

View reviewed changes

review feedback, small renaming / cleanup

f590223

bnaecker mentioned this pull request Nov 19, 2025

test failed in CI: task::tests::tq_proxy #9424

Closed

Merge remote-tracking branch 'origin/main' into dual-stack-internal-n…

8970abf

…ic-types

bnaecker enabled auto-merge (squash) November 20, 2025 00:02

bnaecker merged commit f062472 into main Nov 20, 2025
17 checks passed

bnaecker deleted the dual-stack-internal-nic-types branch November 20, 2025 03:21

Add dual-stack IP config for internal NICs #9389

Add dual-stack IP config for internal NICs #9389

Uh oh!

Conversation

bnaecker commented Nov 12, 2025

Uh oh!

bnaecker commented Nov 12, 2025

Uh oh!

bnaecker commented Nov 15, 2025

Uh oh!

bnaecker commented Nov 17, 2025

Uh oh!

rcgoodfellow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rcgoodfellow Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jgallagher Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgallagher Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bnaecker commented Nov 19, 2025

Uh oh!

jgallagher left a comment

Choose a reason for hiding this comment

Uh oh!

jgallagher Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

bnaecker Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jgallagher Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

jgallagher Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

bnaecker Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rcgoodfellow Nov 18, 2025 •

edited

Loading