TQ: Support proxying Nexus-related API requests (#9403)

When committing configurations, the trust quorum protocol relies on
Nexus to be up and for Nexus to interact with each trust quorum node.
This allows Nexus to continuously try to commit nodes in an RPW and
record in the database which nodes have acked. This mirrors many of our
existing designs where Nexus observes and records information about
successful operations and retries via RPW. Specifically it matches how
we do things in the `Reconfigurator` and the `TUF Repo Depot`.

Unfortunately, Nexus cannot communicate with new sleds that are not yet
running a sled-agent, but still stuck in the bootstrap-agent. This is
because the bootstrap agents (and trust quorum protocol) only
communicate over the bootstrap network, which Nexus does not have access
to. Nodes must already be part of an existing configuration, running
sled-agent, and on the underlay network to talk to Nexus. In this common
case, Nexus sends trust quorum related messages to the sled-agent which
then calls the api of its local trust quorum `NodeTask`. This is not
possible for newly added sleds. While the trust quroum coordinator node
will tell new nodes to `Prepare` a configuration over the bootstrap
networtk, these new nodes do not have any mechanism to receive commits
from Nexus. Therefore we must proxy these commit related operations to
an existing member of the trust quorum when adding a new node. We also
added the ability to proxy `NodeStatus` requests to aid in debugging.

This PR therefore adds the ability to proxy certain requests from one
node to another so that we can commit nodes to the latest trust quorum
configuration, setup their encrypted storage, and boot their sled-agent.

It's worth noting that this is not the only way we could have solved
this problem. There are a few possibilities in the design space.

1. We could have had the coordinator always send commit operations and
collect acknowledgements as during the `Prepare` phase. Unfortunately,
if the coordinator dies before all nodes ack then Nexus would not be
able to ensure commit at all nodes. To make this reliable, Nexus would
still need to be able to reach out to uncommitted nodes and tell them to
commit. Since we already have to do the latter there is no reason to do
the former.

2. We could commit at the coordinator (or a few nodes), and then have
them gossip around information about commit. This is actually a
promising design, and is essentially what we do for the early network
config. Nexus could then wait for the sled-agent to start for those
nodes and ask them directly if they committed. This would still require
talking to all nodes and it adds some extra complexity, but it still
seems somewhat reasonable. The rationale for our current choice of
proxying was largely one of fitting our existing patterns. It's also
very useful for Nexus to be able directly ask a trust quorum node on
another sled about its status to diagnose issues.

So we went with the proxy mechanism as implemented here. Well, why did
we introduce another level of messages at the `Task` layer instead of
re-using the `CommitAdvance` functionality or adding new variants to the
`PeerMsg` in the `trust_quorum_protocol` crate? The rationale here is
largely that the trust quorum protocol as written in RFD 238 and
specified in TLA+ doesn't include this behavior. It expects commits from
the `Node` "API", meaning from `Nexus`. I didn't want to change that
behavior unnecessarily due to urgency, and an existing solid design.

It was also easier to build proxy operations this way since tracking
operations in async code with oneshot channels is easier than trying to
insert similar tracking into the `sans-io` code. In short, we left the
`trust-quorum-protocol` crate alone, and added some async helpers to the
`trust_quorum` crate.

One additional change was made in this PR. While adding the `tq_proxy`
test I noticed that we were unnecessarily using `wait_for_condition` on
initial commits, after we knew about succesful prepares. These commits
should always complete immediately and so I simplified this code in a
few existing tests.

Author: andrewjstone

Cargo.lock

@@ -16,6 +16,7 @@ camino.workspace = true
 chacha20poly1305.workspace = true
 ciborium.workspace = true
 daft.workspace = true
+debug-ignore.workspace = true
 derive_more.workspace = true
 futures.workspace = true
 gfss.workspace = true

trust-quorum/Cargo.toml

@@ -32,7 +32,9 @@ use crate::{
 use daft::{Diffable, Leaf};
 use gfss::shamir::Share;
 use omicron_uuid_kinds::RackUuid;
+use serde::{Deserialize, Serialize};
 use slog::{Logger, error, info, o, warn};
+use slog_error_chain::SlogInlineError;
 
 /// An entity capable of participating in trust quorum
 ///
@@ -379,6 +381,16 @@ impl Node {
     }
 
     /// A peer node has disconnected from this one
+    ///
+    /// Note: It is safe if a call to `on_disconnect` is missed due to a new
+    /// connection replacing the existing connection and resulting in a call
+    /// to `on_connect` while the node thinks it still maintains a connection.
+    ///
+    /// All active behavior such as retries occur in `on_connect`. If the
+    /// contents of this method change such that it is no longer safe to call
+    /// `on_connect` without first calling `on_disconnect` for an already
+    /// connected peer, then we can call `on_disconnect` first from directly
+    /// within `on_connect`. For now that is unnecessary.
     pub fn on_disconnect(
         &mut self,
         ctx: &mut impl NodeHandlerCtx,
@@ -1063,7 +1075,16 @@ impl Node {
     }
 }
 
-#[derive(Debug, Clone, thiserror::Error, PartialEq, Eq)]
+#[derive(
+    Debug,
+    Clone,
+    thiserror::Error,
+    PartialEq,
+    Eq,
+    SlogInlineError,
+    Serialize,
+    Deserialize,
+)]
 pub enum CommitError {
     #[error("invalid rack id")]
     InvalidRackId(
@@ -1077,7 +1098,16 @@ pub enum CommitError {
     Expunged { epoch: Epoch, from: BaseboardId },
 }
 
-#[derive(Debug, Clone, thiserror::Error, PartialEq, Eq)]
+#[derive(
+    Debug,
+    Clone,
+    thiserror::Error,
+    PartialEq,
+    Eq,
+    SlogInlineError,
+    Serialize,
+    Deserialize,
+)]
 pub enum PrepareAndCommitError {
     #[error("invalid rack id")]
     InvalidRackId(

trust-quorum/protocol/src/node.rs

@@ -12,6 +12,7 @@ use crate::{
 };
 use daft::{BTreeSetDiff, Diffable, Leaf};
 use omicron_uuid_kinds::RackUuid;
+use serde::{Deserialize, Serialize};
 use slog::{Logger, error, info, warn};
 use std::collections::BTreeSet;
 
@@ -57,7 +58,9 @@ pub struct SledExpungedError {
     last_prepared_epoch: Option<Epoch>,
 }
 
-#[derive(Debug, Clone, thiserror::Error, PartialEq, Eq)]
+#[derive(
+    Debug, Clone, thiserror::Error, PartialEq, Eq, Serialize, Deserialize,
+)]
 #[error("mismatched rack id: expected {expected:?}, got {got:?}")]
 pub struct MismatchedRackIdError {
     pub expected: RackUuid,