server: don't exit auto-upgrade if own node isn't live

The auto upgrade loop previously eagerly exited early if all
nodes were on the same version. This optimized for the common
case where we are not attempting an upgrade, and don't care
if a node is currently down.

However, we don't want to exit the auto upgrade process if we can't validate
our own version. Consider the case where the meta1 leaseholder is the first
node to restart to the new version but has transient liveness issues. If we skip
the check here, we will see all other nodes at the same (old) version and exit
the auto upgrade process with UpgradeAlreadyCompleted. Since the meta1 leaseholder
is the only node that can perform the upgrade, this indefinitely blocks the upgrade.

Fixes: #155359

Author: DarrylWong

pkg/server/auto_upgrade.go

@@ -183,6 +183,15 @@ func (s *topLevelServer) upgradeStatus(
 			if notRunningErr == nil {
 				notRunningErr = errors.Errorf("node %d not running (%d), cannot determine version", nodeID, st)
 			}
+			// However, we don't want to exit the auto upgrade process if we can't validate
+			// our own version. Consider the case where the meta1 leaseholder is the first
+			// node to restart to the new version but has transient liveness issues. If we skip
+			// the check here, we will see all other nodes at the same (old) version and exit
+			// the auto upgrade process with UpgradeAlreadyCompleted. Since the meta1 leaseholder
+			// is the only node that can perform the upgrade, this indefinitely blocks the upgrade.
+			if s.node.Descriptor.NodeID == nodeID {
+				return UpgradeBlockedDueToError, errors.Errorf("node %d not running (%d), cannot determine version", nodeID, st)
+			}
 			continue
 		}