storeliveness: smear storeliveness heartbeats

Previously, heartbeat messages were sent immediately when enqueued via
`EnqueueMessage`. In large clusters, many stores might all send heartbeats
simultaneously at the same tick interval, causing a spike in runnable
goroutines that caused issues elsewhere in the database process.

This patch introduces heartbeat smearing logic that batches and smears
heartbeat sends over a configurable duration using a taskpacer. The
smearing logic is enabled by default via the
`kv.store_liveness.heartbeat_smearing.enabled` cluster setting
(defaults to true), and can be configured via
`kv.store_liveness.heartbeat_smearing.refresh` (default: 10ms) and
`kv.store_liveness.heartbeat_smearing.smear` (default: 1ms) settings.

When enabled, messages are enqueued but not sent immediately. Instead,
they wait in per-node queues until `SendAllEnqueuedMessages()` is called,
which signals the coordinator. The coordinator waits briefly
(`batchDuration`) to collect messages from multiple stores, then paces the
signaling to each queue's `processQueue` goroutine using the pacer to
spread sends over time.

Fixes: #148210
Release note: None

Author: dodeca12

pkg/kv/kvserver/store_test.go

@@ -248,7 +248,7 @@ func createTestStoreWithoutStart(
 		supportGracePeriod := rpcContext.StoreLivenessWithdrawalGracePeriod()
 		options := storeliveness.NewOptions(livenessInterval, heartbeatInterval, supportGracePeriod)
 		transport, err := storeliveness.NewTransport(
-			cfg.AmbientCtx, stopper, cfg.Clock, cfg.NodeDialer, grpcServer, drpcServer, nil, /* knobs */
+			cfg.AmbientCtx, stopper, cfg.Clock, cfg.NodeDialer, grpcServer, drpcServer, cfg.Settings, nil, /* knobs */
 		)
 		require.NoError(t, err)
 		knobs := cfg.TestingKnobs.StoreLivenessKnobs

pkg/kv/kvserver/storeliveness/BUILD.bazel

@@ -30,10 +30,12 @@ go_library(
         "//pkg/util/envutil",
         "//pkg/util/hlc",
         "//pkg/util/log",
+        "//pkg/util/metamorphic",
         "//pkg/util/metric",
         "//pkg/util/protoutil",
         "//pkg/util/stop",
         "//pkg/util/syncutil",
+        "//pkg/util/taskpacer",
         "//pkg/util/timeutil",
         "@com_github_cockroachdb_errors//:errors",
         "@com_github_cockroachdb_redact//:redact",