smp: Reduce NMI traffic from CSD waiters to CSD destination

JIRA: https://issues.redhat.com/browse/RHEL-16867

commit 0d3a00b
Author: Imran Khan <imran.f.khan@oracle.com>
Date:   Tue, 9 May 2023 08:31:24 +1000

    smp: Reduce NMI traffic from CSD waiters to CSD destination

    On systems with hundreds of CPUs, if most of the CPUs detect a CSD hang,
    then all of these waiting CPUs send an NMI to the destination CPU in
    order to dump its backtrace.

    Given enough NMIs, the destination CPU will spent much of its time
    producing backtraces, thus further delaying that CPU's response to the
    original CSD IPI.  In the worst case, by the time destination CPU is
    done producing all of these backtrace NMIs, the CSD wait timeout will
    have elapsed so that the waiters resend their backtrace NMIs again,
    further delaying forward progress.

    Therefore, to avoid these delays, issue the backtrace NMI only from
    the first waiter.  The destination CPU's other waiters can make use of
    backtrace obtained from the first waiter's NMI.

    Signed-off-by: Imran Khan <imran.f.khan@oracle.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Juergen Gross <jgross@suse.com>
    Cc: Valentin Schneider <vschneid@redhat.com>
    Cc: Yury Norov <yury.norov@gmail.com>
    Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

Signed-off-by: Waiman Long <longman@redhat.com>

Author: Waiman-Long

kernel/smp.c

@@ -46,6 +46,8 @@ static DEFINE_PER_CPU_ALIGNED(struct call_function_data, cfd_data);
 
 static DEFINE_PER_CPU_SHARED_ALIGNED(struct llist_head, call_single_queue);
 
+static DEFINE_PER_CPU(atomic_t, trigger_backtrace) = ATOMIC_INIT(1);
+
 static void __flush_smp_call_function_queue(bool warn_cpu_offline);
 
 int smpcfd_prepare_cpu(unsigned int cpu)
@@ -248,7 +250,8 @@ static bool csd_lock_wait_toolong(struct __call_single_data *csd, u64 ts0, u64 *
 			 *bug_id, !cpu_cur_csd ? "unresponsive" : "handling this request");
 	}
 	if (cpu >= 0) {
-		dump_cpu_task(cpu);
+		if (atomic_cmpxchg_acquire(&per_cpu(trigger_backtrace, cpu), 1, 0))
+			dump_cpu_task(cpu);
 		if (!cpu_cur_csd) {
 			pr_alert("csd: Re-sending CSD lock (#%d) IPI from CPU#%02d to CPU#%02d\n", *bug_id, raw_smp_processor_id(), cpu);
 			arch_send_call_function_single_ipi(cpu);
@@ -429,9 +432,14 @@ static void __flush_smp_call_function_queue(bool warn_cpu_offline)
 	struct llist_node *entry, *prev;
 	struct llist_head *head;
 	static bool warned;
+	atomic_t *tbt;
 
 	lockdep_assert_irqs_disabled();
 
+	/* Allow waiters to send backtrace NMI from here onwards */
+	tbt = this_cpu_ptr(&trigger_backtrace);
+	atomic_set_release(tbt, 1);
+
 	head = this_cpu_ptr(&call_single_queue);
 	entry = llist_del_all(head);
 	entry = llist_reverse_order(entry);