net: fix sk_memory_allocated_{add|sub} vs softirqs

jira LE-1907
Rebuild_History Non-Buildable kernel-5.14.0-427.28.1.el9_4
commit-author Eric Dumazet <edumazet@google.com>
commit 3584718
Empty-Commit: Cherry-Pick Conflicts during history rebuild.
Will be included in final tarball splat. Ref for failed cherry-pick at:
ciq/ciq_backports/kernel-5.14.0-427.28.1.el9_4/3584718c.failed

Jonathan Heathcote reported a regression caused by blamed commit
on aarch64 architecture.

x86 happens to have irq-safe __this_cpu_add_return()
and __this_cpu_sub(), but this is not generic.

I think my confusion came from "struct sock" argument,
because these helpers are called with a locked socket.
But the memory accounting is per-proto (and per-cpu after
the blamed commit). We might cleanup these helpers later
to directly accept a "struct proto *proto" argument.

Switch to this_cpu_add_return() and this_cpu_xchg()
operations, and get rid of preempt_disable()/preempt_enable() pairs.

Fast path becomes a bit faster as a result :)

Many thanks to Jonathan Heathcote for his awesome report and
investigations.

Fixes: 3cd3399 ("net: implement per-cpu reserves for memory_allocated")
	Reported-by: Jonathan Heathcote <jonathan.heathcote@bbc.co.uk>
Closes: https://lore.kernel.org/netdev/VI1PR01MB42407D7947B2EA448F1E04EFD10D2@VI1PR01MB4240.eurprd01.prod.exchangelabs.com/
	Signed-off-by: Eric Dumazet <edumazet@google.com>
	Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
	Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Link: https://lore.kernel.org/r/20240421175248.1692552-1-edumazet@google.com
	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
(cherry picked from commit 3584718)
	Signed-off-by: Jonathan Maple <jmaple@ciq.com>

# Conflicts:
#	include/net/sock.h

Author: PlaidCat

ciq/ciq_backports/kernel-5.14.0-427.28.1.el9_4/3584718c.failed

@@ -0,0 +1,110 @@
+net: fix sk_memory_allocated_{add|sub} vs softirqs
+
+jira LE-1907
+Rebuild_History Non-Buildable kernel-5.14.0-427.28.1.el9_4
+commit-author Eric Dumazet <edumazet@google.com>
+commit 3584718cf2ec7e79b6814f2596dcf398c5fb2eca
+Empty-Commit: Cherry-Pick Conflicts during history rebuild.
+Will be included in final tarball splat. Ref for failed cherry-pick at:
+ciq/ciq_backports/kernel-5.14.0-427.28.1.el9_4/3584718c.failed
+
+Jonathan Heathcote reported a regression caused by blamed commit
+on aarch64 architecture.
+
+x86 happens to have irq-safe __this_cpu_add_return()
+and __this_cpu_sub(), but this is not generic.
+
+I think my confusion came from "struct sock" argument,
+because these helpers are called with a locked socket.
+But the memory accounting is per-proto (and per-cpu after
+the blamed commit). We might cleanup these helpers later
+to directly accept a "struct proto *proto" argument.
+
+Switch to this_cpu_add_return() and this_cpu_xchg()
+operations, and get rid of preempt_disable()/preempt_enable() pairs.
+
+Fast path becomes a bit faster as a result :)
+
+Many thanks to Jonathan Heathcote for his awesome report and
+investigations.
+
+Fixes: 3cd3399dd7a8 ("net: implement per-cpu reserves for memory_allocated")
+	Reported-by: Jonathan Heathcote <jonathan.heathcote@bbc.co.uk>
+Closes: https://lore.kernel.org/netdev/VI1PR01MB42407D7947B2EA448F1E04EFD10D2@VI1PR01MB4240.eurprd01.prod.exchangelabs.com/
+	Signed-off-by: Eric Dumazet <edumazet@google.com>
+	Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
+	Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
+Link: https://lore.kernel.org/r/20240421175248.1692552-1-edumazet@google.com
+	Signed-off-by: Jakub Kicinski <kuba@kernel.org>
+(cherry picked from commit 3584718cf2ec7e79b6814f2596dcf398c5fb2eca)
+	Signed-off-by: Jonathan Maple <jmaple@ciq.com>
+
+# Conflicts:
+#	include/net/sock.h
+diff --cc include/net/sock.h
+index 94057981f7fc,b4b553df7870..000000000000
+--- a/include/net/sock.h
++++ b/include/net/sock.h
+@@@ -1485,33 -1408,36 +1485,55 @@@ sk_memory_allocated(const struct sock *
+  
+  /* 1 MB per cpu, in page units */
+  #define SK_MEMORY_PCPU_RESERVE (1 << (20 - PAGE_SHIFT))
+ -extern int sysctl_mem_pcpu_rsv;
+  
+- static inline void
+- sk_memory_allocated_add(struct sock *sk, int amt)
++ static inline void proto_memory_pcpu_drain(struct proto *proto)
+  {
+- 	int local_reserve;
++ 	int val = this_cpu_xchg(*proto->per_cpu_fw_alloc, 0);
+  
+++<<<<<<< HEAD
+ +	preempt_disable();
+ +	local_reserve = __this_cpu_add_return(*sk->sk_prot->per_cpu_fw_alloc, amt);
+ +	if (local_reserve >= SK_MEMORY_PCPU_RESERVE) {
+ +		__this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve);
+ +		atomic_long_add(local_reserve, sk->sk_prot->memory_allocated);
+ +	}
+ +	preempt_enable();
+++=======
++ 	if (val)
++ 		atomic_long_add(val, proto->memory_allocated);
+++>>>>>>> 3584718cf2ec (net: fix sk_memory_allocated_{add|sub} vs softirqs)
+  }
+  
+  static inline void
+- sk_memory_allocated_sub(struct sock *sk, int amt)
++ sk_memory_allocated_add(const struct sock *sk, int val)
+  {
+- 	int local_reserve;
++ 	struct proto *proto = sk->sk_prot;
+  
+++<<<<<<< HEAD
+ +	preempt_disable();
+ +	local_reserve = __this_cpu_sub_return(*sk->sk_prot->per_cpu_fw_alloc, amt);
+ +	if (local_reserve <= -SK_MEMORY_PCPU_RESERVE) {
+ +		__this_cpu_sub(*sk->sk_prot->per_cpu_fw_alloc, local_reserve);
+ +		atomic_long_add(local_reserve, sk->sk_prot->memory_allocated);
+ +	}
+ +	preempt_enable();
+++=======
++ 	val = this_cpu_add_return(*proto->per_cpu_fw_alloc, val);
++ 
++ 	if (unlikely(val >= READ_ONCE(sysctl_mem_pcpu_rsv)))
++ 		proto_memory_pcpu_drain(proto);
++ }
++ 
++ static inline void
++ sk_memory_allocated_sub(const struct sock *sk, int val)
++ {
++ 	struct proto *proto = sk->sk_prot;
++ 
++ 	val = this_cpu_sub_return(*proto->per_cpu_fw_alloc, val);
++ 
++ 	if (unlikely(val <= -READ_ONCE(sysctl_mem_pcpu_rsv)))
++ 		proto_memory_pcpu_drain(proto);
+++>>>>>>> 3584718cf2ec (net: fix sk_memory_allocated_{add|sub} vs softirqs)
+  }
+  
+  #define SK_ALLOC_PERCPU_COUNTER_BATCH 16
+* Unmerged path include/net/sock.h