Skip to content

Commit a02e215

Browse files
PlaidCatPlaidCat
committed
locking/rwsem: Allow slowpath writer to ignore handoff bit if not set by first waiter
jira LE-1907 Rebuild_History Non-Buildable kernel-rt-5.14.0-284.30.1.rt14.315.el9_2 commit-author Waiman Long <longman@redhat.com> commit 6eebd5f Empty-Commit: Cherry-Pick Conflicts during history rebuild. Will be included in final tarball splat. Ref for failed cherry-pick at: ciq/ciq_backports/kernel-rt-5.14.0-284.30.1.rt14.315.el9_2/6eebd5fb.failed With commit d257cc8 ("locking/rwsem: Make handoff bit handling more consistent"), the writer that sets the handoff bit can be interrupted out without clearing the bit if the wait queue isn't empty. This disables reader and writer optimistic lock spinning and stealing. Now if a non-first writer in the queue is somehow woken up or a new waiter enters the slowpath, it can't acquire the lock. This is not the case before commit d257cc8 as the writer that set the handoff bit will clear it when exiting out via the out_nolock path. This is less efficient as the busy rwsem stays in an unlock state for a longer time. In some cases, this new behavior may cause lockups as shown in [1] and [2]. This patch allows a non-first writer to ignore the handoff bit if it is not originally set or initiated by the first waiter. This patch is shown to be effective in fixing the lockup problem reported in [1]. [1] https://lore.kernel.org/lkml/20220617134325.GC30825@techsingularity.net/ [2] https://lore.kernel.org/lkml/3f02975c-1a9d-be20-32cf-f1d8e3dfafcc@oracle.com/ Fixes: d257cc8 ("locking/rwsem: Make handoff bit handling more consistent") Signed-off-by: Waiman Long <longman@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Acked-by: John Donnelly <john.p.donnelly@oracle.com> Tested-by: Mel Gorman <mgorman@techsingularity.net> Link: https://lore.kernel.org/r/20220622200419.778799-1-longman@redhat.com (cherry picked from commit 6eebd5f) Signed-off-by: Jonathan Maple <jmaple@ciq.com> # Conflicts: # kernel/locking/rwsem.c
1 parent 3ab2ee3 commit a02e215

File tree

1 file changed

+105
-0
lines changed

1 file changed

+105
-0
lines changed

‎ciq/ciq_backports/kernel-rt-5.14.0-284.30.1.rt14.315.el9_2/6eebd5fb.failed‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
locking/rwsem: Allow slowpath writer to ignore handoff bit if not set by first waiter
2+
3+
jira LE-1907
4+
Rebuild_History Non-Buildable kernel-rt-5.14.0-284.30.1.rt14.315.el9_2
5+
commit-author Waiman Long <longman@redhat.com>
6+
commit 6eebd5fb20838f5971ba17df9f55cc4f84a31053
7+
Empty-Commit: Cherry-Pick Conflicts during history rebuild.
8+
Will be included in final tarball splat. Ref for failed cherry-pick at:
9+
ciq/ciq_backports/kernel-rt-5.14.0-284.30.1.rt14.315.el9_2/6eebd5fb.failed
10+
11+
With commit d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more
12+
consistent"), the writer that sets the handoff bit can be interrupted
13+
out without clearing the bit if the wait queue isn't empty. This disables
14+
reader and writer optimistic lock spinning and stealing.
15+
16+
Now if a non-first writer in the queue is somehow woken up or a new
17+
waiter enters the slowpath, it can't acquire the lock. This is not the
18+
case before commit d257cc8cb8d5 as the writer that set the handoff bit
19+
will clear it when exiting out via the out_nolock path. This is less
20+
efficient as the busy rwsem stays in an unlock state for a longer time.
21+
22+
In some cases, this new behavior may cause lockups as shown in [1] and
23+
[2].
24+
25+
This patch allows a non-first writer to ignore the handoff bit if it
26+
is not originally set or initiated by the first waiter. This patch is
27+
shown to be effective in fixing the lockup problem reported in [1].
28+
29+
[1] https://lore.kernel.org/lkml/20220617134325.GC30825@techsingularity.net/
30+
[2] https://lore.kernel.org/lkml/3f02975c-1a9d-be20-32cf-f1d8e3dfafcc@oracle.com/
31+
32+
Fixes: d257cc8cb8d5 ("locking/rwsem: Make handoff bit handling more consistent")
33+
Signed-off-by: Waiman Long <longman@redhat.com>
34+
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
35+
Acked-by: John Donnelly <john.p.donnelly@oracle.com>
36+
Tested-by: Mel Gorman <mgorman@techsingularity.net>
37+
Link: https://lore.kernel.org/r/20220622200419.778799-1-longman@redhat.com
38+
(cherry picked from commit 6eebd5fb20838f5971ba17df9f55cc4f84a31053)
39+
Signed-off-by: Jonathan Maple <jmaple@ciq.com>
40+
41+
# Conflicts:
42+
# kernel/locking/rwsem.c
43+
diff --cc kernel/locking/rwsem.c
44+
index 186ad9eda88d,65f0262f635e..000000000000
45+
--- a/kernel/locking/rwsem.c
46+
+++ b/kernel/locking/rwsem.c
47+
@@@ -342,6 -335,7 +342,10 @@@ struct rwsem_waiter
48+
struct task_struct *task;
49+
enum rwsem_waiter_type type;
50+
unsigned long timeout;
51+
++<<<<<<< HEAD
52+
++=======
53+
+ bool handoff_set;
54+
++>>>>>>> 6eebd5fb2083 (locking/rwsem: Allow slowpath writer to ignore handoff bit if not set by first waiter)
55+
};
56+
#define rwsem_first_waiter(sem) \
57+
list_first_entry(&sem->wait_list, struct rwsem_waiter, list)
58+
@@@ -540,12 -594,12 +546,16 @@@ static void rwsem_mark_wake(struct rw_s
59+
* race conditions between checking the rwsem wait list and setting the
60+
* sem->count accordingly.
61+
*
62+
- * Implies rwsem_del_waiter() on success.
63+
+ * If wstate is WRITER_HANDOFF, it will make sure that either the handoff
64+
+ * bit is set or the lock is acquired with handoff bit cleared.
65+
*/
66+
static inline bool rwsem_try_write_lock(struct rw_semaphore *sem,
67+
- struct rwsem_waiter *waiter)
68+
+ enum writer_wait_state wstate)
69+
{
70+
++<<<<<<< HEAD
71+
++=======
72+
+ struct rwsem_waiter *first = rwsem_first_waiter(sem);
73+
++>>>>>>> 6eebd5fb2083 (locking/rwsem: Allow slowpath writer to ignore handoff bit if not set by first waiter)
74+
long count, new;
75+
76+
lockdep_assert_held(&sem->wait_lock);
77+
@@@ -554,8 -608,22 +564,27 @@@
78+
do {
79+
bool has_handoff = !!(count & RWSEM_FLAG_HANDOFF);
80+
81+
++<<<<<<< HEAD
82+
+ if (has_handoff && wstate == WRITER_NOT_FIRST)
83+
+ return false;
84+
++=======
85+
+ if (has_handoff) {
86+
+ /*
87+
+ * Honor handoff bit and yield only when the first
88+
+ * waiter is the one that set it. Otherwisee, we
89+
+ * still try to acquire the rwsem.
90+
+ */
91+
+ if (first->handoff_set && (waiter != first))
92+
+ return false;
93+
+
94+
+ /*
95+
+ * First waiter can inherit a previously set handoff
96+
+ * bit and spin on rwsem if lock acquisition fails.
97+
+ */
98+
+ if (waiter == first)
99+
+ waiter->handoff_set = true;
100+
+ }
101+
++>>>>>>> 6eebd5fb2083 (locking/rwsem: Allow slowpath writer to ignore handoff bit if not set by first waiter)
102+
103+
new = count;
104+
105+
* Unmerged path kernel/locking/rwsem.c

0 commit comments

Comments
 (0)