releases/4.19.27/sched-wake_q-fix-wakeup-ordering-for-wake_q.patch

   1 From b62099029e96c7743cec11dfa391ff1f0c87980a Mon Sep 17 00:00:00 2001
   2 From: Peter Zijlstra <peterz@infradead.org>
   3 Date: Mon, 17 Dec 2018 10:14:53 +0100
   4 Subject: sched/wake_q: Fix wakeup ordering for wake_q
   5
   6 [ Upstream commit 4c4e3731564c8945ac5ac90fc2a1e1f21cb79c92 ]
   7
   8 Notable cmpxchg() does not provide ordering when it fails, however
   9 wake_q_add() requires ordering in this specific case too. Without this
  10 it would be possible for the concurrent wakeup to not observe our
  11 prior state.
  12
  13 Andrea Parri provided:
  14
  15   C wake_up_q-wake_q_add
  16
  17   {
  18         int next = 0;
  19         int y = 0;
  20   }
  21
  22   P0(int *next, int *y)
  23   {
  24         int r0;
  25
  26         /* in wake_up_q() */
  27
  28         WRITE_ONCE(*next, 1);   /* node->next = NULL */
  29         smp_mb();               /* implied by wake_up_process() */
  30         r0 = READ_ONCE(*y);
  31   }
  32
  33   P1(int *next, int *y)
  34   {
  35         int r1;
  36
  37         /* in wake_q_add() */
  38
  39         WRITE_ONCE(*y, 1);      /* wake_cond = true */
  40         smp_mb__before_atomic();
  41         r1 = cmpxchg_relaxed(next, 1, 2);
  42   }
  43
  44   exists (0:r0=0 /\ 1:r1=0)
  45
  46   This "exists" clause cannot be satisfied according to the LKMM:
  47
  48   Test wake_up_q-wake_q_add Allowed
  49   States 3
  50   0:r0=0; 1:r1=1;
  51   0:r0=1; 1:r1=0;
  52   0:r0=1; 1:r1=1;
  53   No
  54   Witnesses
  55   Positive: 0 Negative: 3
  56   Condition exists (0:r0=0 /\ 1:r1=0)
  57   Observation wake_up_q-wake_q_add Never 0 3
  58
  59 Reported-by: Yongji Xie <elohimes@gmail.com>
  60 Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
  61 Cc: Davidlohr Bueso <dave@stgolabs.net>
  62 Cc: Linus Torvalds <torvalds@linux-foundation.org>
  63 Cc: Peter Zijlstra <peterz@infradead.org>
  64 Cc: Thomas Gleixner <tglx@linutronix.de>
  65 Cc: Waiman Long <longman@redhat.com>
  66 Cc: Will Deacon <will.deacon@arm.com>
  67 Signed-off-by: Ingo Molnar <mingo@kernel.org>
  68 Signed-off-by: Sasha Levin <sashal@kernel.org>
  69 ---
  70  kernel/sched/core.c | 7 ++++---
  71  1 file changed, 4 insertions(+), 3 deletions(-)
  72
  73 diff --git a/kernel/sched/core.c b/kernel/sched/core.c
  74 index 13ddfa46d741f..152a0b0c91bb6 100644
  75 --- a/kernel/sched/core.c
  76 +++ b/kernel/sched/core.c
  77 @@ -405,10 +405,11 @@ void wake_q_add(struct wake_q_head *head, struct task_struct *task)
  78          * its already queued (either by us or someone else) and will get the
  79          * wakeup due to that.
  80          *
  81 -        * This cmpxchg() executes a full barrier, which pairs with the full
  82 -        * barrier executed by the wakeup in wake_up_q().
  83 +        * In order to ensure that a pending wakeup will observe our pending
  84 +        * state, even in the failed case, an explicit smp_mb() must be used.
  85          */
  86 -       if (cmpxchg(&node->next, NULL, WAKE_Q_TAIL))
  87 +       smp_mb__before_atomic();
  88 +       if (cmpxchg_relaxed(&node->next, NULL, WAKE_Q_TAIL))
  89                 return;
  90
  91         get_task_struct(task);
  92 --
  93 2.19.1
  94