queue-4.19/rcu-do-rcu-gp-kthread-self-wakeup-from-softirq-and-interrupt.patch

   1 From 1d1f898df6586c5ea9aeaf349f13089c6fa37903 Mon Sep 17 00:00:00 2001
   2 From: "Zhang, Jun" <jun.zhang@intel.com>
   3 Date: Tue, 18 Dec 2018 06:55:01 -0800
   4 Subject: rcu: Do RCU GP kthread self-wakeup from softirq and interrupt
   5
   6 From: Zhang, Jun <jun.zhang@intel.com>
   7
   8 commit 1d1f898df6586c5ea9aeaf349f13089c6fa37903 upstream.
   9
  10 The rcu_gp_kthread_wake() function is invoked when it might be necessary
  11 to wake the RCU grace-period kthread.  Because self-wakeups are normally
  12 a useless waste of CPU cycles, if rcu_gp_kthread_wake() is invoked from
  13 this kthread, it naturally refuses to do the wakeup.
  14
  15 Unfortunately, natural though it might be, this heuristic fails when
  16 rcu_gp_kthread_wake() is invoked from an interrupt or softirq handler
  17 that interrupted the grace-period kthread just after the final check of
  18 the wait-event condition but just before the schedule() call.  In this
  19 case, a wakeup is required, even though the call to rcu_gp_kthread_wake()
  20 is within the RCU grace-period kthread's context.  Failing to provide
  21 this wakeup can result in grace periods failing to start, which in turn
  22 results in out-of-memory conditions.
  23
  24 This race window is quite narrow, but it actually did happen during real
  25 testing.  It would of course need to be fixed even if it was strictly
  26 theoretical in nature.
  27
  28 This patch does not Cc stable because it does not apply cleanly to
  29 earlier kernel versions.
  30
  31 Fixes: 48a7639ce80c ("rcu: Make callers awaken grace-period kthread")
  32 Reported-by: "He, Bo" <bo.he@intel.com>
  33 Co-developed-by: "Zhang, Jun" <jun.zhang@intel.com>
  34 Co-developed-by: "He, Bo" <bo.he@intel.com>
  35 Co-developed-by: "xiao, jin" <jin.xiao@intel.com>
  36 Co-developed-by: Bai, Jie A <jie.a.bai@intel.com>
  37 Signed-off: "Zhang, Jun" <jun.zhang@intel.com>
  38 Signed-off: "He, Bo" <bo.he@intel.com>
  39 Signed-off: "xiao, jin" <jin.xiao@intel.com>
  40 Signed-off: Bai, Jie A <jie.a.bai@intel.com>
  41 Signed-off-by: "Zhang, Jun" <jun.zhang@intel.com>
  42 [ paulmck: Switch from !in_softirq() to "!in_interrupt() &&
  43   !in_serving_softirq() to avoid redundant wakeups and to also handle the
  44   interrupt-handler scenario as well as the softirq-handler scenario that
  45   actually occurred in testing. ]
  46 Signed-off-by: Paul E. McKenney <paulmck@linux.ibm.com>
  47 Link: https://lkml.kernel.org/r/CD6925E8781EFD4D8E11882D20FC406D52A11F61@SHSMSX104.ccr.corp.intel.com
  48 Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
  49
  50
  51 ---
  52  kernel/rcu/tree.c |   20 ++++++++++++++------
  53  1 file changed, 14 insertions(+), 6 deletions(-)
  54
  55 --- a/kernel/rcu/tree.c
  56 +++ b/kernel/rcu/tree.c
  57 @@ -1689,15 +1689,23 @@ static bool rcu_future_gp_cleanup(struct
  58  }
  59
  60  /*
  61 - * Awaken the grace-period kthread for the specified flavor of RCU.
  62 - * Don't do a self-awaken, and don't bother awakening when there is
  63 - * nothing for the grace-period kthread to do (as in several CPUs
  64 - * raced to awaken, and we lost), and finally don't try to awaken
  65 - * a kthread that has not yet been created.
  66 + * Awaken the grace-period kthread.  Don't do a self-awaken (unless in
  67 + * an interrupt or softirq handler), and don't bother awakening when there
  68 + * is nothing for the grace-period kthread to do (as in several CPUs raced
  69 + * to awaken, and we lost), and finally don't try to awaken a kthread that
  70 + * has not yet been created.  If all those checks are passed, track some
  71 + * debug information and awaken.
  72 + *
  73 + * So why do the self-wakeup when in an interrupt or softirq handler
  74 + * in the grace-period kthread's context?  Because the kthread might have
  75 + * been interrupted just as it was going to sleep, and just after the final
  76 + * pre-sleep check of the awaken condition.  In this case, a wakeup really
  77 + * is required, and is therefore supplied.
  78   */
  79  static void rcu_gp_kthread_wake(struct rcu_state *rsp)
  80  {
  81 -       if (current == rsp->gp_kthread ||
  82 +       if ((current == rsp->gp_kthread &&
  83 +            !in_interrupt() && !in_serving_softirq()) ||
  84             !READ_ONCE(rsp->gp_flags) ||
  85             !rsp->gp_kthread)
  86                 return;