rcu-tasks: Cancel callback laziness if too many callbacks

The various RCU Tasks flavors now do lazy grace periods when there are
only asynchronous grace period requests.  By default, the system will let
250 milliseconds elapse after the first call_rcu_tasks*() callbacki is
queued before starting a grace period.  In contrast, synchronous grace
period requests such as synchronize_rcu_tasks*() will start a grace
period immediately.

However, invoking one of the call_rcu_tasks*() functions in a too-tight
loop can result in a callback flood, which in turn can exhaust memory
if grace periods are delayed for too long.

This commit therefore sets a limit so that the grace-period kthread
will be awakened when any CPU's callback list expands to contain
rcupdate.rcu_task_lazy_lim callbacks elements (defaulting to 32, set to -1
to disable), the grace-period kthread will be awakened, thus cancelling
any ongoing laziness and getting out in front of the potential callback
flood.

Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Martin KaFai Lau <kafai@fb.com>
Cc: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index a0f427e1a5627d79637a973b0935778fb51ade93..44705a3618b95f42399794a4c83031e71ec232e3 100644 (file)
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -5264,6 +5264,13 @@
                        number avoids disturbing real-time workloads,
                        but lengthens grace periods.
 
+       rcupdate.rcu_task_lazy_lim= [KNL]
+                       Number of callbacks on a given CPU that will
+                       cancel laziness on that CPU.  Use -1 to disable
+                       cancellation of laziness, but be advised that
+                       doing so increases the danger of OOM due to
+                       callback flooding.
+
        rcupdate.rcu_task_stall_info= [KNL]
                        Set initial timeout in jiffies for RCU task stall
                        informational messages, which give some indication

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 28e986627e3fa45adf6d30ca7f7a5d9ed0e55837..291d536b1789005378f18bdd99cf9fbf1604c404 100644 (file)
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -176,6 +176,8 @@ static int rcu_task_contend_lim __read_mostly = 100;
 module_param(rcu_task_contend_lim, int, 0444);
 static int rcu_task_collapse_lim __read_mostly = 10;
 module_param(rcu_task_collapse_lim, int, 0444);
+static int rcu_task_lazy_lim __read_mostly = 32;
+module_param(rcu_task_lazy_lim, int, 0444);
 
 /* RCU tasks grace-period state for debugging. */
 #define RTGS_INIT               0
@@ -354,8 +356,9 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
                cblist_init_generic(rtp);
                raw_spin_lock_rcu_node(rtpcp); // irqs already disabled.
        }
-       needwake = func == wakeme_after_rcu;
-       if (havekthread && !timer_pending(&rtpcp->lazy_timer)) {
+       needwake = (func == wakeme_after_rcu) ||
+                  (rcu_segcblist_n_cbs(&rtpcp->cblist) == rcu_task_lazy_lim);
+       if (havekthread && !needwake && !timer_pending(&rtpcp->lazy_timer)) {
                if (rtp->lazy_jiffies)
                        mod_timer(&rtpcp->lazy_timer, rcu_tasks_lazy_time(rtp));
                else