rcu-tasks: Repair RCU Tasks Trace quiescence check

[ Upstream commit 2eb52fa8900e642b3b5054c4bf9776089d2a935f ]

The context-switch-time check for RCU Tasks Trace quiescence expects
current->trc_reader_special.b.need_qs to be zero, and if so, updates
it to TRC_NEED_QS_CHECKED.  This is backwards, because if this value
is zero, there is no RCU Tasks Trace grace period in flight, an thus
no need for a quiescent state.  Instead, when a grace period starts,
this field is set to TRC_NEED_QS.

This commit therefore changes the check from zero to TRC_NEED_QS.

Reported-by: Steven Rostedt <rostedt@goodmis.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Tested-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>

diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 9ad656b28847f71d21f7e1596a5cb7fd3cbbd787..6466c2f792923c1e8cbf470a63394886ed1f3071 100644 (file)
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -189,9 +189,9 @@ void rcu_tasks_trace_qs_blkd(struct task_struct *t);
        do {                                                                    \
                int ___rttq_nesting = READ_ONCE((t)->trc_reader_nesting);       \
                                                                                \
-               if (likely(!READ_ONCE((t)->trc_reader_special.b.need_qs)) &&    \
+               if (unlikely(READ_ONCE((t)->trc_reader_special.b.need_qs) == TRC_NEED_QS) &&    \
                    likely(!___rttq_nesting)) {                                 \
-                       rcu_trc_cmpxchg_need_qs((t), 0, TRC_NEED_QS_CHECKED);   \
+                       rcu_trc_cmpxchg_need_qs((t), TRC_NEED_QS, TRC_NEED_QS_CHECKED); \
                } else if (___rttq_nesting && ___rttq_nesting != INT_MIN &&     \
                           !READ_ONCE((t)->trc_reader_special.b.blocked)) {     \
                        rcu_tasks_trace_qs_blkd(t);                             \