sched: Fix potentially missing balancing with Proxy Exec

K Prateek pointed out that with Proxy Exec, we may have cases
where we context switch in __schedule(), while the donor remains
the same. This could cause balancing issues, since the
put_prev_set_next() logic short-cuts if (prev == next). With
proxy-exec prev is the previous donor, and next is the next
donor. Should the donor remain the same, but different tasks are
picked to actually run, the shortcut will have avoided enqueuing
the sched class balance callback.

So, if we are context switching, add logic to catch the
same-donor case, and trigger the put_prev/set_next calls to
ensure the balance callbacks get enqueued.

Closes: https://lore.kernel.org/lkml/20ea3670-c30a-433b-a07f-c4ff98ae2379@amd.com/
Reported-by: K Prateek Nayak <kprateek.nayak@amd.com>
Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: John Stultz <jstultz@google.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/20260324191337.1841376-4-jstultz@google.com

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index f3306d3f2aa1d94e74d67d2452de668e297fc057..5b7f378af0422a215c6207d298399dcd63d6b108 100644 (file)
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -6826,9 +6826,11 @@ static void __sched notrace __schedule(int sched_mode)
 
 pick_again:
        next = pick_next_task(rq, rq->donor, &rf);
-       rq_set_donor(rq, next);
        rq->next_class = next->sched_class;
        if (sched_proxy_exec()) {
+               struct task_struct *prev_donor = rq->donor;
+
+               rq_set_donor(rq, next);
                if (unlikely(next->blocked_on)) {
                        next = find_proxy_task(rq, next, &rf);
                        if (!next)
@@ -6836,7 +6838,27 @@ pick_again:
                        if (next == rq->idle)
                                goto keep_resched;
                }
+               if (rq->donor == prev_donor && prev != next) {
+                       struct task_struct *donor = rq->donor;
+                       /*
+                        * When transitioning like:
+                        *
+                        *         prev         next
+                        * donor:    B            B
+                        * curr:     A          B or C
+                        *
+                        * then put_prev_set_next_task() will not have done
+                        * anything, since B == B. However, A might have
+                        * missed a RT/DL balance opportunity due to being
+                        * on_cpu.
+                        */
+                       donor->sched_class->put_prev_task(rq, donor, donor);
+                       donor->sched_class->set_next_task(rq, donor, true);
+               }
+       } else {
+               rq_set_donor(rq, next);
        }
+
 picked:
        clear_tsk_need_resched(prev);
        clear_preempt_need_resched();