sched/fair: Do not special case tasks in throttled hierarchy

With the introduction of task based throttle model, task in a throttled
hierarchy is allowed to continue to run till it gets throttled on its
ret2user path.

For this reason, remove those throttled_hierarchy() checks in the
following functions so that those tasks can get their turn as normal
tasks: dequeue_entities(), check_preempt_wakeup_fair() and
yield_to_task_fair().

The benefit of doing it this way is: if those tasks gets the chance to
run earlier and if they hold any kernel resources, they can release
those resources earlier. The downside is, if they don't hold any kernel
resouces, all they can do is to throttle themselves on their way back to
user space so the favor to let them run seems not that useful and for
check_preempt_wakeup_fair(), that favor may be bad for curr.

K Prateek Nayak pointed out prio_changed_fair() can send a throttled
task to check_preempt_wakeup_fair(), further tests showed the affinity
change path from move_queued_task() can also send a throttled task to
check_preempt_wakeup_fair(), that's why the check of task_is_throttled()
in that function.

Signed-off-by: Aaron Lu <ziqianlu@bytedance.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 58f5349d37256bf3e81dd0bdb77c30389f1533df..3dbdfaa69747765e874143858638dcee41f7fa2c 100644 (file)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -7081,7 +7081,7 @@ static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
                         * Bias pick_next to pick a task from this cfs_rq, as
                         * p is sleeping when it is within its sched_slice.
                         */
-                       if (task_sleep && se && !throttled_hierarchy(cfs_rq))
+                       if (task_sleep && se)
                                set_next_buddy(se);
                        break;
                }
@@ -8735,7 +8735,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int
         * lead to a throttle).  This both saves work and prevents false
         * next-buddy nomination below.
         */
-       if (unlikely(throttled_hierarchy(cfs_rq_of(pse))))
+       if (task_is_throttled(p))
                return;
 
        if (sched_feat(NEXT_BUDDY) && !(wake_flags & WF_FORK) && !pse->sched_delayed) {
@@ -9009,8 +9009,8 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)
 {
        struct sched_entity *se = &p->se;
 
-       /* throttled hierarchies are not runnable */
-       if (!se->on_rq || throttled_hierarchy(cfs_rq_of(se)))
+       /* !se->on_rq also covers throttled task */
+       if (!se->on_rq)
                return false;
 
        /* Tell the scheduler that we'd really like se to run next. */