sched/cache: Handle moving single tasks to/from their preferred LLC

Cache aware scheduling mainly does two things:
1. Prevent task from migrating out of its preferred LLC if not
   nessasary.
2. Migrating task to their preferred LLC if nessasary.

For 1:
In the generic load balance, if the busiest runqueue has only one task,
active balancing may be invoked to move it away. However, this migration
might break LLC locality.

Prevent regular load balance from migrating a task that
prefers the current LLC. The load level and imbalance do not warrant
breaking LLC preference per the can_migrate_llc() policy. Here, the
benefit of LLC locality outweighs the power efficiency gained from
migrating the only runnable task away.

Before migration, check whether the task is running on its preferred
LLC: Do not move a lone task to another LLC if it would move the task
away from its preferred LLC or cause excessive imbalance between LLCs.

For 2:
On the other hand, if the migration type is migrate_llc_task, it means
that there are tasks on the env->src_cpu that want to be migrated to
their preferred LLC, launch the active load balance anyway.

Co-developed-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Chen Yu <yu.c.chen@intel.com>
Signed-off-by: Tim Chen <tim.c.chen@linux.intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://patch.msgid.link/9b816d8c27fabf2a9c0e1f61a6b90afe8ec4ad52.1775065312.git.tim.c.chen@linux.intel.com

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 68032efd143b9d3f1dddc66228e8668b5e76a969..bfb6c0c5222136e2fba544b4b18ed8972ee3ca29 100644 (file)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -10293,12 +10293,60 @@ static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu
                               task_util(p), to_pref);
 }
 
+/*
+ * Check if active load balance breaks LLC locality in
+ * terms of cache aware load balance. The load level and
+ * imbalance do not warrant breaking LLC preference per
+ * the can_migrate_llc() policy. Here, the benefit of
+ * LLC locality outweighs the power efficiency gained from
+ * migrating the only runnable task away.
+ */
+static inline bool
+alb_break_llc(struct lb_env *env)
+{
+       if (!sched_cache_enabled())
+               return false;
+
+       if (cpus_share_cache(env->src_cpu, env->dst_cpu))
+               return false;
+       /*
+        * All tasks prefer to stay on their current CPU.
+        * Do not pull a task from its preferred CPU if:
+        * 1. It is the only task running there(not too imbalance); OR
+        * 2. Migrating it away from its preferred LLC would violate
+        *    the cache-aware scheduling policy.
+        */
+       if (env->src_rq->nr_pref_llc_running &&
+           env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable) {
+               unsigned long util = 0;
+               struct task_struct *cur;
+
+               if (env->src_rq->nr_running <= 1)
+                       return true;
+
+               cur = rcu_dereference_all(env->src_rq->curr);
+               if (cur)
+                       util = task_util(cur);
+
+               if (can_migrate_llc(env->src_cpu, env->dst_cpu,
+                                   util, false) == mig_forbid)
+                       return true;
+       }
+
+       return false;
+}
 #else
 static inline bool get_llc_stats(int cpu, unsigned long *util,
                                 unsigned long *cap)
 {
        return false;
 }
+
+static inline bool
+alb_break_llc(struct lb_env *env)
+{
+       return false;
+}
 #endif
 /*
  * can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
@@ -12698,6 +12746,9 @@ static int need_active_balance(struct lb_env *env)
 {
        struct sched_domain *sd = env->sd;
 
+       if (alb_break_llc(env))
+               return 0;
+
        if (asym_active_balance(env))
                return 1;
 
@@ -12717,7 +12768,8 @@ static int need_active_balance(struct lb_env *env)
                        return 1;
        }
 
-       if (env->migration_type == migrate_misfit)
+       if (env->migration_type == migrate_misfit ||
+           env->migration_type == migrate_llc_task)
                return 1;
 
        return 0;