BUG/MEDIUM: task/threads: address a fairness issue between local and global tasks

It is possible to hit a fairness issue in the scheduler when a local
task runs for a long time (i.e. process_stream() returns running), and
a global task wants to run on the same thread and remains in the global
queue. What happens in this case is that the condition to extract tasks
from the global queue will rarely be satisfied for very low task counts
since whatever non-null queue size multiplied by a thread count >1 is
always greater than the small remaining number of tasks in the queue.
In theory another thread should pick the task but we do have some mono
threaded tasks in the global queue as well during inter-thread wakeups.

Note that this can only happen with task counts lower than the thread
counts, typically one task in each queue for more than two threads.

This patch works around the problem by allowing a very small unfairness,
making sure that we can always pick at least one task from the global
queue even if there is already one in the local queue.

A better approach will consist in scanning the two trees in parallel
and always pick the best task. This will be more complex and will
constitute a separate patch.

This fix must be backported to 1.9.

diff --git a/src/task.c b/src/task.c
index 7f34bc5a318fa5782489c7e6ab92454ee7f12978..f8f9d995f2b6d70c28d4e38162b4a2703a132788 100644 (file)
--- a/src/task.c
+++ b/src/task.c
@@ -339,7 +339,7 @@ void process_runnable_tasks()
                 * than the average.
                 */
                rq_next = eb32sc_lookup_ge(&rqueue, rqueue_ticks - TIMER_LOOK_BACK, tid_bit);
-               while ((task_per_thread[tid].task_list_size + task_per_thread[tid].rqueue_size) * global.nbthread <= tasks_run_queue) {
+               while ((task_per_thread[tid].task_list_size + task_per_thread[tid].rqueue_size) * global.nbthread <= tasks_run_queue + global.nbthread - 1) {
                        if (unlikely(!rq_next)) {
                                /* either we just started or we reached the end
                                 * of the tree, typically because <rqueue_ticks>