From: Willy Tarreau <w@1wt.eu>
Date: Wed, 8 Nov 2023 15:44:20 +0000 (+0100)
Subject: BUG/MINOR: pool: check one other random bucket on alloc conflict
X-Git-Tag: v2.9-dev10~135
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=a9ae094b271d8e8d2d71082f541a1e98b88421ee;p=thirdparty%2Fhaproxy.git

BUG/MINOR: pool: check one other random bucket on alloc conflict

Since 2.9-dev3 with commit 7bf829ace ("MAJOR: pools: move the shared
pool's free_list over multiple buckets"), the global pool supports
multiple heads to reduce inter-thread contention. However, when
grabbing a freelist head fails because another thread is already
picking from it, we just skip to the next one and try again.

Unfortunately, it still maintains a bit of contention between thread
pairs when for some reasons only a few threads are used. This may
happen for example when running on a 4- or 8- thread system and
the two most active ones end up on adjacent buckets.

A better and much simpler solution consists in visiting a random bucket
instead of the current one. Tests show that the CPU usage spent in
pool_refill_local_from_shared() reduces at low number of connections
(hence threads).

No backport is needed, as the issue is only in 2.9.
---

diff --git a/src/pool.c b/src/pool.c
index 02b5c6d250..75ffb1ec72 100644
--- a/src/pool.c
+++ b/src/pool.c
@@ -675,7 +675,7 @@ void pool_refill_local_from_shared(struct pool_head *pool, struct pool_cache_hea
 	do {
 		/* look for an apparently non-busy entry */
 		while (unlikely(ret == POOL_BUSY)) {
-			bucket = (bucket + 1) % CONFIG_HAP_POOL_BUCKETS;
+			bucket = statistical_prng() % CONFIG_HAP_POOL_BUCKETS;
 			ret = _HA_ATOMIC_LOAD(&pool->buckets[bucket].free_list);
 		}
 		if (ret == NULL)