From: Willy Tarreau Date: Wed, 8 Nov 2023 15:44:20 +0000 (+0100) Subject: BUG/MEDIUM: pool: try once to allocate from another bucket if empty X-Git-Tag: v2.9-dev10~134 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=a57f2a5cfe9f04f8efe8e1a0572e61753d545f4b;p=thirdparty%2Fhaproxy.git BUG/MEDIUM: pool: try once to allocate from another bucket if empty In order to limit inter-thread contention on the global pool, in 2.9-dev3 with commit 7bf829ace ("MAJOR: pools: move the shared pool's free_list over multiple buckets"), it was decided that if the selected bucket had an empty free list, we would simply give up and fall back to the OS allocator. But this causes allocations to be made from the OS for certain threads, to be released to overloaded pools that are sent back to the OS. One visible effect is that sending a lot of traffic using h2load with 100 parallel streams over 100 connections causes 5-10k buffers to be allocated, then reducing the load to only 10 connections doesn't make these allocations go down, just because some buckets are no longer visited. Tests show that giving a second chance to pick another bucket in this case is sufficient to visit all other buckets and recycle their pending objects. Now "show pools" that starts at 10k buffers at 100 connections goes down to about 150 with 1 connection and 100 streams in a fraction of a second. No backport is needed, as the issue is only in 2.9. --- diff --git a/src/pool.c b/src/pool.c index 75ffb1ec72..91774de576 100644 --- a/src/pool.c +++ b/src/pool.c @@ -672,9 +672,14 @@ void pool_refill_local_from_shared(struct pool_head *pool, struct pool_cache_hea bucket = pool_tbucket(); ret = _HA_ATOMIC_LOAD(&pool->buckets[bucket].free_list); + count = 0; do { - /* look for an apparently non-busy entry */ - while (unlikely(ret == POOL_BUSY)) { + /* look for an apparently non-busy entry. If we hit a busy pool + * we retry with another random bucket. And if we encounter a + * NULL, we retry once with another random bucket. This is in + * order to prevent object accumulation in other buckets. + */ + while (unlikely(ret == POOL_BUSY || (ret == NULL && count++ < 1))) { bucket = statistical_prng() % CONFIG_HAP_POOL_BUCKETS; ret = _HA_ATOMIC_LOAD(&pool->buckets[bucket].free_list); }