From e049bd00ab9b8c65149413d40ef000555f75ba47 Mon Sep 17 00:00:00 2001 From: Willy Tarreau Date: Tue, 13 May 2025 17:58:28 +0200 Subject: [PATCH] MEDIUM: config: change default limits to 1024 threads and 32 groups A test run on a dual-socket EPYC 9845 (2x160 cores) showed that we'll be facing new limits during the lifetime of 3.2 with our current 16 groups and 256 threads max: $ cat test.cfg global cpu-policy perforamnce $ ./haproxy -dc -c -f test.cfg ... Thread CPU Bindings: Tgrp/Thr Tid CPU set 1/1-32 1-32 32: 0-15,320-335 2/1-32 33-64 32: 16-31,336-351 3/1-32 65-96 32: 32-47,352-367 4/1-32 97-128 32: 48-63,368-383 5/1-32 129-160 32: 64-79,384-399 6/1-32 161-192 32: 80-95,400-415 7/1-32 193-224 32: 96-111,416-431 8/1-32 225-256 32: 112-127,432-447 Raising the default limit to 1024 threads and 32 groups is sufficient to buy us enough margin for a long time (hopefully, please don't laugh, you, reader from the future): $ ./haproxy -dc -c -f test.cfg ... Thread CPU Bindings: Tgrp/Thr Tid CPU set 1/1-32 1-32 32: 0-15,320-335 2/1-32 33-64 32: 16-31,336-351 3/1-32 65-96 32: 32-47,352-367 4/1-32 97-128 32: 48-63,368-383 5/1-32 129-160 32: 64-79,384-399 6/1-32 161-192 32: 80-95,400-415 7/1-32 193-224 32: 96-111,416-431 8/1-32 225-256 32: 112-127,432-447 9/1-32 257-288 32: 128-143,448-463 10/1-32 289-320 32: 144-159,464-479 11/1-32 321-352 32: 160-175,480-495 12/1-32 353-384 32: 176-191,496-511 13/1-32 385-416 32: 192-207,512-527 14/1-32 417-448 32: 208-223,528-543 15/1-32 449-480 32: 224-239,544-559 16/1-32 481-512 32: 240-255,560-575 17/1-32 513-544 32: 256-271,576-591 18/1-32 545-576 32: 272-287,592-607 19/1-32 577-608 32: 288-303,608-623 20/1-32 609-640 32: 304-319,624-639 We can change this default now because it has no functional effect without any configured cpu-policy, so this will only be an opt-in and it's better to do it now than to have an effect during the maintenance phase. A tiny effect is a doubling of the number of pool buckets and stick-table shards internally, which means that aside slightly reducing contention in these areas, a dump of tables can enumerate keys in a different order (hence the adjustment in the vtc). The only really visible effect is a slightly higher static memory consumption (29->35 MB on a small config), but that difference remains even with 50k servers so that's pretty much acceptable. Thanks to Erwan Velu for the quick tests and the insights! --- include/haproxy/defaults.h | 4 ++-- .../http-rules/converters_ipmask_concat_strcmp_field_word.vtc | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/haproxy/defaults.h b/include/haproxy/defaults.h index 7eba11ef6b..4e012775a9 100644 --- a/include/haproxy/defaults.h +++ b/include/haproxy/defaults.h @@ -44,7 +44,7 @@ * doesn't engage us too far. */ #ifndef MAX_TGROUPS -#define MAX_TGROUPS 16 +#define MAX_TGROUPS 32 #endif #define MAX_THREADS_PER_GROUP __WORDSIZE @@ -53,7 +53,7 @@ * long bits if more tgroups are enabled. */ #ifndef MAX_THREADS -#define MAX_THREADS ((((MAX_TGROUPS) > 1) ? 4 : 1) * (MAX_THREADS_PER_GROUP)) +#define MAX_THREADS ((((MAX_TGROUPS) > 1) ? 16 : 1) * (MAX_THREADS_PER_GROUP)) #endif #endif // USE_THREAD diff --git a/reg-tests/http-rules/converters_ipmask_concat_strcmp_field_word.vtc b/reg-tests/http-rules/converters_ipmask_concat_strcmp_field_word.vtc index f19b7c148b..b2b8ae83ab 100644 --- a/reg-tests/http-rules/converters_ipmask_concat_strcmp_field_word.vtc +++ b/reg-tests/http-rules/converters_ipmask_concat_strcmp_field_word.vtc @@ -211,7 +211,7 @@ client c1 -connect ${h1_fe1_sock} -proxy2 "192.168.1.101:1234 127.0.0.1:2345" { # cli show be1 stick table haproxy h1 -cli { send "show table be1" - expect ~ "^# table: be1, type: ipv6, size:20, used:3\\n0x[a-f0-9]+: key=::ffff:192\\.168\\.1\\.0 use=0 exp=[[:digit:]]+ shard=0 gpc0=0 conn_cnt=1\\n0x[a-f0-9]+: key=::ffff:192\\.168\\.1\\.101 use=0 exp=[[:digit:]]+ shard=0 gpc0=0 conn_cnt=1\\n0x[a-f0-9]+: key=2001:db8:c001:c01a:[0:]+ use=0 exp=[[:digit:]]+ shard=0 gpc0=0 conn_cnt=1\\n" + expect ~ "^# table: be1, type: ipv6, size:20, used:3\\n0x[a-f0-9]+: key=::ffff:192\\.168\\.1\\.101 use=0 exp=[[:digit:]]+ shard=0 gpc0=0 conn_cnt=1\\n0x[a-f0-9]+: key=::ffff:192\\.168\\.1\\.0 use=0 exp=[[:digit:]]+ shard=0 gpc0=0 conn_cnt=1\\n0x[a-f0-9]+: key=2001:db8:c001:c01a:[0:]+ use=0 exp=[[:digit:]]+ shard=0 gpc0=0 conn_cnt=1\\n" } # concat,strcmp,word,field tests -- 2.47.2