--- /dev/null
+2023-07-04 - automatic grouping for NUMA
+
+
+Xeon: (W2145)
+
+willy@debian:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,8
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,8
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,8
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-15
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+
+Wtap: i7-8650U
+
+willy@wtap:~ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,4
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,4
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,4
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+
+pcw: i7-6700k
+
+willy@pcw:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,4
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,4
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,4
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+
+nfs: N5105, v5.15
+
+willy@nfs:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-3
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-3
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+
+eeepc: Atom N2800, 5.4 : no L3, L2 not shared.
+
+willy@eeepc:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+
+willy@eeepc:~$ grep '' /sys/devices/system/cpu/cpu2/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_list:2-3
+/sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_list:2-3
+/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list:2-3
+/sys/devices/system/cpu/cpu2/cache/index0/type:Data
+/sys/devices/system/cpu/cpu2/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu2/cache/index2/type:Unified
+
+
+dev13: Ryzen 2700X
+
+haproxy@dev13:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+haproxy@dev13:~$ grep '' /sys/devices/system/cpu/cpu8/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu8/cache/index0/shared_cpu_list:8-9
+/sys/devices/system/cpu/cpu8/cache/index1/shared_cpu_list:8-9
+/sys/devices/system/cpu/cpu8/cache/index2/shared_cpu_list:8-9
+/sys/devices/system/cpu/cpu8/cache/index3/shared_cpu_list:8-15
+/sys/devices/system/cpu/cpu8/cache/index0/type:Data
+/sys/devices/system/cpu/cpu8/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu8/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu8/cache/index3/type:Unified
+
+
+dev12: Ryzen 5800X
+
+haproxy@dev12:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,8
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,8
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,8
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-15
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+
+amd24: EPYC 74F3
+
+willy@mt:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,24
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,24
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,24
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-2,24-26
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+willy@mt:~$ grep '' /sys/devices/system/cpu/cpu8/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu8/cache/index0/shared_cpu_list:8,32
+/sys/devices/system/cpu/cpu8/cache/index1/shared_cpu_list:8,32
+/sys/devices/system/cpu/cpu8/cache/index2/shared_cpu_list:8,32
+/sys/devices/system/cpu/cpu8/cache/index3/shared_cpu_list:6-8,30-32
+/sys/devices/system/cpu/cpu8/cache/index0/type:Data
+/sys/devices/system/cpu/cpu8/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu8/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu8/cache/index3/type:Unified
+
+willy@mt:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0,24
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-47
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0-47
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-47
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,24
+
+
+xeon24: Gold 6212U
+
+willy@mt01:~$ grep '' /sys/devices/system/cpu/cpu8/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu8/cache/index0/shared_cpu_list:8,32
+/sys/devices/system/cpu/cpu8/cache/index1/shared_cpu_list:8,32
+/sys/devices/system/cpu/cpu8/cache/index2/shared_cpu_list:8,32
+/sys/devices/system/cpu/cpu8/cache/index3/shared_cpu_list:0-47
+/sys/devices/system/cpu/cpu8/cache/index0/type:Data
+/sys/devices/system/cpu/cpu8/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu8/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu8/cache/index3/type:Unified
+
+
+SPR 8480+
+
+$ grep -a '' /sys/devices/system/node/node*/cpulist
+/sys/devices/system/node/node0/cpulist:0-55,112-167
+/sys/devices/system/node/node1/cpulist:56-111,168-223
+
+$ grep -a '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0,112
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-55,112-167
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0-55,112-167
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-55,112-167
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0,112
+
+$ grep -a '' /sys/devices/system/cpu/cpu0/cache/*/shared_cpu_list
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0,112
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0,112
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0,112
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-55,112-167
+
+
+UP Board - Atom X5-8350 : no L3, exactly like Armada8040
+
+willy@up1:~$ grep '' /sys/devices/system/cpu/cpu{0,1,2,3}/cache/index2/*list
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu1/cache/index2/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list:2-3
+/sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_list:2-3
+
+willy@up1:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+Atom D510 - kernel 2.6.33
+
+$ strings -fn1 sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list: 0,2
+sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list: 0,2
+sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list: 0,2
+sys/devices/system/cpu/cpu0/cache/index0/type: Data
+sys/devices/system/cpu/cpu0/cache/index1/type: Instruction
+sys/devices/system/cpu/cpu0/cache/index2/type: Unified
+
+$ strings -fn1 sys/devices/system/cpu/cpu?/topology/*list
+sys/devices/system/cpu/cpu0/topology/core_siblings_list: 0-3
+sys/devices/system/cpu/cpu0/topology/thread_siblings_list: 0,2
+sys/devices/system/cpu/cpu1/topology/core_siblings_list: 0-3
+sys/devices/system/cpu/cpu1/topology/thread_siblings_list: 1,3
+sys/devices/system/cpu/cpu2/topology/core_siblings_list: 0-3
+sys/devices/system/cpu/cpu2/topology/thread_siblings_list: 0,2
+sys/devices/system/cpu/cpu3/topology/core_siblings_list: 0-3
+sys/devices/system/cpu/cpu3/topology/thread_siblings_list: 1,3
+
+mcbin: Armada 8040 : no L3, no difference with L3 not reported
+
+root@lg7:~# grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+
+root@lg7:~# grep '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-3
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+
+Ampere/monolithic: Ampere Altra 80-26 : L3 not reported
+
+willy@ampere:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+
+willy@ampere:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-79
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-79
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+
+Ampere/Hemisphere: Ampere Altra 80-26 : L3 not reported
+
+willy@ampere:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+
+willy@ampere:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-79
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-79
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+willy@ampere:~$ grep '' /sys/devices/system/node/node*/cpulist
+/sys/devices/system/node/node0/cpulist:0-39
+/sys/devices/system/node/node1/cpulist:40-79
+
+
+LX2A: LX2160A => L3 not reported
+
+willy@lx2a:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-1
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+
+willy@lx2a:~$ grep '' /sys/devices/system/cpu/cpu2/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu2/cache/index0/shared_cpu_list:2
+/sys/devices/system/cpu/cpu2/cache/index1/shared_cpu_list:2
+/sys/devices/system/cpu/cpu2/cache/index2/shared_cpu_list:2-3
+/sys/devices/system/cpu/cpu2/cache/index0/type:Data
+/sys/devices/system/cpu/cpu2/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu2/cache/index2/type:Unified
+
+willy@lx2a:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-15
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-15
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+
+Rock5B: RK3588 (big-little A76+A55)
+
+rock@rock-5b:~$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index3/shared_cpu_list:0-7
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+/sys/devices/system/cpu/cpu0/cache/index3/type:Unified
+
+rock@rock-5b:~$ grep '' /sys/devices/system/cpu/cpu{0,4,6}/topology/*list
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-3
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+/sys/devices/system/cpu/cpu4/topology/core_cpus_list:4
+/sys/devices/system/cpu/cpu4/topology/core_siblings_list:4-5
+/sys/devices/system/cpu/cpu4/topology/die_cpus_list:4
+/sys/devices/system/cpu/cpu4/topology/package_cpus_list:4-5
+/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4
+/sys/devices/system/cpu/cpu6/topology/core_cpus_list:6
+/sys/devices/system/cpu/cpu6/topology/core_siblings_list:6-7
+/sys/devices/system/cpu/cpu6/topology/die_cpus_list:6
+/sys/devices/system/cpu/cpu6/topology/package_cpus_list:6-7
+/sys/devices/system/cpu/cpu6/topology/thread_siblings_list:6
+
+$ grep '' /sys/devices/system/cpu/cpu*/cpu_capacity
+/sys/devices/system/cpu/cpu0/cpu_capacity:414
+/sys/devices/system/cpu/cpu1/cpu_capacity:414
+/sys/devices/system/cpu/cpu2/cpu_capacity:414
+/sys/devices/system/cpu/cpu3/cpu_capacity:414
+/sys/devices/system/cpu/cpu4/cpu_capacity:1024
+/sys/devices/system/cpu/cpu5/cpu_capacity:1024
+/sys/devices/system/cpu/cpu6/cpu_capacity:1024
+/sys/devices/system/cpu/cpu7/cpu_capacity:1024
+
+
+Firefly: RK3399 (2xA72 + 4xA53) kernel 6.1.28
+
+root@firefly:~# grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+grep: /sys/devices/system/cpu/cpu0/cache/index?/shared_cpu_list: No such file or directory
+grep: /sys/devices/system/cpu/cpu0/cache/index?/type: No such file or directory
+
+root@firefly:~# grep '' /sys/devices/system/cpu/cpu*/cache/index?/{shared_cpu_list,type}
+grep: /sys/devices/system/cpu/cpu*/cache/index?/shared_cpu_list: No such file or directory
+grep: /sys/devices/system/cpu/cpu*/cache/index?/type: No such file or directory
+
+root@firefly:~# dmesg|grep cacheinfo
+[ 0.006290] cacheinfo: Unable to detect cache hierarchy for CPU 0
+[ 0.016339] cacheinfo: Unable to detect cache hierarchy for CPU 1
+[ 0.017692] cacheinfo: Unable to detect cache hierarchy for CPU 2
+[ 0.019050] cacheinfo: Unable to detect cache hierarchy for CPU 3
+[ 0.020478] cacheinfo: Unable to detect cache hierarchy for CPU 4
+[ 0.021660] cacheinfo: Unable to detect cache hierarchy for CPU 5
+[ 1.990108] cacheinfo: Unable to detect cache hierarchy for CPU 0
+
+root@firefly:~# grep '' /sys/devices/system/cpu/cpu0/topology/*
+/sys/devices/system/cpu/cpu0/topology/cluster_cpus:0f
+/sys/devices/system/cpu/cpu0/topology/cluster_cpus_list:0-3
+/sys/devices/system/cpu/cpu0/topology/cluster_id:0
+/sys/devices/system/cpu/cpu0/topology/core_cpus:01
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_id:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings:3f
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-5
+/sys/devices/system/cpu/cpu0/topology/package_cpus:3f
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-5
+/sys/devices/system/cpu/cpu0/topology/physical_package_id:0
+/sys/devices/system/cpu/cpu0/topology/thread_siblings:01
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+$ grep '' /sys/devices/system/cpu/cpu*/cpu_capacity
+/sys/devices/system/cpu/cpu0/cpu_capacity:381
+/sys/devices/system/cpu/cpu1/cpu_capacity:381
+/sys/devices/system/cpu/cpu2/cpu_capacity:381
+/sys/devices/system/cpu/cpu3/cpu_capacity:381
+/sys/devices/system/cpu/cpu4/cpu_capacity:1024
+/sys/devices/system/cpu/cpu5/cpu_capacity:1024
+
+
+VIM3L: S905D3 (4*A55), kernel 5.14.10
+
+$ grep '' /sys/devices/system/cpu/cpu0/topology/*
+/sys/devices/system/cpu/cpu0/topology/core_cpus:1
+/sys/devices/system/cpu/cpu0/topology/core_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/core_id:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings:f
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-3
+/sys/devices/system/cpu/cpu0/topology/die_cpus:1
+/sys/devices/system/cpu/cpu0/topology/die_cpus_list:0
+/sys/devices/system/cpu/cpu0/topology/die_id:-1
+/sys/devices/system/cpu/cpu0/topology/package_cpus:f
+/sys/devices/system/cpu/cpu0/topology/package_cpus_list:0-3
+/sys/devices/system/cpu/cpu0/topology/physical_package_id:0
+/sys/devices/system/cpu/cpu0/topology/thread_siblings:1
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+$ grep '' /sys/devices/system/cpu/cpu0/cache/index?/{shared_cpu_list,type}
+/sys/devices/system/cpu/cpu0/cache/index0/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index1/shared_cpu_list:0
+/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_list:0-3
+/sys/devices/system/cpu/cpu0/cache/index0/type:Data
+/sys/devices/system/cpu/cpu0/cache/index1/type:Instruction
+/sys/devices/system/cpu/cpu0/cache/index2/type:Unified
+
+$ grep '' /sys/devices/system/cpu/cpu*/cpu_capacity
+/sys/devices/system/cpu/cpu0/cpu_capacity:1024
+/sys/devices/system/cpu/cpu1/cpu_capacity:1024
+/sys/devices/system/cpu/cpu2/cpu_capacity:1024
+/sys/devices/system/cpu/cpu3/cpu_capacity:1024
+
+
+Odroid-N2: S922X (4*A73 + 2*A53), kernel 4.9.254
+
+willy@n2:~$ grep '' /sys/devices/system/cpu/cpu*/cache/index?/{shared_cpu_list,type}
+grep: /sys/devices/system/cpu/cpu*/cache/index?/shared_cpu_list: No such file or directory
+grep: /sys/devices/system/cpu/cpu*/cache/index?/type: No such file or directory
+
+willy@n2:~$ sudo dmesg|grep -i 'cache hi'
+[ 0.649924] Unable to detect cache hierarchy for CPU 0
+
+No capacity.
+
+Note that it reports 2 physical packages!
+
+willy@n2:~$ grep '' /sys/devices/system/cpu/cpu0/topology/*
+/sys/devices/system/cpu/cpu0/topology/core_id:0
+/sys/devices/system/cpu/cpu0/topology/core_siblings:03
+/sys/devices/system/cpu/cpu0/topology/core_siblings_list:0-1
+/sys/devices/system/cpu/cpu0/topology/physical_package_id:0
+/sys/devices/system/cpu/cpu0/topology/thread_siblings:01
+/sys/devices/system/cpu/cpu0/topology/thread_siblings_list:0
+
+willy@n2:~$ grep '' /sys/devices/system/cpu/cpu4/topology/*
+/sys/devices/system/cpu/cpu4/topology/core_id:2
+/sys/devices/system/cpu/cpu4/topology/core_siblings:3c
+/sys/devices/system/cpu/cpu4/topology/core_siblings_list:2-5
+/sys/devices/system/cpu/cpu4/topology/physical_package_id:1
+/sys/devices/system/cpu/cpu4/topology/thread_siblings:10
+/sys/devices/system/cpu/cpu4/topology/thread_siblings_list:4
+
+StarFive VisionFive2 - JH7110, kernel 5.15
+
+willy@starfive:~/haproxy$ ./haproxy -c -f cps3.cfg
+thr 0 -> cpu 0 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=000 l1=000
+thr 1 -> cpu 1 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=001 l1=001
+thr 2 -> cpu 2 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=002 l1=002
+thr 3 -> cpu 3 onl=1 bnd=1 pk=00 no=-1 l3=-1 cl=000 l2=000 ts=003 l1=003
+Configuration file is valid
+
+Graviton2 / Graviton3 ?
+
+
+On PPC64 not everything is available:
+
+ https://www.ibm.com/docs/en/linux-on-systems?topic=cpus-cpu-topology
+
+ /sys/devices/system/cpu/cpu<N>/topology/thread_siblings
+ /sys/devices/system/cpu/cpu<N>/topology/core_siblings
+ /sys/devices/system/cpu/cpu<N>/topology/book_siblings
+ /sys/devices/system/cpu/cpu<N>/topology/drawer_siblings
+
+ # lscpu -e
+ CPU NODE DRAWER BOOK SOCKET CORE L1d:L1i:L2d:L2i ONLINE CONFIGURED POLARIZATION ADDRESS
+ 0 1 0 0 0 0 0:0:0:0 yes yes horizontal 0
+ 1 1 0 0 0 0 1:1:1:1 yes yes horizontal 1
+ 2 1 0 0 0 1 2:2:2:2 yes yes horizontal 2
+ 3 1 0 0 0 1 3:3:3:3 yes yes horizontal 3
+ 4 1 0 0 0 2 4:4:4:4 yes yes horizontal 4
+ 5 1 0 0 0 2 5:5:5:5 yes yes horizontal 5
+ 6 1 0 0 0 3 6:6:6:6 yes yes horizontal 6
+ 7 1 0 0 0 3 7:7:7:7 yes yes horizontal 7
+ 8 0 1 1 1 4 8:8:8:8 yes yes horizontal 8
+ ...
+
+Intel E5-2600v2/v3 has two L3:
+ https://www.enterpriseai.news/2014/09/08/intel-ups-performance-ante-haswell-xeon-chips/
+
+More info on these, and s390's "books" (mostly L4 in fact):
+ https://groups.google.com/g/fa.linux.kernel/c/qgAxjYq8ohI
+
+########################################
+Analysis:
+ - some server ARM CPUs (Altra, LX2) do not return any L3 info though they
+ DO have some. They stop at L2.
+
+ - other CPUs like Atom N2800 and Armada 8040 do not have L3.
+
+ => there's no apparent way to detect that the server CPUs do have an L3.
+ => or maybe we should consider that it's more likely that there is one
+ than none ? Armada works much better with groups than without. It's
+ basically the same topology as N2800.
+
+ => Do we really care then ? No L3 = same L3 for everyone. The problem is
+ that those really without L3 will make a difference on L2 while the
+ other ones not. Maybe we should consider that it does not make sense
+ to cut groups on L2 (i.e. under no circumstance we'll have one group
+ per core).
+
+ => This would mean:
+ - regardless of L3, consider LLC. If the LLC has more than one
+ core per instance, it's likely the last one (not true on LX2
+ but better use 8 groups of 2 than nothing).
+
+ - otherwise if there's a single core per instance, it's unlikely
+ to be the LLC so we can imagine the LLC is unified. Note that
+ some systems such as LX2/Armada8K (and Neoverse-N1 devices as
+ well) may have 2 cores per L2, yet this doesn't allow to infer
+ anything regarding the absence of an L3. Core2-quad has 2 cores
+ per L2 with no L3, like Armada8K. LX2 has 2 cores per L2 yet does
+ have an L3 which is not necessarily reported.
+
+ - this needs to be done per {node,package} !
+ => core_siblings and thread_siblings seem to be the only portable
+ ones to figure packages and threads
+
+At the very least, when multiple nodes are possibly present, there is a
+symlink "node0", "node1" etc in the cpu entry. It requires a lookup for each
+cpu directory though while reading /sys/devices/system/node/node*/cpulist is
+much cheaper.
+
+There's some redundancy in this. Probably better approach:
+
+1) if there is more than 1 CPU:
+ - if cache/index3 exists, use its cpulist to pre-group entries.
+ - else if topology or node exists, use (node,package,die,core_siblings) to
+ group entries
+ - else pre-create a single large group
+
+2) if there is more than 1 CPU and less than max#groups:
+ - for each group, if no cache/index3 exists and cache/index2 exists and some
+ index2 entries contain at least two CPUs of different cores or a single one
+ for a 2-core system, then use that to re-split the group.
+
+ - if in the end there are too many groups, remerge some of them (?) or stick
+ to the previous layout (?)
+
+ - if in the end there are too many CPUs in a group, cut as needed, if
+ possible with an integral result (/2, /3, ...)
+
+3) L1 cache / thread_siblings should be used to associate CPUs by cores in
+ the same groups.
+
+Maybe instead it should be done bottom->top by collecting info and merging
+groups while keeping CPU lists ordered to ease later splitting.
+
+ 1) create a group per bound CPU
+ 2) based on thread_siblings, detect CPUs that are on the same core, merge
+ their groups. They may not always create similarly sized groups.
+ => eg: epyc keeps 24 groups such as {0,24}, ...
+ ryzen 2700x keeps 4 groups such as {0,1}, ...
+ rk3588 keeps 3 groups {0-3},{4-5},{6-7}
+ 3) based on cache index0/1, detect CPUs that are on the same L1 cache,
+ merge their groups. They may not always create similarly sized groups.
+ 4) based on cache index2, detect CPUs that are on the same L2 cache, merge
+ their groups. They may not always create similarly sized groups.
+ => eg: mcbin now keeps 2 groups {0-1},{2,3}
+ 5) At this point there may possibly be too many groups (still one per CPU,
+ e.g. when no cache info was found or there are many cores with their own
+ L2 like on SPR) or too large one (when all cores are indeed on the same
+ L2).
+
+ 5.1) if there are as many groups as bound CPUs, merge them all together in
+ a single one => lx2, altra, mcbin
+ 5.2) if there are still more than max#groups, merge them all together in a
+ single one since the splitting criterion is not relevant
+ 5.3) if there is a group with too many CPUs, split it in two if integral,
+ otherwise 3, etc, trying to add the least possible number of groups.
+ If too difficult (e.g. result less than half the authorized max),
+ let's just round around N/((N+63)/64).
+ 5.4) if at the end there are too many groups, warn that we can't optimize
+ the setup and are limiting ourselves to the first node or 64 CPUs.
+
+Observations:
+ - lx2 definitely works better with everything bound together than by creating
+ 8 groups (~130k rps vs ~120k rps)
+ => does this mean we should assume a unified L3 if there's no L3 info, and
+ remerge everything ? Likely Altra would benefit from this as well. mcbin
+ doesn't notice any change (within noise in both directions)
+
+ - on x86 13th gen, 2 P-cores and 8 E-cores. The P-cores support HT, not the
+ E-cores. There's no cpu_capacity there, but the cluster_id is properly set.
+ => proposal: when a machine reports both single-threaded cores and SMT,
+ consider the SMT ones bigger and use them.
+
+Problems: how should auto-detection interfer with user-settings ?
+
+- Case 1: program started with a reduced taskset
+ => current: this serves to the the thread count first, and to map default
+ threads to CPUs if they are not affected by a cpu-map.
+
+ => we want to keep that behavior (i.e. use all these threads) but only
+ change how the thread-groups are arranged.
+
+ - example: start on the first 6c12t of an EPYC74F3, should automatically
+ create 2 groups for the two sockets.
+
+ => should we brute-force all thread-groups combinations to figure how the
+ threads will spread over cpu-map and which one is better ? Or should we
+ decide to ignore input mapping as soon as there's at least one cpu-map?
+ But then which one to use ? Or should we consider that cpu-map only works
+ with explicit thread-groups ?
+
+- Case 2: taskset not involved, but nbthread and cpu-map in the config. In
+ fact a pretty standard 2.4-2.8 config.
+ => maybe the presence of cpu-map and no thread-groups should be sufficient
+ to imply a single thread-group to stay compatible ? Or maybe start as
+ many thread-groups as are referenced in cpu-map ? Seems like cpu-map and
+ thread-groups work hand-in-hand regarding topology since cpu-map
+ designates hardware CPUs so the user knows better than haproxy. Thus
+ why should be try to do better ?
+
+- Case 3: taskset not involved, nbthread not involved, cpu-map not involved,
+ only thread-groups
+ => seems like an ideal approach. Take all online CPUs and try to cut them
+ into equitable thread groups ? Or rather, since nbthreads is not forced,
+ better sort the clusters and bind to the N first clusters only ? If too
+ many groups for the clusters, then try to refine them ?
+
+- Case 4: nothing specified at all (default config, target)
+ => current: uses only one thread-group with all threads (max 64).
+ => desired: bind only to performance cores and cut them in a few groups
+ based on l3, package, cluster etc.
+
+- Case 5: nbthread only in the config
+ => might match a docker use case. No group nor cpu-map configured. Figure
+ the best group usage respecting the thread count.
+
+- Case 6: some constraints are enforced in the config (e.g. threads-hard-limit,
+ one-thread-per-core, etc).
+ => like 3, 4 or 5 but with selection adjustment.
+
+- Case 7: thread-groups and generic cpu-map 1/all, 2/all... in the config
+ => user just wants to use cpu-map as a taskset alternative
+ => need to figure number of threads first, then cut them in groups like
+ today, and only then the cpu-map are found. Can we do better ? Not sure.
+ Maybe just when cpu-map is too lax (e.g. all entries reference the same
+ CPUs). Better use a special "cpumap all/all 0-19" for this, but not
+ implemented for now.
+
+Proposal:
+ - if there is any cpu-map, disable automatic CPU assignment
+ - if there is any cpu-map, disable automatic thread group detection
+ - if taskset was forced, disable automatic CPU assignment
+
+### 2023-07-17 ###
+
+=> step 1: mark CPUs enabled at boot (cpu_detect_usable)
+// => step 2: mark CPUs referenced in cpu-map => no, no real meaning
+=> step 3: identify all CPUs topologies + NUMA (cpu_detect_topology)
+
+=> step 4: if taskset && !cpu-map, mark all non-bound CPUs as unusable (UNAVAIL ?)
+ => which is the same as saying if !cpu-map.
+=> step 5: if !cpu-map, sort usable CPUs and find the best set to use
+//=> step 6: if cpu-map, mark all non-covered CPUs are unusable => not necessarily possible if partial cpu-map
+
+=> step 7: if thread-groups && cpu-map, nothing else to do
+=> step 8: if cpu-map && !thread-groups, thread-groups=1
+=> step 9: if thread-groups && !cpu-map, use that value to cut the thread set
+=> step 10: if !cpu-map && !thread-groups, detect the optimal thread-group count
+
+=> step 11: if !cpu-map, cut the thread set into mostly fair groups and assign
+ the group numbers to CPUs; create implicit cpu-maps.
+
+Ideas:
+ - use minthr and maxthr.
+ If nbthread, minthr=maxthr=nbthread, else if taskset_forced, maxthr=taskset_thr,
+ minthr=1, else minthr=1, maxthr=cpus_enabled.
+
+ - use CPU_F_ALLOWED (or DISALLOWED?) and CPU_F_REFERENCED and CPU_F_EXCLUDED ?
+ Note: cpu-map doesn't exclude, it only includes. Taskset does exclude. Also,
+ cpu-map only includes the CPUs that will belong to the correct groups & threads.
+
+ - Usual startup: taskset presets the CPU sets and sets the thread count. Tgrp
+ defaults to 1, then threads indicated in cpu-map get their CPU assigned.
+ Other ones are not changed. If we say that cpu-map => tgrp==1 then it means
+ we can infer automatic grouping for group 1 only ?
+ => it could be said that the CPUs of all enabled groups mentioned in
+ cpu-map are considered usable, but we don't know how many of these
+ will really have threads started on.
+
+ => maybe completely ignore cpu-map instead (i.e. fall back to thread-groups 1) ?
+ => automatic detection would mean:
+ - if !cpu-map && !nbthrgrp => must automatically detect thgrp
+ - if !cpu-map => must automatically detect binding
+ - otherwise nothing
+
+Examples of problems:
+
+ thread-groups 4
+ nbthreads 128
+ cpu-map 1/all 0-63
+ cpu-map 2/all 128-191
+
+ => 32 threads per group, hence grp 1 uses 0-63 and grp 2 128-191,
+ grp 3 and grp 4 unknown, in practice on boot CPUs.
+
+ => could we demand that if one cpu-map is specified, then all groups
+ are covered ? Do we need really this after all ? i.e. let's just not
+ bind other threads and that's all (and what is written).
+
+
+Calls from haproxy.c:
+
+ cpu_detect_usable()
+ cpu_detect_topology()
+
++ thread_detect_count()
+ => compute nbtgroups
+ => compute nbthreads
+
+ thread_assign_cpus() ?
+
+ check_config_validity()
+
+
+BUGS:
+ - cpu_map[0].proc still used for the whole process in daemon mode (though not
+ in foreground mode)
+ -> whole process bound to thread group 1
+ -> binding not working in foreground
+
+ - cpu_map[x].proc ANDed with the thread's map depite thread's map apparently
+ never set
+ -> group binding ignored ?
+
+2023-09-05
+----------
+Remember to make the difference between sorting (used for grouping) and
+preference. We should avoid selecting the first CPUs as it encourages to
+use wrong grouping criteria. E.g. CPU capacity has no business being used
+for grouping, it's used for selecting. Support for HT however, does because
+it allows to pack together threads of the same core.
+
+We should also have an option to enable/disable SMT (e.g. max threads per core)
+so that we can skip siblings of cores already assigned. This can be convenient
+with network running on the other sibling.
+
+
+2024-12-26
+----------
+
+Some interesting cases about intel 14900. The CPU has 8 P-cores and 16 E-cores.
+Experiments in the lab show excellent performance by binding the network to E
+cores and haproxy to P cores. Here's how the clusters are made:
+
+$ grep -h . /sys/devices/system/cpu/cpu*/topology/package_cpus | sort |uniq -c
+ 32 ffffffff
+
+ => expected
+
+$ grep -h . /sys/devices/system/cpu/cpu*/topology/die_cpus | sort |uniq -c
+ 32 ffffffff
+
+ => all CPUs on the same die
+
+$ grep -h . /sys/devices/system/cpu/cpu*/topology/cluster_cpus | sort |uniq -c
+ 2 00000003
+ 2 0000000c
+ 2 00000030
+ 2 000000c0
+ 2 00000300
+ 2 00000c00
+ 2 00003000
+ 2 0000c000
+ 4 000f0000
+ 4 00f00000
+ 4 0f000000
+ 4 f0000000
+
+ => 1 "cluster" per core on each P-core (2 threads, 8 clusters total)
+ => 1 "cluster" per 4 E-cores (4 clusters total)
+ => It can be difficult to split that into groups by just using this topology.
+
+$ grep -h . /sys/devices/system/cpu/cpu*/cache/index3/shared_cpu_list | sort |uniq -c
+ 32 0-31
+
+ => everyone shares a uniform L3 cache
+
+$ grep -h . /sys/devices/system/cpu/cpu*/cache/index2/shared_cpu_map | sort |uniq -c
+ 2 00000003
+ 2 0000000c
+ 2 00000030
+ 2 000000c0
+ 2 00000300
+ 2 00000c00
+ 2 00003000
+ 2 0000c000
+ 4 000f0000
+ 4 00f00000
+ 4 0f000000
+ 4 f0000000
+
+ => L2 is split like the respective "clusters" above.
+
+Semms like one would like to split them into 12 groups :-/ Maybe it still
+remains relevant to consider L3 for grouping, and core performance for
+selection (e.g. evict/prefer E-cores depending on policy).
+
+Differences between P and E cores on 14900:
+
+- acpi_cppc/*perf : pretty useful but not always there (e.g. aloha)
+- cache index0: 48 vs 32k (bigger CPU has smaller cache)
+- cache index1: 32 vs 64k (smaller CPU has bigger cache)
+- cache index2: 2 vs 4M, but dedicated per core vs shared per cluster (4 cores)
+
+=> probably that the presence of a larger "cluster" with less cache per
+ avg core is an indication of a smaller CPU set. Warning however, some
+ CPUs (e.g. S922X) have a large (4) cluster of big cores and a small (2)
+ cluster of little cores.
+
+
+diff -urN cpu0/acpi_cppc/lowest_nonlinear_perf cpu16/acpi_cppc/lowest_nonlinear_perf
+--- cpu0/acpi_cppc/lowest_nonlinear_perf 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/acpi_cppc/lowest_nonlinear_perf 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-20
++15
+diff -urN cpu0/acpi_cppc/nominal_perf cpu16/acpi_cppc/nominal_perf
+--- cpu0/acpi_cppc/nominal_perf 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/acpi_cppc/nominal_perf 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-40
++24
+diff -urN cpu0/acpi_cppc/reference_perf cpu16/acpi_cppc/reference_perf
+--- cpu0/acpi_cppc/reference_perf 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/acpi_cppc/reference_perf 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-40
++24
+diff -urN cpu0/cache/index0/size cpu16/cache/index0/size
+--- cpu0/cache/index0/size 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/cache/index0/size 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-48K
++32K
+diff -urN cpu0/cache/index1/shared_cpu_list cpu16/cache/index1/shared_cpu_list
+--- cpu0/cache/index1/shared_cpu_list 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/cache/index1/shared_cpu_list 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-0-1
++16
+diff -urN cpu0/cache/index1/shared_cpu_map cpu16/cache/index1/shared_cpu_map
+--- cpu0/cache/index1/shared_cpu_map 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/cache/index1/shared_cpu_map 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-00000003
++00010000
+diff -urN cpu0/cache/index1/size cpu16/cache/index1/size
+--- cpu0/cache/index1/size 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/cache/index1/size 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-32K
++64K
+diff -urN cpu0/cache/index2/shared_cpu_list cpu16/cache/index2/shared_cpu_list
+--- cpu0/cache/index2/shared_cpu_list 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/cache/index2/shared_cpu_list 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-0-1
++16-19
+--- cpu0/cache/index2/size 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/cache/index2/size 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-2048K
++4096K
+diff -urN cpu0/topology/cluster_cpus cpu16/topology/cluster_cpus
+--- cpu0/topology/cluster_cpus 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/topology/cluster_cpus 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-00000003
++000f0000
+diff -urN cpu0/topology/cluster_cpus_list cpu16/topology/cluster_cpus_list
+--- cpu0/topology/cluster_cpus_list 2024-12-26 18:39:27.563410317 +0100
++++ cpu16/topology/cluster_cpus_list 2024-12-26 18:40:39.531408186 +0100
+@@ -1 +1 @@
+-0-1
++16-19
+
+For acpi_cppc, the values differ between machines, looks like nominal_perf
+is always usable:
+
+14900k:
+$ grep '' cpu8/acpi_cppc/*
+cpu8/acpi_cppc/feedback_ctrs:ref:85172004640 del:143944480100
+cpu8/acpi_cppc/highest_perf:255
+cpu8/acpi_cppc/lowest_freq:0
+cpu8/acpi_cppc/lowest_nonlinear_perf:20
+cpu8/acpi_cppc/lowest_perf:1
+cpu8/acpi_cppc/nominal_freq:3200
+cpu8/acpi_cppc/nominal_perf:40
+cpu8/acpi_cppc/reference_perf:40
+cpu8/acpi_cppc/wraparound_time:18446744073709551615
+
+$ grep '' cpu16/acpi_cppc/*
+cpu16/acpi_cppc/feedback_ctrs:ref:84153776128 del:112977352354
+cpu16/acpi_cppc/highest_perf:255
+cpu16/acpi_cppc/lowest_freq:0
+cpu16/acpi_cppc/lowest_nonlinear_perf:15
+cpu16/acpi_cppc/lowest_perf:1
+cpu16/acpi_cppc/nominal_freq:3200
+cpu16/acpi_cppc/nominal_perf:24
+cpu16/acpi_cppc/reference_perf:24
+cpu16/acpi_cppc/wraparound_time:18446744073709551615
+
+altra:
+$ grep '' /sys/devices/system/cpu/cpu0/acpi_cppc/*
+feedback_ctrs:ref:227098452801 del:590247062111
+highest_perf:260
+lowest_freq:1000
+lowest_nonlinear_perf:200
+lowest_perf:100
+nominal_freq:2600
+nominal_perf:260
+reference_perf:100
+
+w3-2345:
+$ grep '' /sys/devices/system/cpu/cpu0/acpi_cppc/*
+feedback_ctrs:ref:4775674480779 del:5675950973600
+highest_perf:45
+lowest_freq:0
+lowest_nonlinear_perf:8
+lowest_perf:5
+nominal_freq:0
+nominal_perf:31
+reference_perf:31
+wraparound_time:18446744073709551615
+
+Other approaches may consist in checking the CPU's max frequency via
+cpufreq, e.g on the N2:
+
+ $ grep . /sys/devices/system/cpu/cpu?/cpufreq/scaling_max_freq
+ /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:2016000
+ /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:2016000
+ /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:2400000
+ /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:2400000
+ /sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq:2400000
+ /sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq:2400000
+
+However on x86, the cores no longer all have the same frequency, like below on
+the W3-2345, so it cannot always be used to split them into groups, it may at
+best be used to sort them.
+
+ $ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
+ /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq:4500000
+ /sys/devices/system/cpu/cpu1/cpufreq/scaling_max_freq:4500000
+ /sys/devices/system/cpu/cpu2/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu3/cpufreq/scaling_max_freq:4400000
+ /sys/devices/system/cpu/cpu4/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu5/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu6/cpufreq/scaling_max_freq:4400000
+ /sys/devices/system/cpu/cpu7/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu8/cpufreq/scaling_max_freq:4500000
+ /sys/devices/system/cpu/cpu9/cpufreq/scaling_max_freq:4500000
+ /sys/devices/system/cpu/cpu10/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu11/cpufreq/scaling_max_freq:4400000
+ /sys/devices/system/cpu/cpu12/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu13/cpufreq/scaling_max_freq:4300000
+ /sys/devices/system/cpu/cpu14/cpufreq/scaling_max_freq:4400000
+ /sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq:4300000
+
+On 14900, not cool either:
+
+ $ grep -h . /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq|sort|uniq -c
+ 16 4400000
+ 12 5700000
+ 4 6000000
+
+Considering that values that are within +/-10% of a cluster's min/max are still
+part of it would seem to work and would make a good rule of thumb.
+
+On x86, the model number might help, here on w3-2345:
+
+ $ grep '^model\s\s' /proc/cpuinfo |sort|uniq -c
+ 16 model : 143
+
+But not always (here: 14900K with 8xP and 16xE):
+
+ $ grep '^model\s\s' /proc/cpuinfo |sort|uniq -c
+ 32 model : 183
+
+On ARM it's rather the part number:
+
+ # a9
+ $ grep part /proc/cpuinfo
+ CPU part : 0xc09
+ CPU part : 0xc09
+
+ # a17
+ $ grep part /proc/cpuinfo
+ CPU part : 0xc0d
+ CPU part : 0xc0d
+ CPU part : 0xc0d
+ CPU part : 0xc0d
+
+ # a72
+ $ grep part /proc/cpuinfo
+ CPU part : 0xd08
+ CPU part : 0xd08
+ CPU part : 0xd08
+ CPU part : 0xd08
+
+ # a53+a72
+ $ grep part /proc/cpuinfo
+ CPU part : 0xd03
+ CPU part : 0xd03
+ CPU part : 0xd03
+ CPU part : 0xd03
+ CPU part : 0xd08
+ CPU part : 0xd08
+
+ # a53+a73
+ $ grep 'part' /proc/cpuinfo
+ CPU part : 0xd03
+ CPU part : 0xd03
+ CPU part : 0xd09
+ CPU part : 0xd09
+ CPU part : 0xd09
+ CPU part : 0xd09
+
+ # a55+a76
+ $ grep 'part' /proc/cpuinfo
+ CPU part : 0xd05
+ CPU part : 0xd05
+ CPU part : 0xd05
+ CPU part : 0xd05
+ CPU part : 0xd0b
+ CPU part : 0xd0b
+ CPU part : 0xd0b
+ CPU part : 0xd0b
+
+
+2024-12-27
+----------
+
+Such machines with P+E cores are becoming increasingly common. Some like the
+CIX-P1 can even provide 3 levels of performance: 4 big cores (A720-2.8G), 4
+medium cores (A720-2.4G), 4 little cores (A520-1.8G). Architectures like below
+will become the norm, and can be used under different policies:
+
+ +-----------------------------+
+ | L3 |
+ +---+----------+----------+---+
+ | | |
+ +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+
+ Policy: | P | P | | E | E | | E | E |
+ ------- +---+---+ +---+---+ +---+---+
+ 1 group, min: N/A 0 0
+ 1 group, max: 0 N/A N/A
+ 1 group, all: 0 0 0
+ 2 groups, min: N/A 0 1
+ 2 groups, full: 0 1 1
+ 3 groups: 0 1 2
+
+In dual-socket or multiple dies it can even become more complicated:
+
+ +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+
+ | | |
+ +---+----------+----------+---+
+ | L3.0 |
+ +-----------------------------+
+
+ +-----------------------------+
+ | L3.1 |
+ +---+----------+----------+---+
+ | | |
+ +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+
+
+Setting only a thread count would yield interesting things above:
+ 1-4T: P.0
+ 5-8T: P.0, P.1 (2 grp)
+ 9-16T: P.0, E.0, P.1, E.1 (3-4 grp)
+17-24T: PEE.0, PEE.1 (5-6 grp)
+
+With forced tgrp = 1:
+ - only fill node 0 first (P then PE, then PEE)
+
+With forced tgrp = 2:
+
+ def: P.0, P.1
+ 2-4T: P.0 only ?
+ 6-8T: P.0, P.1
+ 9-24T: PEE.0, PEE.1
+
+With dual-socket, dual-die, it becomes:
+
+ +---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
+ | | | ' | | |
+ +---+----------+----------+---+ ' +---+----------+----------+---+
+ | L3.0.0 | ' | L3.1.0 |
+ +-----------------------------+ ' +-----------------------------+
+ '
+ +-----------------------------+ ' +-----------------------------+
+ | L3.0.1 | ' | L3.1.1 |
+ +---+----------+----------+---+ ' +---+----------+----------+---+
+ | | | ' | | |
+ +---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
+ | P | P | | E | E | | E | E | ' | P | P | | E | E | | E | E |
+ +---+---+ +---+---+ +---+---+ ' +---+---+ +---+---+ +---+---+
+
+In such conditions, it could make sense to first enumerate all the available
+cores with all their characteristics, and distribute them between "buckets"
+representing the thread groups:
+
+ 1. create the min number of tgrp (tgrp.min)
+ 2. it's possible to automatically create more until tgrp.max
+ -> cores are sorted by performance then by proximity. They're
+ distributed in order into existing buckets, and if too distant,
+ then new groups are created. It could allow for example to use
+ all P-cores in the DSDD model above, split into 4 tgrp.
+ -> the total number of threads is then discovered at the end.
+
+
+It seems in the end that such binding policies (P, E, single/multi dies,
+single/multi sockets etc) should be made more accessible to the user. What
+we're missing in "cpu-map" is the ability to apply to the whole process in
+fact, so that it can supersede taskset. Indeed, right now, cpu-map requires
+too many details and that's why it often remains easier to deal with taskset,
+particularly when dealing with thread groups.
+
+We can revisit the situation differently. First, let's keep in mind that
+cpu-map is a restriction. It means "use no more than these", it does not
+mean "use all of these". So it totally makes sense to use it to replace
+taskset at the process level without interfering with groups detection.
+We could then have:
+
+ - "cpu-map all|process|global|? ..." to apply to the whole process
+ - then special keywords for the CPUs designation, among:
+ - package (socket) number
+ - die number (CCD)
+ - L3 number (CCX)
+ - cluster type (big/performant, medium, little/efficient)
+ - use of SMT or not, and which ones
+ - maybe optional numbers before these to indicate (any two of them),
+ e.g. "4P" to indicate "4 performance cores".
+
+Question: how would we designate "only P cores of socket 0" ? Or
+ "only thread 0 of all P cores" ?
+
+One benefit of such a declaration method is that it can make nbthread often
+useless and automatic while still portable across a whole fleet of servers. E.g.
+if "cpu-map all S0P*T0" would designate thread 0 of all P-cores of socket 0, it
+would mean the same on all machines.
+
+Another benefit is that we can make cpu-map and automatic detection more
+exclusive:
+ - cpu-map all => equivalent of taskset, leaves auto-detection on
+ - cpu-map thr => disables auto-detection
+
+So in the end:
+ - cpu-map all restricts the CPUs the process may use
+ -> auto-detection starts from here and sorts them
+ - thread-groups offers more "buckets" to arrange distant CPUs in the
+ same process
+ - nbthread limits the number of threads we'll use
+ -> pick the most suited ones (at least thr.min, at most thr.max)
+ and distribute them optimally among the number of thread groups.
+
+One question remains: is it always possible to automatically configure
+thread-groups ? Maybe it's possible after the detection to set an optimal
+one between grp.min and grp.max ? (e.g. socket count, core types, etc).
+
+It still seems that a policy such as "optimize-for resources|perfomance"
+would still help quite a bit.
+
+-> what defines a match between a CPU core and a group:
+ - cluster identification:
+ - either cluster_cpus if present (and sometimes), or:
+ - pkg+die+ccd number
+ - same LLC instance (L3 if present, L2 if no L3 etc)
+ - CPU core model ("model" on x86, "CPU part" on arm)
+ - number of SMT per core
+ - speed if known:
+ - /sys/devices/system/cpu/cpu0/acpi_cppc/nominal_perf if available
+ - or /sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq +/- 10%
+
+PB: on intel P+E, clusters of E cores sharing the same L2+L3, but P cores are
+alone on their L3 => poor grouping.
+
+Maybe one approach could be to characterize how L3/L2 are used. E.g. on the
+14900, we have:
+ - L3 0 => all cpus there
+ - L2 0..7 => 1C2T per L2
+ - L2 8..11 => 4C4T per L2
+ => it's obvious that CPUs connected to L2 #8..11 are not the same as those
+ on L2 #0..7. We could make something with them.
+ => it does not make sense to ditch the L2 distinction due to L3 being
+ present and the same, though it doesn't make sense to use L3 either.
+ Maybe elements with a cardinality of 1 should just be ignored. E.g.
+ cores per cache == 1 => ignore L2. Probably not true per die/pkg
+ though.
+ => replace absent or irrelevant info with "?"
+
+Note that for caches we have the list of CPUs, not the list of cores, so
+we need to remap that invidivually to cores.
+
+Warning: die_id, core_id etc are per socket, not per system. Worse, on Altra,
+core_id has gigantic values (multiples of +1 and +256). However core_cpus_list
+indicates other threads and could be a solution to create our own global core
+ID. Also, cluster_id=-1 found on all cores for A8040 on kernel 6.1.
+
+Note that LLC is always the first discriminator. But within a same LLC we can
+have the issues above (e.g. 14900).
+
+Would an intermediate approach like this work ?
+-----------------------------------------------
+ 1) first split by LLC (also test with L3-less A8040, N2800, x5-8350)
+ 2) within LLC, check of we have different cores (model, perf, freq?)
+ and resplit
+ 3) divide again so that no group has more than 64 CPUs
+
+ => it looks like from the beginning that's what we're trying to do:
+ preserve locality first, then possibly trim down the number of cores
+ if some don't bring sufficient benefit. It possibly avoids the need
+ to identify dies etc. It still doesn't completely solve the 14900
+ though.
+
+Multi-die CPUs worth checking:
+ Pentium-D (Presler, Dempsey: two 65nm dies)
+ Core2Quad Q6600/Q6700 (Kentsfield, Clowertown: two 65nm dual-core dies)
+ Core2Quad Q8xxx/Q9xxx (Yorkfield, Harpertown, Tigerton: two 45nm dual-core dies)
+ - atom 330 ("diamondville") is really a dual-die
+ - note that atom x3-z8350 ("cherry trail"), N2800 ("cedar trail") and D510
+ ("pine trail") are single-die (verified) but have two L2 caches and no L3.
+ Note that these are apparently not identified as multi-die (Q6600 has die=0).
+
+It *seems* that in order to form groups we'll first have to sort by topology,
+and only after that sort by performance so as to choose preferred CPUs.
+Otherwise we could end up trying to form inter-socket CPU groups first in
+case we're forced to mix adjacent CPUs due to too many groups.
+
+
+2025-01-07
+----------
+
+What is needed in fact is to act on two directions:
+
+ - binding restrictions: the user doesn't want the process to run on
+ second node, on efficient cores, second thread of each core, so
+ they're indicating where (not) to bind. This is a strict choice,
+ and it overrides taskset. That's the process-wide cpu-map.
+
+ - user preferences / execution profile: the user expresses their wishes
+ about how to allocate resources. This is only a binding order strategy
+ among a few existing ones that help easily decide which cores to select.
+ In this case CPUs are not enumerated. We can imagine choices as:
+
+ - full : use all permitted cores
+ - performance: use all permitted performance cores (all sockets)
+ - single-node: (like today): use all cores of a single node
+ - balanced: use a reasonable amount of perf cores (e.g. all perf
+ cores of a single socket)
+ - resources: use a single cluster of efficient cores
+ - minimal: use a single efficient core
+
+By sorting CPUs first on the performance, then applying the filtering based on
+the profile to eliminate more CPUs, then applying the limit on the desired max
+number of threads, then sorting again on the topology, it should be possible to
+draw a list of usable CPUs that can then be split in groups along the L3s.
+
+It even sounds likely that the CPU profile or allocation strategy will affect
+the first sort method. E.g:
+ - full: no sort needed though we'll use the same as perf so as to enable
+ the maximum possible high-perf threads when #threads is limited
+ - performance: probably that we should invert the topology so as to maximize
+ memory bandwidth across multiple sockets, i.e. visite node1.core0 just
+ after node0.core0 etc, and visit their threads later.
+ - bandwidth: that could be the same as "performance" one above in fact
+ - (low-)latency: better stay local first
+ - balanced: sort by perf then sockets (i.e. P0, P1, E0, E1)
+ - resources: sort on perf first.
+ - etc
+
+The strategy will also help determine the number of threads when it's not fixed
+in the configuration.
+
+Plan:
+ 1) make the profile configurable and implement the sort:
+ - option name? cpu-tuning, cpu-strategy, cpu-policy, cpu-allocation,
+ cpu-selection, cpu-priority, cpu-optimize-for, cpu-prefer, cpu-favor,
+ cpu-profile
+ => cpu-selection
+
+ 2) make the process-wide cpu-map configurable
+ 3) extend cpu-map to make it possible to designate symbolic groups
+ (e.g. "ht0/ht1, node 0, 3*CCD, etc)
+
+Also, offering an option to the user to see how haproxy sees the CPUs and the
+bindings for various profiles would be a nice improvement helping them make
+educated decisions instead of trying blindly.
+
+2025-01-11
+----------
+
+Configuration profile: there are multiple dimensions:
+ - preferences between cores types
+ - never use a given cpu type
+ - never use a given cpu location
+
+Better use something like:
+ - ignore-XXX -> never use XXX
+ - avoid-XXX -> prefer not to use XXX
+ - prefer-XXX -> prefer to use XXX
+ - restrict-XXX -> only use XXX
+
+"XXX" could be "single-threaded", "dual-threaded", "first-thread",
+"second-thread", "first-socket", "second-socket", "slowest", "fastest",
+"node-XXX" etc.
+
+We could then have:
+ - cpu-selection restrict-first-socket,ignore-slowest,...
+
+Then some of the keywords could simply be shortcuts for these.
+
+2025-01-30
+----------
+Problem: we need to set the restrictions first to eliminate undesired CPUs,
+ then sort according to the desired preferences so as to pick what
+ is considered the best CPUs. So the preference really looks like
+ a different setting.
+
+More precisely, the final stategy involves multiple criteria. For example,
+let's say that the number of threads is set to 4 and we've restricted ourselves
+to using the first thread of each CPU core. We're on an EPYC74F3, there are 3
+cores per CCX. One algorithm (resource) would create one group with 3 threads
+on the first CCX and 1 group of 1 thread on the next one, then let each of
+these threads bind to all the enabled CPU cores of their respective groups.
+Another algo (performance) would avoid sharing and would want to place one
+thread per CCX, causing the creation of 4 groups of 1 thread each. A third
+algo (balanced) would probably say that 4 threads require 2 CCX hence 2
+groups, thus there should be 2 threads per group, and it would bind 2 threads
+on all cores of the first CCX and the 2 remaining ones on the second.
+
+And if the thread count is not set, these strategies will also do their best
+to figure the optimal count. Resource would probably use 1 core max, moderate
+one CCX max, balanced one node max, performance all of them.
+
+This means that these CPU selection strategries should provide multiple
+functions:
+ - how to sort CPUs
+ - how to count how many is best within imposed rules
+
+The other actions seem to only be static. This also means that "avoid" or
+"prefer" should maybe not be used in the end, even in the sorting algo ?
+
+Or maybe these are just enums or bits in a strategy and all are considered
+at the same time everywhere. For example the thread counting could consider
+the presence of "avoid-XXX" during the operations. But how to codify XXX is
+complicated then.
+
+Maybe a scoring system could work:
+ - default: all CPUs score = 1000
+ - ignore-XXX: foreach(XXX) set score to 0
+ - restrict-XXX: foreach(YYY not XXX), set score to 0
+ - avoid-XXX: foreach(XXX) score *= 0.8
+ - prefer-XXX: foreach(XXX) score *= 1.25
+
+This supports being ignored for up to 30 different reasons before being
+permanently disabled, which is sufficient.
+
+Then sort according to score, and pick at least min_thr CPUs and continue as
+long as not max_thr or score < 1000 ("avoid"). This gives the thread count. It
+does not permit anything inter-CPU though. E.g. large vs medium vs small cores,
+or sort by locality or frequency. But maybe these ones would use a different
+strategy then and would use the score as a second sorting key (after which
+one?). Or maybe there would be 2 passes, one which avoids <1000 and another
+one which completes up to #min_thr including those <1000, in which case we
+never sort per score.
+
+We can do a bit better to respect the tgrp min/max as well: we can count what
+it implies in terms of number of tgrps (#LLC or clusters) and decide to refrain
+from adding theads which would exceed max_tgrp, but we'd possibly continue to
+add score<1000 CPUs until at least enough threads to reach min_tgrp.
+
+######## new captures ###########
+CIX-P1 / radxa Orion O6 (no topology exported):
+$ ~/haproxy/haproxy -dc -f /dev/null
+grp=[1..12] thr=[1..12]
+first node = 0
+Note: threads already set to 12
+going to start with nbthread=12 nbtgroups=1
+[keep] thr= 0 -> cpu= 0 pk=00 no=-1 di=00 cl=000 ts=000 capa=1024
+[keep] thr= 1 -> cpu= 1 pk=00 no=-1 di=00 cl=000 ts=001 capa=278
+[keep] thr= 2 -> cpu= 2 pk=00 no=-1 di=00 cl=000 ts=002 capa=278
+[keep] thr= 3 -> cpu= 3 pk=00 no=-1 di=00 cl=000 ts=003 capa=278
+[keep] thr= 4 -> cpu= 4 pk=00 no=-1 di=00 cl=000 ts=004 capa=278
+[keep] thr= 5 -> cpu= 5 pk=00 no=-1 di=00 cl=000 ts=005 capa=905
+[keep] thr= 6 -> cpu= 6 pk=00 no=-1 di=00 cl=000 ts=006 capa=905
+[keep] thr= 7 -> cpu= 7 pk=00 no=-1 di=00 cl=000 ts=007 capa=866
+[keep] thr= 8 -> cpu= 8 pk=00 no=-1 di=00 cl=000 ts=008 capa=866
+[keep] thr= 9 -> cpu= 9 pk=00 no=-1 di=00 cl=000 ts=009 capa=984
+[keep] thr= 10 -> cpu= 10 pk=00 no=-1 di=00 cl=000 ts=010 capa=984
+[keep] thr= 11 -> cpu= 11 pk=00 no=-1 di=00 cl=000 ts=011 capa=1024
+########
+
+2025-02-25 - clarification on the configuration
+-----------------------------------------------
+
+The "two dimensions" above can in fact be summarized like this:
+
+ - exposing the ability for the user to perform the same as "taskset",
+ i.e. restrict the usage to a static subset of the CPUs. We could then
+ have "cpu-set only-node0", "0-39", "ignore-smt1", "ignore-little", etc.
+ => the user defines precise sets to be kept/evicted.
+
+ - then letting the user express what they want to do with the remaining
+ cores. This is a strategy/policy that is used to:
+ - count the optimal number of threads (when not forced), also keeping
+ in mind that it cannot be more than 32/64 * maxtgroups if set.
+ - sort CPUs by order of preference (for when threads are forced or
+ a thread-hard-limit is set).
+
+ It can, partially overlap with the first one. For example, the default
+ strategy could be to focus on a single node. If the user has limited its
+ usage to cores of both nodes, the policy could still further limit this.
+ But this time it should only be a matter of sorting and preference, i.e.
+ nbthread and cpuset are respected. If a policy prefers the node with more
+ cores first, it will sort them according to this, and its algorithm for
+ counting cores will only be used if nbthread is not set, otherwise it may
+ very well end up on two nodes to respect the user's choice.
+
+And once all of this is done, thread groups should be formed based on the
+remaining topology. Similarly, if the number of tgroups is not set, the
+algorithm must try to propose one based on the topology and the maxtgroups
+setting (i.e. find a divider of the #LLC that's lower than or equal to
+maxtgroups), otherwise the configured number of tgroups is respected. Then
+the number of LLCs will be divided by this number of tgroups, and as many
+threads as enabled CPUs of each LLC will be assigned to these respective
+groups.
+
+In the end we should have groups bound to cpu sets, and threads belonging
+to groups mapped to all accessible cpus of these groups.
+
+Note: clusters may be finer than LLCs because they could report finer
+information. We could have a big and a medium cluster share the same L3
+for example. However not all boards report their cluster number (see CIX-P1
+above). However the info about the capacity still allows to figure that and
+should probably be used for that. At this point it would seem logical to say
+that the cluster number is re-adjusted based on the claimed capacity, at
+least to avoid accidentally mixing workloads on heterogenous cores. But
+sorting by cluster number might not necessarily work if allocated randomly.
+So we might need a distinct metric that doesn't require to override the
+system's numbering, like a "set", "group", "team", "bond", "bunch", "club",
+"band", ... that would be first sorted based on LLC (and no finer), and
+second based on capacity, then on L2 etc. This way we should be able to
+respect topology when forming groups.
+
+Note: We need to consider as LLC a level which has more than one core!
+ Otherwise it's supposed to exist and be unique/shared but not reported.
+ => maybe this should be done very early when counting CPUs ?
+ We need to store the LLC level somewhere in the topo.
projects / thirdparty / haproxy.git / commitdiff
