From: Uladzislau Rezki (Sony) Date: Thu, 17 Apr 2025 16:12:16 +0000 (+0200) Subject: vmalloc: align nr_vmalloc_pages and vmap_lazy_nr X-Git-Tag: v6.16-rc1~92^2~151 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=d0966120486833cd837feea9917b7ed06e74f58c;p=thirdparty%2Flinux.git vmalloc: align nr_vmalloc_pages and vmap_lazy_nr Currently both atomics share one cache-line: ... ffffffff83eab400 b vmap_lazy_nr ffffffff83eab408 b nr_vmalloc_pages ... those are global variables and they are only 8 bytes apart. Since they are modified by different threads this causes a false sharing. This can lead to a performance drop due to unnecessary cache invalidations. After this patch it is aligned to a cache line boundary: ... ffffffff8260a600 d vmap_lazy_nr ffffffff8260a640 d nr_vmalloc_pages ... Link: https://lkml.kernel.org/r/20250417161216.88318-4-urezki@gmail.com Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Baoquan He Reviewed-by: Adrian Huang Tested-by: Adrian Huang Cc: Mateusz Guzik Cc: Christop Hellwig Signed-off-by: Andrew Morton --- diff --git a/mm/vmalloc.c b/mm/vmalloc.c index dc33ebeb8b1bd..3fd802134e4ea 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1008,7 +1008,8 @@ static BLOCKING_NOTIFIER_HEAD(vmap_notify_list); static void drain_vmap_area_work(struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work); -static atomic_long_t nr_vmalloc_pages; +static __cacheline_aligned_in_smp atomic_long_t nr_vmalloc_pages; +static __cacheline_aligned_in_smp atomic_long_t vmap_lazy_nr; unsigned long vmalloc_nr_pages(void) { @@ -2117,8 +2118,6 @@ static unsigned long lazy_max_pages(void) return log * (32UL * 1024 * 1024 / PAGE_SIZE); } -static atomic_long_t vmap_lazy_nr = ATOMIC_LONG_INIT(0); - /* * Serialize vmap purging. There is no actual critical section protected * by this lock, but we want to avoid concurrent calls for performance