Hao Ge [Thu, 4 Jun 2026 06:59:38 +0000 (14:59 +0800)]
alloc_tag: fix use-after-free in /proc/allocinfo after module unload
allocinfo_start() only reinitializes the codetag iterator at position 0.
For subsequent reads (position > 0), it reuses cached iterator state from
the previous batch. allocinfo_stop() drops mod_lock between read batches,
which allows module unload to complete and free the module memory that the
cached iterator still references:
After free_mod_mem() frees the module's .rodata, allocinfo_show()
dereferences ct->filename, ct->function which point there.
Save the iterator state in allocinfo_next() and resume from it in
allocinfo_start() with codetag_next_ct(), which detects module removal via
idr_find() returning NULL and skips to the next module.
Link: https://lore.kernel.org/20260604065938.105991-1-hao.ge@linux.dev Fixes: 9f44df50fee4 ("alloc_tag: keep codetag iterator active between read()") Signed-off-by: Hao Ge <hao.ge@linux.dev> Suggested-by: Suren Baghdasaryan <surenb@google.com> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Kent Overstreet <kent.overstreet@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Hao Ge [Thu, 4 Jun 2026 02:40:08 +0000 (10:40 +0800)]
mm/alloc_tag: replace fixed-size early PFN array with dynamic linked list
Pages allocated before page_ext is available have their codetag left
uninitialized. Track these early PFNs and clear their codetag in
clear_early_alloc_pfn_tag_refs() to avoid "alloc_tag was not set" warnings
when they are freed later.
Currently a fixed-size array of 8192 entries is used, with a warning if
the limit is exceeded. However, the number of early allocations depends
on the number of CPUs and can be larger than 8192.
Replace the fixed-size array with a dynamically allocated linked list of
pfn_pool structs. Each node is allocated via alloc_page() and mapped to a
pfn_pool containing a next pointer, an atomic slot counter, and a PFN
array that fills the remainder of the page.
The tracking pages themselves are allocated via alloc_page(), which would
trigger __pgalloc_tag_add() -> alloc_tag_add_early_pfn() and recurse
indefinitely. Introduce __GFP_NO_CODETAG (reuses the %__GFP_NO_OBJ_EXT
bit) and pass gfp_flags through pgalloc_tag_add() so that the early path
can skip recording allocations that carry this flag.
Link: https://lore.kernel.org/20260604024008.46592-1-hao.ge@linux.dev Signed-off-by: Hao Ge <hao.ge@linux.dev> Suggested-by: Suren Baghdasaryan <surenb@google.com> Acked-by: Suren Baghdasaryan <surenb@google.com> Cc: Kent Overstreet <kent.overstreet@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Dev Jain [Thu, 4 Jun 2026 05:53:06 +0000 (05:53 +0000)]
selftests/mm/hmm-tests: test pagemap reads of PMD device-private entries
To cover pagemap paths scanning PMD entries, add assertions to check
whether a device-private PMD entry has the correct pagemap information -
the PM_SWAP bit must be on in the pagemap entry. Before that, we must
assert through HMM_DMIRROR_SNAPSHOT snapshot that the leaf entry is at PMD
level and not PTE level.
Link: https://lore.kernel.org/20260604055308.1947679-3-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Balbir Singh <balbirs@nvidia.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Oscar Salvador (SUSE) <osalvador@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Dev Jain [Thu, 4 Jun 2026 05:53:05 +0000 (05:53 +0000)]
fs/proc/task_mmu: do not warn on seeing non-migration pmd entry
Patch series "mm/hmm: A fix and a selftest", v3.
Patch 1 fixes a stale warning present from the time when only migration
softleaf entries were supported at the PMD level.
Patch 2 adds some code into hmm-tests.c which exercises the pagemap path
for PMD device-private entries.
This patch (of 2):
pagemap_pmd_range_thp() warns if a non-present PMD is not a migration
entry. This became false once device-private entries at the PMD level
were added.
Therefore, remove the stale migration-only assertion.
Link: https://lore.kernel.org/20260604055308.1947679-1-dev.jain@arm.com Link: https://lore.kernel.org/20260604055308.1947679-2-dev.jain@arm.com Fixes: a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages") Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Balbir Singh <balbirs@nvidia.com> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Tested-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
JP Kobryn [Thu, 4 Jun 2026 06:17:25 +0000 (23:17 -0700)]
mm/compaction: cap compact_gap() at COMPACT_CLUSTER_MAX
compact_gap() returns 2 << order, which is used as watermark headroom in
__compaction_suitable() and as a threshold in kswapd reclaim decisions.
The computed value scales exponentially by order. For order-9 THP
allocations this evaluates to 1024 pages, but the compaction free
scanner's working set is bounded by COMPACT_CLUSTER_MAX (32 pages). The
scanner stops isolating free pages once it matches the migration batch.
The current gap over-reserves by 32x.
On fragmented production hosts, kswapd will try to reclaim up to the gap,
but it only reaches that threshold in 18% of attempts. As a result,
reclaim continues in the majority of cases despite many lower-order free
pages being available. The over-sized gap also causes 46% of order-9
compaction suitability checks to fail unnecessarily: the zone has
sufficient free pages for the scanner to operate, but not enough to clear
the inflated threshold.
Cap compact_gap() at COMPACT_CLUSTER_MAX so the watermark headroom
reflects the scanner's actual capacity. This function is used by two key
heuristics. The first is when kswapd can stop high-order reclaim and
downgrade to order-0 balancing, allowing kcompactd to be woken for the
original higher allocation order. The second is zone suitability
checking, where the smaller gap allows compaction to start sooner.
Note that orders 0-4 are unaffected since their gap is already less than
or equal to COMPACT_CLUSTER_MAX.
A/B test on v6.13-based instagram production hosts (64GB, 60s
measurement):
Additional tests were also performed using a workload of similar shape and
based on mm-new at the time of testing. Across three 60s runs, the patch
showed improvements consistent with the previous test: reduced kswapd
reclaim and fewer THP fault fallbacks.
Youngjun Park [Mon, 23 Mar 2026 16:08:22 +0000 (01:08 +0900)]
mm/swap: remove redundant swap device reference in alloc/free
In the previous commit, uswsusp was modified to pin the swap device when
the swap type is determined, ensuring the device remains valid throughout
the hibernation I/O path.
Therefore, it is no longer necessary to repeatedly get and put the swap
device reference for each swap slot allocation and free operation.
For hibernation via the sysfs interface, user-space tasks are frozen
before swap allocation begins, so swapoff cannot race with allocation.
After resume, tasks remain frozen while swap slots are freed, so
additional reference management is not required there either.
Remove the redundant swap device get/put operations from the hibernation
swap allocation and free paths.
Also remove the SWP_WRITEOK check before allocation, as the cluster
allocation logic already validates the swap device state.
Update function comments to document the caller's responsibility for
ensuring swap device stability.
Link: https://lore.kernel.org/20260323160822.1409904-3-youngjun.park@lge.com Signed-off-by: Youngjun Park <youngjun.park@lge.com> Reviewed-by: Kairui Song <kasong@tencent.com> Cc: Baoquan He <bhe@redhat.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: "Rafael J . Wysocki" <rafael@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Youngjun Park [Mon, 23 Mar 2026 16:08:21 +0000 (01:08 +0900)]
mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device
Patch series "mm/swap, PM: hibernate: fix swapoff race in uswsusp by
pinning swap device", v8.
Currently, in the uswsusp path, only the swap type value is retrieved at
lookup time without holding a reference. If swapoff races after the type
is acquired, subsequent slot allocations operate on a stale swap device.
Additionally, grabbing and releasing the swap device reference on every
slot allocation is inefficient across the entire hibernation swap path.
This patch series addresses these issues:
- Patch 1: Fixes the swapoff race in uswsusp by pinning the swap device
from the point it is looked up until the session completes.
- Patch 2: Removes the overhead of per-slot reference counting in alloc/free
paths and cleans up the redundant SWP_WRITEOK check.
This patch (of 2):
Hibernation via uswsusp (/dev/snapshot ioctls) has a race window: after
selecting the resume swap area but before user space is frozen, swapoff
may run and invalidate the selected swap device.
Fix this by pinning the swap device with SWP_HIBERNATION while it is in
use. The pin is exclusive, which is sufficient since hibernate_acquire()
already prevents concurrent hibernation sessions.
The kernel swsusp path (sysfs-based hibernate/resume) uses
find_hibernation_swap_type() which is not affected by the pin. It freezes
user space before touching swap, so swapoff cannot race.
Introduce dedicated helpers:
- pin_hibernation_swap_type(): Look up and pin the swap device.
Used by the uswsusp path.
- find_hibernation_swap_type(): Lookup without pinning.
Used by the kernel swsusp path.
- unpin_hibernation_swap_type(): Clear the hibernation pin.
While a swap device is pinned, swapoff is prevented from proceeding.
tanze [Mon, 1 Jun 2026 11:04:23 +0000 (19:04 +0800)]
mm/filemap: use folio_next_index() for start
Use folio_next_index() instead of open-coding folio->index +
folio_nr_pages(folio) when updating @start in filemap_get_folios_contig(),
filemap_get_folios_tag(), and filemap_get_folios_dirty().
Link: https://lore.kernel.org/20260601110425.44784-1-tanze@kylinos.cn Signed-off-by: tanze <tanze@kylinos.cn> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Hui Zhu [Fri, 29 May 2026 01:41:30 +0000 (09:41 +0800)]
vmalloc: fix NULL pointer dereference in is_vm_area_hugepages()
find_vm_area() can return NULL if the given address is not a valid vmalloc
area. Check the return value before dereferencing it to avoid a kernel
crash.
Link: https://lore.kernel.org/20260529014130.671291-1-hui.zhu@linux.dev Fixes: 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings") Signed-off-by: Hui Zhu <zhuhui@kylinos.cn> Reviewed-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com> Cc: Nicholas Piggin <npiggin@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Mon, 1 Jun 2026 08:48:44 +0000 (16:48 +0800)]
sparc/mm: drop vmemmap_check_pmd helper and use generic code
The generic implementations now suffice; remove the sparc copy.
Link: https://lore.kernel.org/20260601084845.3792171-6-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Mon, 1 Jun 2026 08:48:43 +0000 (16:48 +0800)]
loongarch/mm: drop vmemmap_check_pmd helper and use generic code
The generic implementations now suffice; remove the loongarch copy.
Link: https://lore.kernel.org/20260601084845.3792171-5-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Mon, 1 Jun 2026 08:48:42 +0000 (16:48 +0800)]
riscv/mm: drop vmemmap_pmd helpers and use generic code
The generic implementations now suffice; remove the riscv copies.
Link: https://lore.kernel.org/20260601084845.3792171-4-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Mon, 1 Jun 2026 08:48:41 +0000 (16:48 +0800)]
arm64/mm: drop vmemmap_pmd helpers and use generic code
The generic implementations now suffice; remove the arm64 copies.
Link: https://lore.kernel.org/20260601084845.3792171-3-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: Will Deacon <will@kernel.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: WANG Xuerui <kernel@xen0n.name> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Mon, 1 Jun 2026 08:48:40 +0000 (16:48 +0800)]
mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd()
Patch series "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and
vmemmap_check_pmd()", v3.
The weak vmemmap_set_pmd() and vmemmap_check_pmd() hooks are currently
no-ops in the generic code, which leaves architectures that need PMD-level
handling to open-code the same logic locally.
This series provides generic implementations for both helpers in
mm/sparse-vmemmap.c. vmemmap_set_pmd() installs a huge PMD with
PAGE_KERNEL protection, and vmemmap_check_pmd() verifies a present leaf
PMD before reusing the existing vmemmap_verify() helper.
With those generic helpers in place, patches 2-5 remove the now redundant
arch-specific implementations from arm64, riscv, loongarch, and sparc.
This patch (of 5):
The two weak functions are currently no-ops on every architecture, forcing
each platform that needs them to duplicate the same handful of lines.
Provide a generic implementation:
- vmemmap_set_pmd() simply sets a huge PMD with PAGE_KERNEL protection.
- vmemmap_check_pmd() verifies that the PMD is present and leaf,
then calls the existing vmemmap_verify() helper.
Architectures that need special handling can continue to override the weak
symbols; everyone else gets the standard version for free.
Link: https://lore.kernel.org/20260601084845.3792171-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20260601084845.3792171-2-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Cc: Albert Ou <aou@eecs.berkeley.edu> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Andreas Larsson <andreas@gaisler.com> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: David S. Miller <davem@davemloft.net> Cc: Huacai Chen <chenhuacai@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Palmer Dabbelt <palmer@dabbelt.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: WANG Xuerui <kernel@xen0n.name> Cc: Will Deacon <will@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
`Page::nid` is a trivial wrapper around the C function `page_to_nid`. It
does not make sense to go through a trivial wrapper for this function, so
mark it inline.
This follows commit 878620c5a93a ("rust: page: optimize rust symbol
generation for Page"), which did the same for `alloc_page` and `drop`.
Link: https://github.com/Rust-for-Linux/linux/issues/1145 Link: https://lore.kernel.org/20260529085316.27432-1-nakamura.shuta@gmail.com Signed-off-by: Nakamura Shuta <nakamura.shuta@gmail.com> Reviewed-by: Alice Ryhl <aliceryhl@google.com> Reviewed-by: Gary Guo <gary@garyguo.net> Cc: Andreas Hindborg <a.hindborg@kernel.org> Cc: Björn Roy Baron <bjorn3_gh@protonmail.com> Cc: Danilo Krummrich <dakr@kernel.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Miguel Ojeda <ojeda@kernel.org> Cc: Trevor Gross <tmgross@umich.edu> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks
The VMA flags bitmap is a single word today: NUM_VMA_FLAG_BITS is
BITS_PER_LONG, so on 32-bit vma_flags_t holds only 32 bits. (The bitmap
type exists so this can grow past BITS_PER_LONG later; until it does,
anything declared above the first word is out of range on 32-bit.) The bit
enum nevertheless declares some bits unconditionally above BITS_PER_LONG
-- VMA_UFFD_MINOR_BIT is 41, with VM_UFFD_MINOR == VM_NONE on 32-bit so no
VMA actually carries the bit.
__VMA_UFFD_FLAGS feeds VMA_UFFD_MINOR_BIT to mk_vma_flags()
unconditionally. On 32-bit that becomes __set_bit(41, &one_long), a write
one word past the end of the single-word bitmap. The compiler folds the
out-of-bounds store with wraparound (1UL << (41 % 32) == bit 9) into the
first word; bit 9 is already in __VMA_UFFD_FLAGS so the mask happens to
come out right today, but it is an out-of-bounds write all the same, and
any high-numbered bit whose mod-BITS_PER_LONG position is otherwise unused
would silently OR an extra bit into the mask.
Rather than feed bit numbers that may not exist on the current build to
mk_vma_flags(), build the mask from whole per-mode masks that collapse to
EMPTY_VMA_FLAGS when their feature is unavailable. Add
mk_vma_flags_from_masks() for that, and define VMA_UFFD_MISSING / _WP /
_MINOR alongside the VM_UFFD_* flags, gating VMA_UFFD_MINOR on the same
config as VM_UFFD_MINOR (which implies 64BIT, where bit 41 fits). An
out-of-range bit is then never materialised, on any arch, and the in-range
fast path stays a compile-time constant.
Link: https://lore.kernel.org/20260529172331.356655-7-kas@kernel.org Fixes: 9ea35a25d51b ("mm: introduce VMA flags bitmap type") Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Reported-by: Sashiko AI review <sashiko-bot@kernel.org> Suggested-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Assisted-by: Claude:claude-opus-4-8 Cc: David Hildenbrand <david@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Xu <peterx@redhat.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Balbir Singh <balbirs@nvidia.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
userfaultfd: gate must_wait writability check on pte_present()
userfaultfd_must_wait() and userfaultfd_huge_must_wait() read the PTE
without taking the page table lock and then apply pte_write() /
huge_pte_write() to it. Those accessors decode bits from the present
encoding only; on a swap or migration entry they read the offset bits that
happen to share the same position and return an undefined result.
The intent of the check is "is this fault still WP-blocked?". A
non-marker swap entry means the page is in transit -- the userfault
context the original fault delivered against is no longer the same, and
the swap-in or migration completion path will re-deliver a fresh fault if
userspace still needs to handle it. Worst case under the current code the
garbage write bit says "wait", and the thread stays asleep until a
UFFDIO_WAKE that may never arrive.
Gate the writability check on pte_present() so the lockless re-check only
inspects present-PTE bits when the entry is actually present. The
non-present, non-marker case returns "don't wait" and lets the fault path
retry.
Link: https://lore.kernel.org/20260529172331.356655-6-kas@kernel.org Fixes: 369cd2121be4 ("userfaultfd: hugetlbfs: userfaultfd_huge_must_wait for hugepmd ranges") Fixes: 63b2d4174c4a ("userfaultfd: wp: add the writeprotect API to userfaultfd ioctl") Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Reported-by: Sashiko AI review <sashiko-bot@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Xu <peterx@redhat.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Balbir Singh <balbirs@nvidia.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/huge_memory: preserve pmd_swp_uffd_wp on device-private PMD downgrade
change_non_present_huge_pmd() rewrites a writable device-private PMD swap
entry into a readable one without carrying pmd_swp_uffd_wp() across. The
PTE-level change_softleaf_pte() does this correctly; mirror that here,
matching what copy_huge_pmd() does for the fork path. Without the carry,
a plain mprotect() over a UFFD_WP-marked device-private THP strips the bit
and the trap is bypassed on swap-in.
Link: https://lore.kernel.org/20260529172331.356655-5-kas@kernel.org Fixes: 368076f52ebe ("mm/huge_memory: add device-private THP support to PMD operations") Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Reported-by: Sashiko AI review <sashiko-bot@kernel.org> Reviewed-by: Balbir Singh <balbirs@nvidia.com> Cc: David Hildenbrand <david@kernel.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Xu <peterx@redhat.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
fs/proc/task_mmu: fix hugetlb self-deadlock in pagemap_scan_pte_hole()
A PAGEMAP_SCAN ioctl requesting PM_SCAN_WP_MATCHING on a hugetlb VMA hangs
the calling thread, unkillably, as soon as the scan reaches an unpopulated
part of the range:
do_pagemap_scan()
walk_page_range()
walk_hugetlb_range()
hugetlb_vma_lock_read() # take the vma lock for read ...
pagemap_scan_pte_hole() # ... ->pte_hole() for a hole
uffd_wp_range()
change_protection()
hugetlb_change_protection()
hugetlb_vma_lock_write() # ... and block taking it for write
walk_hugetlb_range() holds the hugetlb vma lock for read across the whole
walk. A present entry goes to ->hugetlb_entry(); an unpopulated one goes
to ->pte_hole(), i.e. pagemap_scan_pte_hole(). To write-protect the hole
that handler calls uffd_wp_range(), which on a hugetlb VMA reaches
hugetlb_change_protection() and takes the same vma lock for write. The
thread then blocks in down_write() waiting for the read lock it is itself
holding.
The populated path avoids this: pagemap_scan_hugetlb_entry()
write-protects the entry inline under the page-table lock and never enters
hugetlb_change_protection().
Do the same for holes. Fault in the page table and install the uffd-wp
marker directly with make_uffd_wp_huge_pte() under the page-table lock,
rather than routing through uffd_wp_range(). That is the same sequence
hugetlb_change_protection() runs for an unpopulated entry, minus the vma
write lock -- which is safe to skip because PMD sharing is disabled on
uffd-wp VMAs (hugetlb_unshare_all_pmds() runs at registration), leaving
nothing for that lock to serialise against.
Link: https://lore.kernel.org/20260529172331.356655-4-kas@kernel.org Fixes: 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs") Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Reported-by: Sashiko AI review <sashiko-bot@kernel.org> Assisted-by: Claude:claude-opus-4-8 Cc: David Hildenbrand <david@kernel.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Xu <peterx@redhat.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Balbir Singh <balbirs@nvidia.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
fs/proc/task_mmu: use huge_page_size() in pagemap_scan_hugetlb_entry()
The partial-page check compares against HPAGE_SIZE (PMD_SIZE), which is
wrong for gigantic hugetlb hstates (e.g. 1G). The walker hands the
callback a huge_page_size()-sized range, never start + HPAGE_SIZE, so the
comparison always declares it partial and aborts the WP. Compare against
the actual hstate's page size.
Link: https://lore.kernel.org/20260529172331.356655-3-kas@kernel.org Fixes: 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs") Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Reported-by: Sashiko AI review <sashiko-bot@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Dev Jain <dev.jain@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Xu <peterx@redhat.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Balbir Singh <balbirs@nvidia.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "userfaultfd/pagemap: pre-existing fixes".
These are pre-existing bug fixes that were carried at the front of the
userfaultfd RWP working-set-tracking series up to v5 [1]. Per review
feedback that fixes should not sit in the middle of a feature series, they
are split out and sent on their own; the RWP series is reposted rebased on
top of this.
All six were flagged by the Sashiko AI review of the RWP series and carry Reported-by: Sashiko AI review <sashiko-bot@kernel.org>. They are
independent of RWP, apply to mm-new directly, and carry Cc: stable@.
1: fs/proc/task_mmu: a missing huge_ptep_modify_prot_start() in
make_uffd_wp_huge_pte() can lose hardware Dirty/Accessed updates
when PAGEMAP_SCAN write-protects a hugetlb PTE.
2: fs/proc/task_mmu: pagemap_scan_hugetlb_entry() compares the range
against HPAGE_SIZE rather than the hstate page size, so it never
write-protects gigantic hugetlb pages.
3: fs/proc/task_mmu: PAGEMAP_SCAN with PM_SCAN_WP_MATCHING over an
unpopulated hugetlb range self-deadlocks -- pagemap_scan_pte_hole()
calls uffd_wp_range() while walk_hugetlb_range() holds the hugetlb
vma lock for read, and hugetlb_change_protection() then takes it
for write. Install the marker inline instead.
4: mm/huge_memory: change_non_present_huge_pmd() drops pmd_swp_uffd_wp
on a device-private PMD permission downgrade, silently losing the
uffd-wp marker.
5: userfaultfd: must_wait() applies pte_write() to a locklessly read
PTE without checking pte_present(), so swap/migration entries
decode random offset bits and a thread can stay parked on a stale
fault.
6: userfaultfd: __VMA_UFFD_FLAGS feeds VMA_UFFD_MINOR_BIT (41) to
mk_vma_flags() unconditionally, an out-of-bounds write into the
single-word vma_flags_t on 32-bit. Build the mask from config-gated
per-mode masks so an unavailable bit is never materialised.
This patch (of 6):
make_uffd_wp_huge_pte() arms the UFFD_WP bit on a present HugeTLB PTE by
calling huge_ptep_modify_prot_commit() with a ptent snapshot that was
fetched without the corresponding huge_ptep_modify_prot_start(). The
start helper is what atomically clears the entry so the kernel-owned
snapshot stays consistent until the commit; without it, the hardware may
set Dirty or Accessed in the live PTE between the original read and the
commit, and huge_ptep_modify_prot_commit() (whose generic implementation
just calls set_huge_pte_at()) then writes the stale snapshot back over the
live hardware bits, losing the update.
The non-hugetlb sibling make_uffd_wp_pte() does this correctly via
ptep_modify_prot_start() / ptep_modify_prot_commit(). Mirror that pattern
for the present-PTE branch. The migration case stays as-is -- migration
entries are non-present, so there's no hardware update to race against.
Link: https://lore.kernel.org/20260529172331.356655-1-kas@kernel.org Link: https://lore.kernel.org/20260529172331.356655-2-kas@kernel.org Link: https://lore.kernel.org/all/20260526130509.2748441-1-kirill@shutemov.name/ Fixes: 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs") Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Reported-by: Sashiko AI review <sashiko-bot@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Dev Jain <dev.jain@arm.com> Cc: David Hildenbrand <david@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Xu <peterx@redhat.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Balbir Singh <balbirs@nvidia.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Brendan Jackman [Mon, 1 Jun 2026 11:40:09 +0000 (11:40 +0000)]
mm: delete stale comment about cachelines
These comments have been wrong since commit a211c6550efc ("mm: page_alloc:
defrag_mode kswapd/kcompactd watermarks") added NR_FREE_PAGES_BLOCKS.
Since nobody has complained about it in the last year, it seems unlikely
these comments were particularly useful anyway, so delete them.
Link: https://lore.kernel.org/20260601-zone_stat_item-comment-v1-1-f452dd91d5eb@google.com Signed-off-by: Brendan Jackman <jackmanb@google.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Cunlong Li [Thu, 28 May 2026 02:48:45 +0000 (10:48 +0800)]
zram: drop unused bio parameter from write helpers
After "zram: fix use-after-free in zram_bvec_write_partial()",
zram_bvec_write_partial() always passes NULL to zram_read_page() and no
longer needs the parent bio. Mirror the read side
(zram_bvec_read_partial() has not taken a bio since commit 4e3c87b9421d
("zram: fix synchronous reads")) and drop the parameter from
zram_bvec_write_partial() and zram_bvec_write().
No functional change.
Link: https://lore.kernel.org/20260528-zram-v3-2-cab86eef8764@gmail.com Signed-off-by: Cunlong Li <shenxiaogll@gmail.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org> Cc: Jens Axboe <axboe@kernel.dk> Cc: Minchan Kim <minchan@kernel.org> Cc: Yisheng Xie <xieyisheng1@huawei.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/page_vma_mapped_walk: use ptep_get_lockless() for lockless access
When not holding the lock, there is a chance that the pte gets modified
under our feet, so we need to use the lockless API to make sure that the
entries remain consistent during the read."
Switch from ptep_get() to ptep_get_lockless() accessor for PTE reads when
no lock is taken.
[osalvador@suse.de: changelog addition] Link: https://lore.kernel.org/ahhNq0pFKvSKZQbR@localhost.localdomain Link: https://lore.kernel.org/20260528075507.1821939-1-agordeev@linux.ibm.com Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Liam Howlett <liam@infradead.org> Cc: Rik van Riel <riel@surriel.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
fujunjie [Tue, 26 May 2026 09:12:48 +0000 (09:12 +0000)]
mm/page_alloc: fix deferred compaction accounting
COMPACT_DEFERRED means compaction did not start because past failures
caused the zone to be deferred. try_to_compact_pages() returns the
maximum result seen while walking the zonelist, so a final
COMPACT_DEFERRED result means no later zone reported that compaction
actually ran.
__alloc_pages_direct_compact() skips COMPACTSTALL and COMPACTFAIL
accounting when try_to_compact_pages() returns COMPACT_SKIPPED, but not
when it returns COMPACT_DEFERRED. A deferred-only direct compaction
attempt can therefore look like a stall, and then a failure if the
allocation still cannot be satisfied.
Treat COMPACT_DEFERRED like COMPACT_SKIPPED in this accounting path. If a
later zone runs compaction and returns a result above COMPACT_DEFERRED, or
compact_zone_order() reports COMPACT_SUCCESS for a captured page, the
final result is not COMPACT_DEFERRED and the existing accounting still
runs.
Link: https://lore.kernel.org/tencent_368AF1F3821E46232637BE16D65C45CF3308@qq.com Fixes: 06dac2f467fe ("mm: compaction: update the COMPACT[STALL|FAIL] events properly") Signed-off-by: fujunjie <fujunjie1@qq.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Usama Arif [Mon, 1 Jun 2026 10:21:18 +0000 (03:21 -0700)]
mm: use mapping_max_folio_order() for force_thp_readahead order
The force_thp_readahead path in do_sync_mmap_readahead() is gated on
HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always requests HPAGE_PMD_ORDER
/ HPAGE_PMD_NR. On configurations where HPAGE_PMD_ORDER exceeds
MAX_PAGECACHE_ORDER, notably arm64 with a 64K base page size, VM_HUGEPAGE
mappings cannot use this path and fall back to the non-forced mmap
readahead path even when the mapping supports useful large folios.
Enable forced readahead for mappings that support large folios and request
the max folio order supported by the mapping, capped at 2M. 2MB is chosen
as the cap because it matches the PMD size on x86_64 and on arm64 with 4K
base pages, so the size/memory-pressure tradeoff for folios of that size
is already well understood. On arm64 with 16K and 64K base page sizes,
2MB is also the contiguous-PTE (contpte) block size, so the resulting
folios coalesce into a single TLB entry and reduce TLB pressure on the
readahead path. This will result in 32M folios not being faulted in with
16K base page size for arm64, but with contpte, the performance difference
should be negligible.
The final allocation order may still be clamped by page_cache_ra_order()
to the mapping and request geometry, but this gives VM_HUGEPAGE mappings
on such configurations a large-folio readahead request instead of dropping
back to base-page readahead.
Link: https://lore.kernel.org/20260601102205.3985788-3-usama.arif@linux.dev Signed-off-by: Usama Arif <usama.arif@linux.dev> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Cc: Alistair Popple <apopple@nvidia.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christian Brauner <brauner@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Heiher <r@hev.cc> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kees Cook <kees@kernel.org> Cc: Kevin Brodsky <kevin.brodsky@arm.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nico Pache <npache@redhat.com> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Rohan McLure <rmclure@linux.ibm.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Cc: Kiryl Shutsemau (Meta) <kas@kernel.org> Cc: Oscar Salvador (SUSE) <osalvador@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Usama Arif [Mon, 1 Jun 2026 10:21:17 +0000 (03:21 -0700)]
mm: bypass mmap_miss heuristic for VM_EXEC readahead
Patch series "mm: improve large folio readahead for exec memory", v7.
Two checks in do_sync_mmap_readahead() limit large-folio readahead:
1. The mmap_miss heuristic is meant to throttle wasteful speculative
readahead. It is currently also applied to the VM_EXEC readahead
path, which is targeted rather than speculative. Once mmap_miss exceeds
MMAP_LOTSAMISS, exec readahead - including the large-folio
order requested by exec_folio_order() - is disabled. On
configurations where the mmap_miss decrement paths are not
active (see patch 1) the counter only grows, so exec readahead
is permanently disabled after the first 100 faults.
2. The force_thp_readahead path is gated only on
HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER and always drives the
readahead at HPAGE_PMD_ORDER. Configurations where
HPAGE_PMD_ORDER exceeds MAX_PAGECACHE_ORDER never reach this
path, even when the mapping itself supports usefully large
folios well below the cap.
Both issues are most visible on arm64 with a 64K base page size, where
HPAGE_PMD_ORDER is 13 (512MB) -- above MAX_PAGECACHE_ORDER (11) -- and
where fault_around_pages collapses to 1 disabling should_fault_around()
(one of the two mmap_miss decrement sites). However the fixes are
architecture-agnostic: patch 1 reflects the nature of VM_EXEC readahead
regardless of base page size, and patch 2 generalises the gate so any
mapping advertising a usefully large maximum folio order can benefit.
I created a benchmark that mmaps a large executable file madvises it as
huge and calls RET-stub functions at PAGE_SIZE offsets across it. "Cold"
measures fault + readahead cost. "Random" first faults in all pages with
a sequential sweep (not measured), then measures time for calling random
offsets, isolating iTLB miss cost for scattered execution.
The benchmark results on Neoverse V2 (Grace), arm64 with 64K base pages,
512MB executable file on ext4, averaged over 3 runs:
Phase | Baseline | Patched | Improvement
-----------|--------------|--------------|------------------
Cold fault | 83.4 ms | 41.3 ms | 50% faster
Random | 76.0 ms | 58.3 ms | 23% faster
This patch (of 2):
The mmap_miss heuristic is intended to stop speculative mmap readahead
when a file looks like a random-access workload. That does not fit the
VM_EXEC path very well.
VM_EXEC readahead is already constrained differently from ordinary mmap
read-around: it is bounded by the VMA, uses exec_folio_order() to choose
an order useful for executable mappings, and sets async_size to 0 so it
does not create follow-on readahead. When VM_HUGEPAGE is also present,
the larger readahead is an explicit userspace opt-in.
The mmap_miss counter is decremented from cache-hit paths in
do_async_mmap_readahead() and filemap_map_pages(). Those paths are not
always enough to balance the synchronous miss increments for executable
mappings. In particular, when fault-around is effectively disabled, such
as configurations where fault_around_pages is 1, filemap_map_pages() is
not reached from the fault path. The counter can then become a stale
throttle for VM_EXEC mappings and suppress the readahead behavior that the
executable-specific path is trying to provide.
Skip both mmap_miss increments and decrements for VM_EXEC mappings,
matching the existing VM_SEQ_READ treatment and keeping the counter
accounting symmetric.
Link: https://lore.kernel.org/20260601102205.3985788-1-usama.arif@linux.dev Link: https://lore.kernel.org/20260601102205.3985788-2-usama.arif@linux.dev Signed-off-by: Usama Arif <usama.arif@linux.dev> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Kiryl Shutsemau (Meta) <kas@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Cc: Alistair Popple <apopple@nvidia.com> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Christian Brauner <brauner@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Heiher <r@hev.cc> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kees Cook <kees@kernel.org> Cc: Kevin Brodsky <kevin.brodsky@arm.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nico Pache <npache@redhat.com> Cc: Pasha Tatashin <pasha.tatashin@soleen.com> Cc: Rohan McLure <rmclure@linux.ibm.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
fujunjie [Tue, 26 May 2026 12:22:41 +0000 (12:22 +0000)]
mm/compaction: respect cpusets when checking retry suitability
should_compact_retry() handles COMPACT_SKIPPED by asking
compaction_zonelist_suitable() whether reclaim can make a later compaction
attempt worthwhile. That answer is used for the current allocation, so it
should follow the same zone eligibility rules as the allocation itself.
When cpusets are enabled, allocator slowpath decisions are marked with
ALLOC_CPUSET. The allocation path, direct compaction and reclaim retry
all skip zones rejected by __cpuset_zone_allowed().
compaction_zonelist_suitable() does not apply that filter. It only walks
ac->zonelist/ac->nodemask, so it can return true because a zone that is
not usable for the current allocation would pass __compaction_suitable().
That does not let the allocation use the disallowed zone. Later
allocation and direct compaction paths still apply cpuset filtering.
However, it can make should_compact_retry() retry based on memory that
this allocation cannot use.
Pass gfp_mask down and apply the same ALLOC_CPUSET check in
compaction_zonelist_suitable(). This keeps the retry decision aligned
with the zones that the allocation is allowed to use.
A temporary debugfs probe was also used to call the old and new
compaction_zonelist_suitable() predicates in the same two-node NUMA guest.
The task was restricted to mems=0 while ac->nodemask covered nodes 0-1.
After putting pressure on node0, node0 failed __compaction_suitable() for
order-10 and node1 passed it, but node1 was rejected by
__cpuset_zone_allowed(). In that state the old predicate returned true
and the patched predicate returned false.
Link: https://lore.kernel.org/tencent_F59F2BA2CC5779308E10DF54593C736D3E0A@qq.com Fixes: 435b3894e742 ("mm:page_alloc: fix the NULL ac->nodemask in __alloc_pages_slowpath()") Signed-off-by: fujunjie <fujunjie1@qq.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Lance Yang [Tue, 2 Jun 2026 04:34:53 +0000 (12:34 +0800)]
mm/thp: clear deferred split shrinker bits when queues drain
deferred_split_count() returns the raw list_lru count. When the
per-memcg, per-node list is empty, that count is 0.
That skips scanning, but it does not tell memcg reclaim that the shrinker
is empty. shrink_slab_memcg() only clears the memcg shrinker bit when the
count callback reports SHRINK_EMPTY.
Return SHRINK_EMPTY for an empty deferred split list, so the bit can be
cleared once the queue has drained.
Link: https://lore.kernel.org/20260602043453.67597-1-lance.yang@linux.dev Signed-off-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Usama Arif <usama.arif@linux.dev> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kairui Song <ryncsn@gmail.com> Cc: Kefeng Wang <wangkefeng.wang@huawei.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:16 +0000 (16:45 -0400)]
mm: switch deferred split shrinker to list_lru
The deferred split queue handles cgroups in a suboptimal fashion. The
queue is per-NUMA node or per-cgroup, not the intersection. That means on
a cgrouped system, a node-restricted allocation entering reclaim can end
up splitting large pages on other nodes:
The shrinker bit adds an imperfect guard rail. As soon as the cgroup has
a single large page on the node of interest, all large pages owned by that
memcg, including those on other nodes, will be split.
list_lru properly sets up per-node, per-cgroup lists. As a bonus, it
streamlines a lot of the list operations and reclaim walks. It's used
widely by other major shrinkers already. Convert the deferred split queue
as well.
The list_lru per-memcg heads are instantiated on demand when the first
object of interest is allocated for a cgroup, by calling
folio_memcg_alloc_deferred(). Add calls to where splittable pages are
created: anon faults, swapin faults, khugepaged collapse.
These calls create all possible node heads for the cgroup at once, so the
migration code (between nodes) doesn't need any special care.
[akpm@linux-foundation.org: fix build with CONFIG_TRANSPARENT_HUGEPAGE=n] Link: https://lore.kernel.org/202605281620.lc3rtkBm-lkp@intel.com
[hannes@cmpxchg.org: fix cgroup.memory=nokmem handling] Link: https://lore.kernel.org/ah9PGv12mqai84ES@cmpxchg.org Link: https://lore.kernel.org/20260527204757.2544958-10-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reported-by: Mikhail Zaslonko <zaslonko@linux.ibm.com> Tested-by: Mikhail Zaslonko <zaslonko@linux.ibm.com> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Acked-by: Usama Arif <usama.arif@linux.dev> Reviewed-by: Kairui Song <kasong@tencent.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Cc: kernel test robot <lkp@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:15 +0000 (16:45 -0400)]
mm: memory: flatten alloc_anon_folio() retry loop
alloc_anon_folio() uses a top-level if (folio) that buries the success
path four levels deep. This makes for awkward long lines and wrapping.
The next patch will add more code here, so flatten this now to keep things
clean and simple.
The next label is already there, use it for !folio.
No functional change intended.
Link: https://lore.kernel.org/20260527204757.2544958-9-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Suggested-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Acked-by: Usama Arif <usama.arif@linux.dev> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Dev Jain <dev.jain@arm.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
memcg_list_lru_alloc() is called every time an object that may end up on
the list_lru is created. It needs to quickly check if the list_lru heads
for the memcg already exist, and allocate them when they don't.
Doing this with folio objects is tricky: folio_memcg() is not stable and
requires either RCU protection or pinning the cgroup. But it's desirable
to make the existence check lightweight under RCU, and only pin the memcg
when we need to allocate list_lru heads and may block.
In preparation for switching the THP shrinker to list_lru, add a helper
function for allocating list_lru heads coming from a folio.
Link: https://lore.kernel.org/20260527204757.2544958-8-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:13 +0000 (16:45 -0400)]
mm: list_lru: introduce caller locking for additions and deletions
Locking is currently internal to the list_lru API. However, a caller
might want to keep auxiliary state synchronized with the LRU state.
For example, the THP shrinker uses the lock of its custom LRU to keep
PG_partially_mapped and vmstats consistent.
To allow the THP shrinker to switch to list_lru, provide normal and
irqsafe locking primitives as well as caller-locked variants of the
addition and deletion functions.
Link: https://lore.kernel.org/20260527204757.2544958-7-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:12 +0000 (16:45 -0400)]
mm: list_lru: deduplicate lock_list_lru()
The MEMCG and !MEMCG paths have the same pattern. Share the code.
Link: https://lore.kernel.org/20260527204757.2544958-6-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:11 +0000 (16:45 -0400)]
mm: list_lru: move list dead check to lock_list_lru_of_memcg()
Only the MEMCG variant of lock_list_lru() needs to check if there is a
race with cgroup deletion and list reparenting. Move the check to the
caller, so that the next patch can unify the lock_list_lru() variants.
Link: https://lore.kernel.org/20260527204757.2544958-5-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:10 +0000 (16:45 -0400)]
mm: list_lru: deduplicate unlock_list_lru()
The MEMCG and !MEMCG variants are the same. lock_list_lru() has the same
pattern when bailing. Consolidate into a common implementation.
Link: https://lore.kernel.org/20260527204757.2544958-4-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:09 +0000 (16:45 -0400)]
mm: list_lru: lock_list_lru_of_memcg() cannot return NULL if !skip_empty
skip_empty is only for the shrinker to abort and skip a list that's empty
or whose cgroup is being deleted.
For list additions and deletions, the cgroup hierarchy is walked upwards
until a valid list_lru head is found, or it will fall back to the node
list. Acquiring the lock won't fail. Remove the NULL checks in those
callers.
Link: https://lore.kernel.org/20260527204757.2544958-3-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reviewed-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Usama Arif <usama.arif@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Johannes Weiner [Wed, 27 May 2026 20:45:08 +0000 (16:45 -0400)]
mm: list_lru: fix set_shrinker_bit() call during race with cgroup deletion
Patch series "mm: switch THP shrinker to list_lru", v5.
The open-coded deferred split queue has issues. It's not NUMA-aware (when
cgroup is enabled), and it's more complicated in the callsites interacting
with it. Switching to list_lru fixes the NUMA problem and streamlines
things. It also simplifies planned shrinker work.
Patch 1 fixes a pre-existing list_lru bug where the shrinker bit is set on
the caller's memcg rather than the ancestor whose sublist the item
actually lands on after a walk-up. Standalone, backportable; the rest of
the series depends on it.
Patches 2-5 are cleanups and small refactors in list_lru code. They're
basically independent, but make the THP shrinker conversion easier.
Patch 6 extends the list_lru API to allow the caller to control the
locking scope. The THP shrinker has private state it needs to keep
synchronized with the LRU state.
Patch 7 extends the list_lru API with a convenience helper to do list_lru
head allocation (memcg_list_lru_alloc) when coming from a folio. Anon
THPs are instantiated in several places, and with the folio reparenting
patches pending, folio_memcg() access is now a more delicate dance. This
avoids having to replicate that dance everywhere.
Patch 8 flattens the alloc_anon_folio() retry loop so the next patch's
list_lru hook lands as a clean addition rather than nested deep inside an
if (folio) block.
Patch 9 finally switches the deferred_split_queue to list_lru.
This patch (of 9):
When list_lru_add() races with cgroup deletion, the shrinker bit is set on
the wrong group and lost. This can cause a shrinker run to miss the
cgroup that actually has the object.
When the passed in memcg is dead, the function finds the first non-dead
parent from the passed in memcg and adds the object there; but the
shrinker bit is set on the memcg that was passed in.
This bug is as old as the shrinker bitmap itself.
Fix it by returning the "effective" memcg from the locking function, and
have the caller use that.
Link: https://lore.kernel.org/20260527204757.2544958-1-hannes@cmpxchg.org Link: https://lore.kernel.org/20260527204757.2544958-2-hannes@cmpxchg.org Fixes: fae91d6d8be5 ("mm/list_lru.c: set bit in memcg shrinker bitmap on first list_lru item appearance") Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Reported-by: Usama Arif <usama.arif@linux.dev> Reported-by: Sashiko Acked-by: Usama Arif <usama.arif@linux.dev> Reviewed-by: Wei Yang <richard.weiyang@gmail.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dave Chinner <david@fromorbit.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Kairui Song <ryncsn@gmail.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Mikhail Zaslonko <zaslonko@linux.ibm.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Nico Pache <npache@redhat.com> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Vasily Gorbik <gor@linux.ibm.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit 0dfe54071d7c8 ("nodemask: Fix return values to be unsigned")
changed a number of nodemask operations that used to return int to
returning a bool instead. However, it did not update the comment block
that described these functions, leaving the documentation incorrect.
Fix the comment block to accurately describe the functions. Also fix a
typo (unsigend --> unsigned), and fix a callsite in mempolicy.c that did
not get updated during the conversion.
No functional changes intended; changes are purely cosmetic.
Link: https://lore.kernel.org/20260529202755.1846800-1-joshua.hahnjy@gmail.com Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Alistair Popple <apopple@nvidia.com> Cc: Byungchul Park <byungchul@sk.com> Cc: David Hildenbrand <david@kernel.org> Cc: Gregory Price <gourry@gourry.net> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Rasmus Villemoes <linux@rasmusvillemoes.dk> Cc: Ying Huang <ying.huang@linux.alibaba.com> Cc: Yury Norov (NVIDIA) <yury.norov@gmail.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Brian Masney [Thu, 28 May 2026 13:45:10 +0000 (09:45 -0400)]
docs: mm: clarify that user_reserve_kbytes has no effect when overcommit_memory is set to 0 or 1
Looking at __vm_enough_memory() in mm/util.c, user_reserve_kbytes has no
effect when overcommit_memory is set to 0 or 1. The documentation for
overcommit_memory already references user_reserve_kbytes when the flag
is set to 2.
Let's go ahead and add a clarification to user_reserve_kbytes in vm.rst
that it has no effect when overcommit_memory is set to 0 or 1.
Link: https://lore.kernel.org/20260528-mm-clarify-docs-v1-1-aa88e83b4bfd@redhat.com Signed-off-by: Brian Masney <bmasney@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Shuah Khan <skhan@linuxfoundation.org> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Brian Masney [Thu, 28 May 2026 13:56:14 +0000 (09:56 -0400)]
MAINTAINERS: add vm.rst to memory management core
The vm.rst file is currently not listed in the MAINTAINERS file, so let's
go ahead and add to the MM core subsystem so that the maintainers are CCed
when changes to the documentation are proposed.
Link: https://lore.kernel.org/20260528-mm-vm-rst-maintainers-file-v1-1-306631c0a610@redhat.com Signed-off-by: Brian Masney <bmasney@redhat.com> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org> Reviewed-by: Liam R. Howlett (Oracle) <liam@infradead.org> Reviewed-by: SeongJae Park <sj@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
zram: clear trailing bytes of compressed writeback pages
Patch series "zram: writeback fixes", v2.
Brian (privately) reported a "leak" of writeback bitmap in certain cases,
so that backing device can store less pages; and a theoretical data leak
in the trailing bytes of compressed writeback pages. Both issues are low
risk.
This patch (of 2):
When compressed writeback is available writtenback pages contain "garbage"
in PAGE_SIZE - obj_size trailing bytes. That "garbage" is, basically,
whatever data that page held before we got it for writeback. To get
advantage of it an attacker needs to be able to read from active backing
swap device, which is already catastrophic. Still, just in case, zero out
those trailing bytes before writeback to a backing device so that we only
store swap-ed out data there.
zram_writeback_slots() loop can terminate with valid reserved backing
device blk_idx. The problem is that cleanup code doesn't release that
reserved blk_idx before zram_writeback_slots() returns, which leads to
blk_idx leak (it becomes permanently busy and can not be used for actual
writeback.) This does not lead to any system instabilities, it only means
that we can writeback less pages. The scenario is hard to hit in practice
as it requires writeabck to race with modification (slot-free or
overwrite) of the final post-processing slot.
Release reserved but unused blk_idx before returning from
zram_writeback_slots().
Link: https://lore.kernel.org/20260526022754.2377730-2-senozhatsky@chromium.org Fixes: f405066a1f0db ("zram: introduce writeback bio batching") Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org> Suggested-by: Brian Geffon <bgeffon@google.com> Cc: Jens Axboe <axboe@kernel.dk> Cc: Minchan Kim <minchan@kernel.org> Cc: Richard Chang <richardycc@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Shakeel Butt [Tue, 26 May 2026 03:39:31 +0000 (20:39 -0700)]
memcg: multi objcg charge support
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA node
so that reparenting LRU folios can take per-node lru locks. As a side
effect, the per-CPU obj_stock_pcp -- which caches exactly one cached_objcg
-- thrashes on workloads where threads of the same memcg run on different
NUMA nodes. The kernel test robot reported a 67.7% regression on
stress-ng.switch.ops_per_sec from this pattern.
Mirror the multi-slot pattern already used by memcg_stock_pcp: turn
nr_bytes and cached_objcg into NR_OBJ_STOCK-element arrays, scan all slots
on consume/refill/account, prefer empty slots when inserting, and evict a
slot round-robin only when full. With multiple slots a CPU can hold the
per-node objcg variants of one memcg plus a few siblings without ever
forcing a drain.
A single int8_t index records which slot the cached slab stats belong to;
the stats are flushed on slot or pgdat change. With NR_OBJ_STOCK = 5 the
layout (verified with pahole) is:
Shakeel Butt [Tue, 26 May 2026 03:39:30 +0000 (20:39 -0700)]
memcg: int16_t for cached slab stats
Currently struct obj_stock_pcp stores cached slab stats in 'int' which is
4 bytes per counter on 64-bit machines. Switch them to int16_t to shrink
the cached metadata.
The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at
PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB
pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a
sufficiently long run of accumulations would overflow the cache. Add an
explicit S16_MAX guard before each add: when the next add would push
abs(*bytes) past S16_MAX, fold the cached value into @nr and flush
directly via mod_objcg_mlstate() before the accumulation.
Link: https://lore.kernel.org/20260526033931.1760588-4-shakeel.butt@linux.dev Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev> Tested-by: kernel test robot <oliver.sang@intel.com> Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org> Acked-by: Qi Zheng <qi.zheng@linux.dev> Acked-by: Muchun Song <muchun.song@linux.dev> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Shakeel Butt [Tue, 26 May 2026 03:39:29 +0000 (20:39 -0700)]
memcg: uint16_t for nr_bytes in obj_stock_pcp
Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int' which
is 4 bytes on 64-bit machines. Switch the field to uint16_t to shrink the
per-CPU cache.
The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and _256KB
(see HAVE_PAGE_SIZE_* in arch/Kconfig). After the PAGE_SIZE-aligned flush
in __refill_obj_stock(), the sub-page remainder fits in uint16_t up
through 64KiB pages where PAGE_SIZE - 1 == U16_MAX, but on 256KiB pages
PAGE_SIZE - 1 == 0x3FFFF exceeds U16_MAX. The accumulator also needs to
stay within uint16_t between page-aligned flushes on 64KiB pages where
PAGE_SIZE itself is U16_MAX + 1.
Accumulate the new total in an 'unsigned int' local, then on PAGE_SHIFT <=
16 flush whenever the accumulator would hit U16_MAX; together with the
existing allow_uncharge flush at PAGE_SIZE this keeps the uint16_t safe.
On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and powerpc
44x, both 32-bit), uint16_t cannot represent the sub-page remainder.
Define obj_stock_bytes_t as 'unsigned int' on those archs so nr_bytes can
hold the full remainder and the normal page-boundary flush in
__refill_obj_stock() and the page extraction in drain_obj_stock() both
work correctly.
The single-cache-line layout target only applies to PAGE_SHIFT <= 16;
those archs are 32-bit embedded and not the optimization target.
Link: https://lore.kernel.org/20260526033931.1760588-3-shakeel.butt@linux.dev Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev> Tested-by: kernel test robot <oliver.sang@intel.com> Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org> Acked-by: Qi Zheng <qi.zheng@linux.dev> Acked-by: Muchun Song <muchun.song@linux.dev> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Shakeel Butt [Tue, 26 May 2026 03:39:28 +0000 (20:39 -0700)]
memcg: store node_id instead of pglist_data pointer
Patch series "memcg: shrink obj_stock_pcp and cache multiple objcgs", v3.
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA node
so that reparenting LRU folios can take per-node lru locks. As a side
effect, the per-CPU obj_stock_pcp -- which caches a single cached_objcg
pointer -- thrashes on workloads where threads of the same memcg run on
different NUMA nodes. The kernel test robot reported a 67.7% regression
on stress-ng.switch.ops_per_sec from this pattern.
Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg
pointer") landed as a temporary fix by treating sibling per-node objcgs as
equivalent for the cache lookup, intended to be reverted once per-node
kmem accounting is introduced. This series takes a more general approach:
cache multiple objcgs per CPU using the multi-slot pattern memcg_stock_pcp
already uses, so the per-node objcg variants of one memcg can all coexist
in the stock without ever forcing a drain. The temporary fix can then be
reverted.
To avoid increasing the per-CPU cache footprint, the first three patches
shrink the existing single-slot obj_stock_pcp fields. The final patch
converts cached_objcg and nr_bytes into NR_OBJ_STOCK=5 slot arrays and
reorders the struct so the entire consume/refill/account hot path fits
within a single 64-byte cache line on non-debug 64-bit builds (verified
with pahole).
This patch (of 4):
The struct obj_stock_pcp stores a pointer to pglist_data for the slab
stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The
pointer is not strictly required: NODE_DATA() can recover it from the node
id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE as the
"no stats cached" sentinel.
At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is
plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.
Link: https://lore.kernel.org/20260526033931.1760588-1-shakeel.butt@linux.dev Link: https://lore.kernel.org/20260526033931.1760588-2-shakeel.butt@linux.dev Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type") Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev> Tested-by: kernel test robot <oliver.sang@intel.com> Acked-by: Muchun Song <muchun.song@linux.dev> Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org> Acked-by: Qi Zheng <qi.zheng@linux.dev> Cc: Alexandre Ghiti <alex@ghiti.fr> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Michal Hocko <mhocko@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
selftests/memfd: fix -Wmaybe-uninitialized warning in memfd_test
Patch series "selftests/memfd: fix compilation warnings".
This patchset fixes warnings about unused but initialized variables, and
unused dummy buffer passed to pwrite() syscall in the tests.
This patch (of 2):
memfd_test.c: In function 'mfd_fail_grow_write.part.0':
memfd_test.c:685:13: warning: '<unknown>' may be used uninitialized
[-Wmaybe-uninitialized]
685 | l = pwrite(fd, buf, mfd_def_size * 8, 0);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
pwrite() is declared with attribute 'access (read_only, 2, 3)', so GCC
knows it reads from the buffer. malloc() returns uninitialized memory,
hence the warning. Use calloc() to zero-initialize the buffer. The
actual contents don't matter here since the test verifies that pwrite()
fails on a sealed memfd.
Ran Xiaokai [Mon, 25 May 2026 10:27:00 +0000 (10:27 +0000)]
mm: shmem: refactor thpsize_shmem_enabled_show() with helper arrays
Replace the hardcoded if/else chain of test_bit() calls and string
literals in thpsize_shmem_enabled_show() with a loop over
huge_shmem_orders_by_mode[] and huge_shmem_enabled_mode_strings[] arrays.
This makes thpsize_shmem_enabled_show() consistent with
thpsize_shmem_enabled_store() and eliminates duplicated mode name strings.
Link: https://lore.kernel.org/20260525102700.68707-3-ranxiaokai627@163.com Signed-off-by: Ran Xiaokai <ran.xiaokai@zte.com.cn> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Breno Leitao <leitao@debian.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com> Tested-by: Lance Yang <lance.yang@linux.dev> Acked-by: David Hildenbrand (arm) <david@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Hugh Dickins <hughd@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Ran Xiaokai [Mon, 25 May 2026 10:26:59 +0000 (10:26 +0000)]
mm: shmem: refactor thpsize_shmem_enabled_store() with sysfs_match_string()
Patch series "refactors thpsize_shmem_enabled_store() and
thpsize_shmem_enabled_show()", v4.
This patch (of 2):
Inspired by commit 82d9ff648c6c ("mm: huge_memory: refactor
anon_enabled_store() with set_anon_enabled_mode()"), refactor
thpsize_shmem_enabled_store() using sysfs_match_string(). This eliminates
the duplicated spin_lock/unlock(), set/clear_bit(), calls across all
branches, reducing code duplication.
Behavioral change:
Call start_stop_khugepaged() only when the mode actually changes.
If unchanged, call set_recommended_min_free_kbytes() to preserve
legacy watermark behavior. This avoids unnecessary khugepaged restarts.
Tested with selftests ./run_kselftest.sh -t mm:ksft_thp.sh,
all test cases passed.
Link: https://lore.kernel.org/20260525102700.68707-1-ranxiaokai627@163.com Link: https://lore.kernel.org/20260525102700.68707-2-ranxiaokai627@163.com Signed-off-by: Ran Xiaokai <ran.xiaokai@zte.com.cn> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Barry Song <baohua@kernel.org> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com> Tested-by: Lance Yang <lance.yang@linux.dev> Acked-by: David Hildenbrand (arm) <david@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Breno Leitao <leitao@debian.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Hugh Dickins <hughd@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Nico Pache <npache@redhat.com> Cc: Ran Xiaokai <ran.xiaokai@zte.com.cn> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Usama Arif [Mon, 25 May 2026 14:57:51 +0000 (07:57 -0700)]
mm: make mmap_miss accounting symmetric for VM_SEQ_READ
do_sync_mmap_readahead() skips both the mmap_miss increment and the
MMAP_LOTSAMISS check for VM_SEQ_READ mappings, since sequential access is
non-speculative and should always read ahead. The two decrement sites in
do_async_mmap_readahead() and filemap_map_pages() do not mirror this skip,
so concurrent faults on a VM_SEQ_READ mapping can still drive
ra->mmap_miss down to zero through the decrement paths even though nothing
in the sync path ever increments it. The counter itself is per-file
(file->f_ra.mmap_miss), so it can be moved by any VMA mapping the file,
not just the one currently faulting.
Skip the decrement for VM_SEQ_READ in both decrement sites so the counter
only moves for mappings that also participate in the increment side. No
functional change for VM_SEQ_READ users, since the increment-side gate
already prevents the counter from being consulted on their behalf, but it
stops a VM_SEQ_READ mapping from biasing the counter for other mappings of
the same file.
Link: https://lore.kernel.org/20260525145751.2671248-1-usama.arif@linux.dev Signed-off-by: Usama Arif <usama.arif@linux.dev> Closes: https://lore.kernel.org/all/8edc8cd0-f65c-4456-9b3f-362e744c9a96@linux.dev/ Reviewed-by: William Kucharski <william.kucharski@linux.dev> Reviewed-by: Jan Kara <jack@suse.cz> Cc: David Hildenbrand <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Add tearing tests for /proc/pid/smaps file. New tests reuse the same
logic as with maps file but skipping all the data except for the VMA
addresses, which are the only part relevant for the tearing tests. Skip
PROCMAP_QUERY parts of the tests because smaps does not implement that
ioctl.
Link: https://lore.kernel.org/20260426062718.1238437-4-surenb@google.com Signed-off-by: Suren Baghdasaryan <surenb@google.com> Reviewed-by: Liam R. Howlett <liam@infradead.org> Cc: Jann Horn <jannh@google.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: "Paul E . McKenney" <paulmck@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Shuah Khan <shuah@kernel.org> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
selftests/proc: ensure the test is performed at the right page boundary
When running tearing tests we need to ensure the pages we use include VMAs
that were mapped by the child process for this test. Currently we always
use the first two pages, checking VMAs at their boundaries and this works,
however once we add tests for /proc/pid/smaps, the first two pages might
not contain the VMAs that child modifies. Locate the page that contains
the first VMA mapped by the child and use that and the next page for the
test.
Link: https://lore.kernel.org/20260426062718.1238437-3-surenb@google.com Signed-off-by: Suren Baghdasaryan <surenb@google.com> Reviewed-by: Liam R. Howlett <liam@infradead.org> Cc: Jann Horn <jannh@google.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: "Paul E . McKenney" <paulmck@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Shuah Khan <shuah@kernel.org> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
fs/proc/task_mmu: read proc/pid/{smaps|numa_maps} under per-vma lock
Patch series "use vma locks for proc/pid/{smaps|numa_maps} reads", v2.
Use per-vma locks when reading /proc/pid/smaps and /proc/pid/numa_maps
similar to /proc/pid/maps to reduce contention on central mmap_lock. One
major difference between maps and smaps/numa_maps reading is that the
latter executes page table walk which can't be done under RCU due to a
possibility of sleeping. Therefore we drop RCU read lock before this walk
while keeping the VMA locked. After the walk we retake RCU read lock,
reset VMA iterator and proceed with the next VMA.
The last two patches extend /proc/pid/maps test to cover /proc/pid/smaps
reading during concurrent address space modification.
This patch (of 3):
proc/pid/{smaps|numa_maps} can be read using the combination of RCU and
VMA read locks, similar to proc/pid/maps. RCU is required to safely
traverse the VMA tree and VMA lock stabilizes the VMA being processed and
the pagetable walk.
Link: https://lore.kernel.org/20260426062718.1238437-1-surenb@google.com Link: https://lore.kernel.org/20260426062718.1238437-2-surenb@google.com Signed-off-by: Suren Baghdasaryan <surenb@google.com> Reviewed-by: Liam R. Howlett <liam@infradead.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: "Paul E . McKenney" <paulmck@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Shuah Khan <shuah@kernel.org> Cc: Wei Yang <richard.weiyang@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:07:06 +0000 (02:07 +0800)]
mm/vmscan: unify writeback reclaim statistic and throttling
Currently MGLRU and non-MGLRU handle the reclaim statistic and writeback
handling very differently, especially throttling. Basically MGLRU just
ignored the throttling part.
Let's just unify this part, use a helper to deduplicate the code so both
setups will share the same behavior.
Kairui Song [Mon, 27 Apr 2026 18:07:03 +0000 (02:07 +0800)]
mm/mglru: remove no longer used reclaim argument for folio protection
Now dirty reclaim folios are handled after isolation, not before, since
dirty reactivation must take the folio off LRU first, and that helps to
unify the dirty handling logic.
So this argument is no longer needed. Just remove it.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-12-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chen Ridong <chenridong@huaweicloud.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:07:02 +0000 (02:07 +0800)]
mm/mglru: simplify and improve dirty writeback handling
Right now the flusher wakeup mechanism for MGLRU is less responsive and
unlikely to trigger compared to classical LRU. The classical LRU wakes
the flusher if one batch of folios passed to shrink_folio_list is
unevictable due to under writeback. MGLRU instead check and handle this
after the whole reclaim loop is done.
We previously even saw OOM problems due to passive flusher, which were
fixed but still not perfect [1].
We have just unified the dirty folio counting and activation routine, now
just move the dirty flush into the loop right after shrink_folio_list.
This improves the performance a lot for workloads involving heavy
writeback and prepares for throttling too.
Test with YCSB workloadb showed a major performance improvement:
The performance is a lot better with significantly lower refault. We also
observed similar or higher performance gain for other real-world
workloads.
We were concerned that the dirty flush could cause more wear for SSD: that
should not be the problem here, since the wakeup condition is when the
dirty folios have been pushed to the tail of LRU, which indicates that
memory pressure is so high that writeback is blocking the workload
already.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-11-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Link: https://lore.kernel.org/linux-mm/20241026115714.1437435-1-jingxiangzeng.cas@gmail.com/ Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chen Ridong <chenridong@huaweicloud.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:07:01 +0000 (02:07 +0800)]
mm/mglru: use the common routine for dirty/writeback reactivation
Currently MGLRU will move the dirty writeback folios to the second oldest
gen instead of reactivate them like the classical LRU. This might help to
reduce the LRU contention as it skipped the isolation. But as a result we
will see these folios at the LRU tail more frequently leading to
inefficient reclaim.
Besides, the dirty / writeback check after isolation in shrink_folio_list
is more accurate and covers more cases. So instead, just drop the special
handling for dirty writeback, use the common routine and re-activate it
like the classical LRU.
This should in theory improve the scan efficiency. These folios will be
rotated back to LRU tail once writeback is done so there is no risk of
hotness inversion. And now each reclaim loop will have a higher success
rate. This also prepares for unifying the writeback and throttling
mechanism with classical LRU, we keep these folios far from tail so
detecting the tail batch will have a similar pattern with classical LRU.
The micro optimization that avoids LRU contention by skipping the
isolation is gone, which should be fine. Compared to IO and writeback
cost, the isolation overhead is trivial.
And using the common routine also keeps the folio's referenced bits (tier
bits), which could improve metrics in the long term. Also no more need to
clean reclaim bit as the common routine will make use of it.
Note the common routine updates a few throttling and writeback counters,
which are not used, and never have been for the MGLRU case. We will start
making use of these in later commits.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-10-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Chen Ridong <chenridong@huaweicloud.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:07:00 +0000 (02:07 +0800)]
mm/mglru: remove redundant swap constrained check upon isolation
Remove the swap-constrained early reject check upon isolation. This check
is a micro optimization when swap IO is not allowed, so folios are
rejected early. But it is redundant and overly broad since
shrink_folio_list() already handles all these cases with proper
granularity.
Notably, this check wrongly rejected lazyfree folios, and it doesn't cover
all rejection cases. shrink_folio_list() uses may_enter_fs(), which
distinguishes non-SWP_FS_OPS devices from filesystem-backed swap and does
all the checks after folio is locked, so flags like swap cache are stable.
This check also covers dirty file folios, which are not a problem now
since sort_folio() already bumps dirty file folios to the next generation,
but causes trouble for unifying dirty folio writeback handling.
And there should be no performance impact from removing it. We may have
lost a micro optimization, but unblocked lazyfree reclaim for NOIO
contexts, which is not a common case in the first place.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-9-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Chen Ridong <chenridong@huaweicloud.com> Reviewed-by: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:06:59 +0000 (02:06 +0800)]
mm/mglru: don't abort scan immediately right after aging
Right now, if eviction triggers aging, the reclaimer will abort. This is
not the optimal strategy for several reasons.
Aborting the reclaim early wastes a reclaim cycle when under pressure, and
for concurrent reclaim, if the LRU is under aging, all concurrent
reclaimers might fail. And if the age has just finished, new cold folios
exposed by the aging are not reclaimed until the next reclaim iteration.
What's more, the current aging trigger is quite lenient, having 3 gens
with a reclaim priority lower than default will trigger aging, and blocks
reclaiming from one memcg. This wastes reclaim retry cycles easily. And
in the worst case, if the reclaim is making slower progress and all
following attempts fail due to being blocked by aging, it triggers
unexpected early OOM.
And if a lruvec requires aging, it doesn't mean it's hot. Instead, the
lruvec could be idle for quite a while, and hence it might contain lots of
cold folios to be reclaimed.
While it's helpful to rotate memcg LRU after aging for global reclaim, as
global reclaim fairness is coupled with the rotation in shrink_many, memcg
fairness is instead handled by cgroup iteration in shrink_node_memcgs.
So, for memcg level pressure, this abort is not the key part for keeping
the fairness. And in most cases, there is no need to age, and fairness
must be achieved by upper-level reclaim control.
So instead, just keep the scanning going unless one whole batch of folios
failed to be isolated or enough folios have been scanned, which is
triggered by evict_folios returning 0. And only abort for global reclaim
after one batch, so when there are fewer memcgs, progress is still made,
and the fairness mechanism described above still works fine.
And in most cases, the one more batch attempt for global reclaim might
just be enough to satisfy what the reclaimer needs, hence improving global
reclaim performance by reducing reclaim retry cycles.
Rotation is still there after the reclaim is done, which still follows the
comment in mmzone.h. And fairness still looking good.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-8-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: Chen Ridong <chenridong@huaweicloud.com> Reviewed-by: Barry Song <baohua@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:06:58 +0000 (02:06 +0800)]
mm/mglru: use a smaller batch for reclaim
With a fixed number to reclaim calculated at the beginning, making each
following step smaller should reduce the lock contention and avoid
over-aggressive reclaim of folios, as it will abort earlier when the
number of folios to be reclaimed is reached.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-7-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: Chen Ridong <chenridong@huaweicloud.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/mglru: avoid reclaim type fall back when isolation makes no progress
While isolation makes no progress in scan_folios(), we quickly fall back
to the other type in isolate_folios(). This is incorrect, as the current
type may still have sufficient folios. Falling back can undermine the
positive_ctrl_err() result from get_type_to_scan(), which is derived from
swappiness.
So just continue scanning this type for another round.
Worth noting if the cold generations are all reclaimed, scan will no
longer make any progress either, which may undermine the swappiness again.
This is not a new issue and hence better be fixed later [1].
Link: https://lore.kernel.org/linux-mm/CAGsJ_4zjdOYEtuO6gNjABm7NDxW0skzBFNRNee-k2D6VwsYEQA@mail.gmail.com/ Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-6-02fabb92dc43@tencent.com Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org> Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Kairui Song <kasong@tencent.com> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Chen Ridong <chenridong@huaweicloud.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:06:56 +0000 (02:06 +0800)]
mm/mglru: scan and count the exact number of folios
Make the scan helpers return the exact number of folios being scanned or
isolated. Since the reclaim loop now has a natural scan budget that
controls the scan progress, returning the scan number and consuming the
budget makes the scan more accurate and easier to follow.
The number of scanned folios for each iteration is always larger than 0,
unless the reclaim must stop for a forced aging, so there is no more need
for any special handling when there is no progress made:
- `return isolated || !remaining ? scanned : 0` in scan_folios: both
the function and the call now just return the exact scan count, combined
with the scan budget introduced in the previous commit to avoid livelock
or under scan.
- `scanned += try_to_inc_min_seq` in evict_folios: adding a bool as a
scan count was kind of confusing and no longer needed, as scan number
should never be zero as long as there are still evictable gens. We may
encounter a empty old gen that returns 0 scan count, to avoid that, do a
try_to_inc_min_seq before toisolation which have slight to none overhead
in most cases.
- `evictable_min_seq + MIN_NR_GENS > max_seq` guard in evict_folios: the
per-type get_nr_gens == MIN_NR_GENS check in scan_folios naturally
returns 0 when only two gens remain and breaks the loop.
Also change try_to_inc_min_seq to return void, as its return value is no
longer used by any caller. Call it before isolate_folios to flush any
empty gens left by external folio freeing, and again after isolate_folios
when scanning moved or protected folios may have emptied the oldest gen.
The scan still stops if only two gens are left, as the scan number will be
zero. This matches the previous behavior. This forced gen protection may
be removed or softened later to improve reclaim further.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-5-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: Chen Ridong <chenridong@huaweicloud.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:06:55 +0000 (02:06 +0800)]
mm/mglru: restructure the reclaim loop
The current loop will calculate the scan number on each iteration. The
number of folios to scan is based on the LRU length, with some unclear
behaviors, e.g, the scan number is only shifted by reclaim priority when
aging is not needed or when at the default priority, and it couples the
number calculation with aging and rotation.
Adjust, simplify it, and decouple aging and rotation. Just calculate the
scan number for once at the beginning of the reclaim, always respect the
reclaim priority, and make the aging and rotation more explicit.
This slightly changes how aging and offline memcg reclaim works:
Previously, aging was skipped at DEF_PRIORITY even when eviction was no
longer possible, so the reclaimer wasted an iteration until the priority
escalated. Now aging runs immediately whenever it is needed to make
progress; the DEF_PRIORITY skip only applies when eviction is still
viable. This may avoid wasted iterations that over-reclaim slab and break
reclaim balance in multi-cgroup setups.
Similar for offline memcg. Previously, offline memcg wouldn't be aged
unless it didn't have any evictable folios. Now, we might age it if it
has only 3 generations, which should be fine. On one hand, offline memcg
might still hold long-term folios, and in fact, a long-existing offline
memcg must be pinned by some long-term folios like shmem. These folios
might be used by other memcg, so aging them as ordinary memcg seems
correct. Besides, aging enables further reclaim of an offlined memcg,
which will certainly happen if we keep shrinking it. And offline memcg
might soon be no longer an issue with reparenting.
Overall, the memcg LRU rotation, as described in mmzone.h, remains the
same.
Note that because the scan budget is now pinned at loop entry, tiny lruvec
might skip this reclaim pass, also skipping aging, which could be
beneficial as aging is not helpful since it will still be un-reclaimable
after aging. Reclaim will go on as usual once priority escalates.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-4-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chen Ridong <chenridong@huaweicloud.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:06:54 +0000 (02:06 +0800)]
mm/mglru: relocate the LRU scan batch limit to callers
Same as active / inactive LRU, MGLRU isolates and scans folios in batches.
The batch split is done hidden deep in the helper, which makes the code
harder to follow. The helper's arguments are also confusing since callers
usually request more folios than the batch size, so the helper almost
never processes the full requested amount.
Move the batch splitting into the top loop to make it cleaner, there
should be no behavior change.
Link: https://lore.kernel.org/20260428-mglru-reclaim-v7-3-02fabb92dc43@tencent.com Signed-off-by: Kairui Song <kasong@tencent.com> Reviewed-by: Axel Rasmussen <axelrasmussen@google.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Chen Ridong <chenridong@huaweicloud.com> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: David Stevens <stevensd@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kalesh Singh <kaleshsingh@google.com> Cc: Leno Hou <lenohou@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vernon Yang <vernon2gm@gmail.com> Cc: Wei Xu <weixugc@google.com> Cc: Yafang <laoar.shao@gmail.com> Cc: Yuanchu Xie <yuanchu@google.com> Cc: Yu Zhao <yuzhao@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Kairui Song [Mon, 27 Apr 2026 18:06:52 +0000 (02:06 +0800)]
mm/mglru: consolidate common code for retrieving evictable size
Patch series "mm/mglru: improve reclaim loop and dirty folio", v7.
This series cleans up and slightly improves MGLRU's reclaim loop and dirty
writeback handling. As a result, we can see an up to ~30% increase in
some workloads like MongoDB with YCSB and a huge decrease in file refault,
no swap involved. Other common benchmarks have no regression, and LOC is
reduced, with less unexpected OOM, too.
Some of the problems were found in our production environment, and others
were mostly exposed while stress testing during the development of the
LSM/MM/BPF topic on improving MGLRU [1]. This series cleans up the code
base and fixes several performance issues, preparing for further work.
MGLRU's reclaim loop is a bit complex, and hence these problems are
somehow related to each other. The aging, scan number calculation, and
reclaim loop are coupled together, and the dirty folio handling logic is
quite different, making the reclaim loop hard to follow and the dirty
flush ineffective.
This series slightly cleans up and improves these issues using a scan
budget by calculating the number of folios to scan at the beginning of the
loop, and decouples aging from the reclaim calculation helpers. Then,
move the dirty flush logic inside the reclaim loop so it can kick in more
effectively. These issues are somehow related, and this series handles
them and improves MGLRU reclaim in many ways.
Test results: All tests are done on a 48c96t NUMA machine with 2 nodes and
a 128G memory machine using NVME as storage. Classical (non-MGLRU) LRU
numbers are included as "MGLRU disabled" for each benchmark below; see [8]
and [9] for the longer write-up.
MongoDB
=======
Running YCSB workloadb [2] (recordcount:20000000 operationcount:6000000,
threads:32), which does 95% read and 5% update to generate mixed read and
dirty writeback. MongoDB is set up in a 10G cgroup using Docker, and the
WiredTiger cache size is set to 4.5G, using NVME as storage. This is
close to the case we observed regressing in our production environment:
mixed read and writeback pressure, so it is a practical case for
evaluation.
Not using SWAP. The intent is to isolate the file LRU writeback path.
Enabling SWAP would just add noise from anonymous reclaim.
We can see a significant performance improvement after this series. The
test is done on NVME and the performance gap would be even larger for slow
devices, such as HDD or network storage. We observed over 100% gain for
some workloads with slow IO.
Note, classical LRU is still faster for this benchmark, MGLRU may catch up
later with further work [7].
Chrome & Node.js [3]
====================
Using Yu Zhao's test script [3], testing on a x86_64 NUMA machine with 2
nodes and 128G memory, using 256G ZRAM as swap and spawn 32 memcg 64
workers. Many memcgs each applying roughly equal pressure exercises the
LRU's ability to detect/protect each tenant's working set and to balance
reclamation fairly between tenants, which makes this a meaningful test for
the reclaim mechanism.
Fairness is reported via Jain's fairness index (1.0 means all tenants get
exactly equal allocation, lower is worse). Under equal pressure, all
memcgs should make roughly equal forward progress. See [8] for the longer
rationale and per-memcg breakdown.
Reclaim is still fair and effective, total requests number seems slightly
better.
OOM issue with aging and throttling
===================================
For the throttling OOM issue, it can be easily reproduced using dd and
cgroup limit as demonstrated and fixed by a later patch in this series.
The aging OOM is a bit tricky, a specific reproducer can be used to
simulate what we encountered in production environment [4]: Spawns
multiple workers that keep reading the given file using mmap, and pauses
for 120ms after one file read batch. It also spawns another set of
workers that keep allocating and freeing a given size of anonymous memory.
The total memory size exceeds the memory limit (eg. 14G anon + 8G file,
which is 22G vs a 16G memcg limit).
- MGLRU disabled:
Finished 128 iterations.
- MGLRU enabled:
OOM with following info after about ~10-20 iterations:
[ 62.624130] file_anon_mix_p invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
[ 62.624999] memory: usage 16777216kB, limit 16777216kB, failcnt 24460
[ 62.640200] swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[ 62.640823] Memory cgroup stats for /demo:
[ 62.641017] anon 10604879872
[ 62.641941] file 6574858240
OOM occurs despite there being still evictable file folios.
- MGLRU enabled after this series:
Finished 128 iterations.
Worth noting there is another OOM related issue reported in V1 of this
series, which is tested and looking OK now [5].
MySQL:
======
Testing with innodb_buffer_pool_size=26106127360, in a 2G memcg, using
ZRAM as swap and test command:
sysbench /usr/share/sysbench/oltp_read_only.lua --mysql-db=sb \
--tables=48 --table-size=2000000 --threads=48 --time=600 run
A 24G InnoDB buffer pool inside a 2G memcg with ZRAM as swap forces
aggressive eviction of cached database anon pages, which exercises the
LRU's hot page detection and the eviction path under swap pressure. The
workload is practical, and the pressure is higher than what we usually see
in production but it is intended to expose the extreme case.
Random buffered mmap read on a ramdisk strips out storage variance and
stresses purely the LRU's ability to evict and recycle the page cache
under heavy random read pressure.
Also seem only noise level changes and no regression or slightly better.
Build kernel:
=============
Build kernel test using ZRAM as swap, kernel source on tmpfs, in a memcg
with memory.max=3G, using make -j96 and defconfig, measuring system time,
6 test run each. Building the kernel is a classical mixed anon + file
workload (lots of small file reads/writes plus parallel anon allocations
from cc/ld) and is representative of many real compilation jobs.
Also seem only noise level changes, no regression or very slightly better.
Android:
========
Xinyu reported a performance gain on Android, too, with this series. The
test consisted of cold-starting multiple applications sequentially under
moderate system load [6]; this is a real Android user-visible scenario,
dominated by the LRU's ability to keep the right working set resident and
re-fault launch-critical pages quickly.
Before:
Launch Time Summary (all apps, all runs)
Mean 868.0ms
P50 888.0ms
P90 1274.2ms
P95 1399.0ms
After:
Launch Time Summary (all apps, all runs)
Mean 850.5ms (-2.07%)
P50 861.5ms (-3.04%)
P90 1179.0ms (-8.05%)
P95 1228.0ms (-12.2%)
This patch (of 15):
Merge commonly used code for counting evictable folios in a lruvec.
userfaultfd: make functions that are not used outside uffd static
After merging fs/userfaultfd.c into mm/userfaultfd.c, several functions
that were previously shared between the two files are now only used within
mm/userfaultfd.c.
Make them static and remove their declarations from
include/linux/userfaultfd_k.h.
Link: https://lore.kernel.org/20260523173759.3964908-3-rppt@kernel.org Assisted-by: Copilot:claude-opus-4-6 Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: Christian Brauner <brauner@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jan Kara <jack@suse.cz> Cc: "Kirill A. Shutemov" <kirill@shutemov.name> Cc: Peter Xu <peterx@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c
Patch series "userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c",
v3.
These patches merge fs/userfaultfd.c into mm/userfaultfd.c and make
functions used only inside mm/userfaultfd.c static.
This patch (of 2):
Historically userfaultfd implementation has been split between
fs/userfaultfd.c and mm/userfaultfd.c.
The mm/ part implemented memory management operations, while the fs/ part
implemented file descriptor handling and called into the mm/ part for the
actual memory management work.
This separation is quite artificial and fs/userfaultfd.c does not seem to
belong to fs/ because it's only a user if vfs APIs and like for other
users, for example, memfd and secretmem, the file descriptor handling
could live in mm/ as well.
"Append" fs/userfaultfd.c to mm/userfaultfd and update fs/Makefile and
MAINTAINERS accordingly.
No intended functional changes.
Link: https://lore.kernel.org/20260523173759.3964908-1-rppt@kernel.org Link: https://lore.kernel.org/20260523173759.3964908-2-rppt@kernel.org Assisted-by: Copilot:claude-opus-4-6 Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Christian Brauner (Amutable) <brauner@kernel.org> Cc: Al Viro <viro@zeniv.linux.org.uk> Cc: David Hildenbrand <david@kernel.org> Cc: Jan Kara <jack@suse.cz> Cc: "Kirill A. Shutemov" <kirill@shutemov.name> Cc: Peter Xu <peterx@redhat.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/mglru: use folio_mark_accessed to replace folio_set_active
MGLRU gives high priority to folios mapped in page tables. As a result,
folio_set_active() is invoked for all folios read during page faults. In
practice, however, readahead can bring in many folios that are never
accessed via page tables.
A previous attempt by Lei Liu proposed introducing a separate LRU for
readahead[1] to make readahead pages easier to reclaim, but that approach
is likely over-engineered.
Before commit 4d5d14a01e2c ("mm/mglru: rework workingset protection"),
folios with PG_active were always placed in the youngest generation,
leading to over-protection and increased refaults. After that commit,
PG_active folios are placed in the second youngest generation, which is
still too optimistic given the presence of readahead. In contrast, the
classic active/inactive scheme is more conservative.
This patch switches to using folio_mark_accessed() and
begins prefaulted file folios from the second oldest
generation instead of active generations.
We should also adjust the following accordingly:
- WORKINGSET_ACTIVATE: aligned with setting active for refaulted workingset
folios;
- lru_gen_folio_seq(): place (pre)faulted file folios into the second
oldest generation;
- promote second-scanned folios to workingset in
folio_check_references(): we now have to depend on
folio_lru_refs() > 1, since we previously relied on PG_referenced
being set during the first scan, but PG_referenced is now set
earlier.
On x86, running a kernel build inside a memcg with a 1GB memory
limit using 20 threads.
Wang Wensheng [Sun, 24 May 2026 03:10:53 +0000 (11:10 +0800)]
kasan/test: only do kmalloc_double_kzfree for generic mode
kmalloc_double_kzfree() would corrupt kernel memory when the just freed
memory were allocated by another thread before the second call to
kfree_sensitive() and the new allocation tag happened to match the old
one.
This could not happen in GENERIC mode as it uses quarantine.
Link: https://lore.kernel.org/20260524031053.381776-1-wsw9603@163.com Signed-off-by: Wang Wensheng <wsw9603@163.com> Reviewed-by: Andrey Konovalov <andreyknvl@gmail.com> Cc: Alexander Potapenko <glider@google.com> Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com> Cc: Dmitry Vyukov <dvyukov@google.com> Cc: Vincenzo Frascino <vincenzo.frascino@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Wed, 20 May 2026 15:03:10 +0000 (08:03 -0700)]
mm/damon/core: trace esz at first setup
DAMON traces effective size quota from the second update, only if a change
has been made by the update. Tracing only changed updates was an
intentional decision to avoid unnecessary same value tracing. Always
skipping the first value is just an unintended mistake.
The mistake makes the tracepoint based investigation incomplete, because
the first effective size quota is never traced. It is not a big issue
when the 'consist' quota tuner is used, because it keeps changing the
quota in the usual setup.
However, when the 'temporal' tuner is used, the quota value is not changed
before the goal achievement status is completely changed. For example, if
the DAMOS scheme is started with an under-achieved goal, the quota is set
to the maximum value, and kept the same value until the goal is achieved.
Because DAMON skips the first value, the user cannot know what effective
quota the current scheme is using. Only after the goal is achieved, the
effective quota is changed to zero, and traced.
Unconditionally trace the initial quota value to fix this problem.
Note that the 'temporal' quota tuner was introduced by commit af738a6a00c1
("mm/damon/core: introduce DAMOS_QUOTA_GOAL_TUNER_TEMPORAL"), which was
added to 7.1-rc1. But even with the 'consist' quota tuner, the tracing is
unintentionally incomplete. Hence this commit marks the introduction of
the trace event as the broken commit.
Link: https://lore.kernel.org/20260520150311.80925-1-sj@kernel.org Fixes: a86d695193bf ("mm/damon: add trace event for effective size quota") Signed-off-by: SeongJae Park <sj@kernel.org> Cc: <stable@vger.kernel.org> # 6.17.x Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Maksym Shcherba [Thu, 21 May 2026 20:20:20 +0000 (23:20 +0300)]
Docs/admin-guide/mm/damon/usage: clarify current_value of quota goals
The sysfs interface for DAMON quota goals includes a `current_value` file.
This file is not updated by the kernel and only serves to receive user
input.
Clarify in the documentation that the kernel does not update
`current_value`, and that reading it only has meaning when `target_metric`
is set to `user_input`.
While at it, fix missing commas in the goal files list.
Maksym Shcherba [Thu, 21 May 2026 20:20:19 +0000 (23:20 +0300)]
mm/damon: fix missing parens in macro arguments
Patch series "mm/damon: fix macro arguments and clarify quota goals doc",
v2.
This patch (of 2):
The DAMON iterator macros do not wrap their pointer arguments with
parentheses. This can cause build failures when the argument is a complex
expression due to operator precedence issues.
Add missing parentheses around the arguments in the following macros
to prevent potential build failures:
- damon_for_each_region()
- damon_for_each_region_from()
- damon_for_each_region_safe()
- damos_for_each_quota_goal()
SeongJae Park [Fri, 22 May 2026 15:40:22 +0000 (08:40 -0700)]
selftests/damon/sysfs.py: stop kdamonds before failing
When an assertion is failed, sysfs.py DAMON selftest immediately exits the
test program leaving the DAMON running behind. Many of the following
tests need to start DAMON on their own. But because DAMON that was
started by sysfs.py is still running, those start attempts fail, and the
tests are failed or skipped. Update sysfs.py to stop DAMON before exiting
the test program due to the assertion failure.
Link: https://lore.kernel.org/20260522154026.80546-12-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:21 +0000 (08:40 -0700)]
mm/damon/tests/core-kunit: add damon_set_regions() test cases
damon_set_regions() is one of the main DAMON kernel API functions that set
up the monitoring target memory region boundaries. Implement unit tests
for verifying its basic functionalities.
Link: https://lore.kernel.org/20260522154026.80546-11-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:20 +0000 (08:40 -0700)]
mm/damon/core: remove damon_verify_nr_regions()
When CONFIG_DAMON_DEBUG_SANITY is enabled, damon_verify_nr_regions() is
called for each damon_nr_regions() invocation. damon_veify_nr_regions()
iterates all regions. damon_nr_regions() is called for each region in
kdamond_reset_aggregated() and damos_apply_scheme(). Hence it imposes
O(n**2) overhead where n is the number of regions.
Though the verification is enabled only under DAMON_DEBUG_SANITY, which is
not for production use cases, it could be too high overhead. Meanwhile,
damon_verify_ctx() is doing the damon_nr_regions() test. Because
damon_verify_ctx() is called for each kdamond_call(), the test coverage
from damon_verify_ctx() could be sufficient. Remove damon_nr_regions()
verification.
Link: https://lore.kernel.org/20260522154026.80546-10-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
kdamond_call() is the place where DAMON API callers are allowed to access
the DAMON context's public internal state including the monitoring
results. Hence it is important to ensure it is called with the expected
DAMON context state. Do the check under DAMON_DEBUG_SANITY.
Link: https://lore.kernel.org/20260522154026.80546-9-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:18 +0000 (08:40 -0700)]
mm/damon/core: hide damon_destroy_region()
damon_destroy_region() is being used by only DAMON core, but exposed to
DAMON API callers. Exposing something that is not really being used by
others will only increase the maintenance cost. Hide it.
Link: https://lore.kernel.org/20260522154026.80546-8-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:17 +0000 (08:40 -0700)]
mm/damon/core: hide damon_insert_region()
damon_insert_region() is being used by only DAMON core, but exposed to
DAMON API callers. Exposing something that is not really being used by
others will only increase the maintenance cost. Hide it.
Link: https://lore.kernel.org/20260522154026.80546-7-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:16 +0000 (08:40 -0700)]
mm/damon/core: hide damon_add_region()
damon_add_region() is being used by only DAMON core, but exposed to DAMON
API callers. Exposing something that is not really being used by others
will only increase the maintenance cost. Hide it.
Link: https://lore.kernel.org/20260522154026.80546-6-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:15 +0000 (08:40 -0700)]
mm/damon/tests/vaddr-kunit: replace damon_add_region() with damon_set_regions()
DAMON virtual address operation set (vaddr) unit tests is using
damon_add_region() for setup of DAMON monitoring target region boundaries
setup. But, damon_set_regions() is designed for exactly the purpose. All
other DAMON API callers use the function for the purpose. Replace
damon_add_region() usage in the unit tests with damon_set_regions(), for
unifying the use case and reducing the maintenance cost.
Link: https://lore.kernel.org/20260522154026.80546-5-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:14 +0000 (08:40 -0700)]
samples/damon/mtier: replace damon_add_region() with damon_set_regions()
mtier DAMON sample module and DAMON virtual address operation set (vaddr)
unit tests are using damon_add_region() for setup of DAMON monitoring
target region boundaries setup. But, damon_set_regions() is designed for
exactly the purpose. All other DAMON API callers use the function for the
purpose. Replace damon_add_region() usage in mtier sample module with
damon_set_regions(), for unifying the use case and reducing the
maintenance cost.
Link: https://lore.kernel.org/20260522154026.80546-4-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:13 +0000 (08:40 -0700)]
mm/damon/core: do not use region out of a loop in damon_set_regions()
damon_set_regions() assumes the DAMON region iterator is referencing the
last region after the region iteration loop is completed. The code is
indeed implemented in the way, but that is not a documented safe behavior.
Hence it is unreliable and difficult to read. Cleanup the code to avoid
the case.
No behavioral change is intended.
Link: https://lore.kernel.org/20260522154026.80546-3-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Fri, 22 May 2026 15:40:12 +0000 (08:40 -0700)]
mm/damon/core: safely handle no region case in damon_set_regions()
Patch series "mm/damon: minor improvements for code readability and tests".
Implement minor improvements on code readability and tests for DAMON.
First seven patches are for DAMON code readability and resulting
maintenance. Patches 1 and 2 make damon_set_regions() safer and easier to
read. Patches 3 and 4 remove fragmented DAMON API use cases. Patches 5-7
hides unused core functions that are unnecessarily exposed to API callers.
The following seven patches are for DAMON tests improvement. Patches 8
and 9 adds and removes DAMON_DEBUG_SANITY verifications to ensure
reasonable test coverage without too high overhead. Patch 10 adds a new
kunit test for damon_set_regions(). Patch 11 makes sysfs.py selftest more
gracefully finishes under test failures. Patches 12-13 adds simple
sysfs.sh test cases for the monitoring intervals goal directory, the
addr_unit file and the pause file.
This patch (of 14):
damon_set_regions() calls damon_first_region() regardless of the number of
DAMON regions in a given DAMON target. damon_first_region() internally
uses list_first_entry(), which clearly documents the list is expected to
be not empty. Due to the internal implementation of the macro,
damon_set_regions() is safe for now. But the internal implementation of
the macro can be changed in future. Refactor the function to explicitly
and safely handle the empty region list case without depending on the
internal implementation.
Rather than providing a hook, simplify things by providing the ability to
override mmap action errors. This allows us to more carefully validate
the value provided and thus ensure only a valid error code is specified,
and simplifies the interface.
This way, we eliminate all hooks but mmap_prepare and allow only mmap
actions to be specified (which core mm controls).
This significantly improves robustness and eliminates any unnecessary code
duplication in driver mmap hooks.
We also update the /dev/mem logic (the only user) to use
mmap_action->error_override instead.
Link: https://lore.kernel.org/55d13f7d016b827c459946d46a56105635be111c.1780397980.git.ljs@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Lorenzo Stoakes [Tue, 2 Jun 2026 11:06:26 +0000 (12:06 +0100)]
mm/vma: remove mmap_action->success_hook
This hook was introduced to work around code that seemed to absolutely
require access to a VMA pointer upon mmap().
However, providing this hook leaves a backdoor to drivers getting access
to the very thing mmap_prepare eliminates - a pointer to the VMA.
Let's solve this contradiction by removing it. The key intended user was
hugetlb, however it seems that the best course now is to avoid allowing
all drivers the ability to work around mmap_prepare, and find a different
solution there.
Link: https://lore.kernel.org/f79434e6d30af6d92999be6b76e197f1847105fa.1780397980.git.ljs@kernel.org Signed-off-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org> Cc: Jann Horn <jannh@google.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
projects / thirdparty / linux.git / log
