Li Wang [Fri, 1 May 2026 02:20:57 +0000 (10:20 +0800)]
selftests/cgroup: fix hardcoded page size in test_percpu_basic
Patch series "selftests/cgroup: Fix false positive failures in
test_percpu_basic", v2.
This patch series addresses two separate issues that cause false
positive failures in the test_percpu_basic test within the cgroup
kmem selftests.
The first issue stems from a hardcoded assumption about the system
page size, which breaks the test on architectures with larger page
sizes.
The second issue is an overly strict memory check that fails to
account for the slab metadata allocated during cgroup creation.
This patch (of 2):
MAX_VMSTAT_ERROR uses a hardcoded page size of 4096, which assumes 4K
pages. This causes test_percpu_basic to fail on systems where the kernel
is configured with a larger page size, such as aarch64 systems using 16K
or 64K pages, where the maximum permissible discrepancy between
memory.current and percpu charges is proportionally larger.
Replace the hardcoded 4096 with sysconf(_SC_PAGESIZE) to correctly derive
the page size at runtime regardless of the underlying architecture or
kernel configuration.
Link: https://lore.kernel.org/20260501022058.18024-1-li.wang@linux.dev Link: https://lore.kernel.org/20260501022058.18024-2-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Acked-by: Waiman Long <longman@redhat.com> Reviewed-by: Sayali Patil <sayalip@linux.ibm.com> Cc: Christoph Lameter <cl@linux.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Shakeel Butt <shakeelb@google.com> Cc: Tejun Heo <tj@kernel.org> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/filemap: do not count FAULT_FLAG_TRIED retries as mmap hits
A fault that starts synchronous mmap readahead can return VM_FAULT_RETRY
after dropping mmap_lock. The retry may then map the folio brought in by
that same miss.
Do not let this retry decrement mmap_miss. The retry still maps the folio
from the page cache; it just does not count as a useful mmap readahead
hit.
Link: https://lore.kernel.org/tencent_22E6B8849EC1141FE7773C64467E6F1E2C09@qq.com Signed-off-by: fujunjie <fujunjie1@qq.com> Reviewed-by: Jan Kara <jack@suse.cz> Reviewed-by: Vishal Moola <vishal.moola@gmail.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/filemap: count only the faulting address as a mmap hit
Patch series "mm/filemap: tighten mmap_miss hit accounting", v3.
mmap_miss is increased when synchronous mmap readahead is needed, and
decreased when filemap_map_pages() maps folios that are already in the
page cache. The decrease side can over-credit hits in two cases:
- fault-around installs nearby PTEs even though the fault only proves
that the faulting address was accessed;
- after synchronous mmap readahead returns VM_FAULT_RETRY, the retry
can find the folio brought in by the same miss and immediately
cancel that miss.
Current evidence comes from a local KVM/data-disk microbenchmark using
mmap_miss_probe, with an 8 GiB guest, 2 vCPUs, 8192 KiB read_ahead_kb,
cold page cache before each run, 1% of the file accessed, and medians of 3
runs.
mmap_miss_probe mmap()s a prepared file with MADV_NORMAL and then touches
one byte at selected base-page offsets. The access order is random,
sequential, or a fixed page stride. The harness drops caches before each
run and samples /proc/vmstat around that access loop.
The 20 GiB case below is a larger-than-memory file case in an 8 GiB guest.
No separate memory hog was used. The 4 GiB case uses the same 8 GiB
guest but keeps the file fit-in-memory.
Each case used a fresh temporary qcow2 data disk, seen by the guest as
/dev/vda, formatted as ext4 and mounted at /mnt/mmap-matrix.
Each result is "pgpgin GiB / elapsed seconds". "pgpgin GiB" is the delta
of the guest /proc/vmstat pgpgin counter, converted from KiB to GiB; it is
used here as an approximate block input counter, not as resident memory or
exact application IO. "Elapsed seconds" is the wall-clock runtime of the
whole mmap_miss_probe access pass, not per-access latency.
For the 20 GiB larger-than-memory case:
workload before after
random 223.377 GiB/101.293s 1.010 GiB/4.790s
stride1021 204.214 GiB/97.557s 204.208 GiB/108.086s
stride2053 409.584 GiB/193.700s 0.970 GiB/3.685s
stride4099 406.452 GiB/134.241s 0.975 GiB/3.499s
sequential 0.212 GiB/0.050s 0.212 GiB/0.057s
For the 4 GiB fit-in-memory case:
workload before after
random 3.987 GiB/1.960s 0.980 GiB/1.221s
stride1021 4.002 GiB/1.838s 4.002 GiB/1.851s
stride2053 3.991 GiB/1.835s 0.811 GiB/0.985s
stride4099 4.001 GiB/1.836s 0.819 GiB/1.037s
sequential 0.056 GiB/0.013s 0.056 GiB/0.018s
The 20 GiB setup also has an ablation. P1 is only the faulting-address
hit accounting change. P2-only is only the FAULT_FLAG_TRIED retry
filter. P1+P2 is the combined accounting change:
This does not claim to solve every sparse pattern. The stride1021 rows
are intentionally shown as a boundary: with 8192 KiB read_ahead_kb,
file->f_ra.ra_pages is 2048 base pages, and synchronous mmap read-around
uses a 2048-page window centered around the fault, roughly [index - 1024,
index + 1023]. stride1021 is 1021 * 4 KiB = 4084 KiB, so the next access
lands inside the previous read-around window. About every other access
can be a real faulting-address page-cache hit, and the other half can each
read about 8 MiB. For about 52k accesses in the 20 GiB/1% run, half of
them times 8 MiB is about 205 GiB, matching the observed 204 GiB.
This patch (of 2):
filemap_map_pages() reduces file->f_ra.mmap_miss when fault-around maps
folios that are already present in the page cache. That hit accounting is
too generous because fault-around can install PTEs around the faulting
address even though the fault only proves that the faulting address was
accessed.
Move the mmap_miss update back into filemap_map_pages(), drop the
mmap_miss argument from the helper functions, and decrement mmap_miss only
when the helper return value shows that the faulting address was mapped.
Keep the existing workingset-folio behavior unchanged.
mm: use zone lock guard in set_migratetype_isolate()
Use spinlock_irqsave scoped lock guard in set_migratetype_isolate() to
replace the explicit lock/unlock pattern with automatic scope-based
cleanup. The scoped variant is used to keep dump_page() outside the
locked section to avoid a lockdep splat.
Link: https://lore.kernel.org/6883351ad7f74d20875fff30e0e3214a089cea97.1777462630.git.d@ilvokhin.com Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com> Suggested-by: Steven Rostedt <rostedt@goodmis.org> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Zi Yan <ziy@nvidia.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: David Hildenbrand <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: use zone lock guard in unreserve_highatomic_pageblock()
Use spinlock_irqsave zone lock guard in unreserve_highatomic_pageblock()
to replace the explicit lock/unlock pattern with automatic scope-based
cleanup.
Link: https://lore.kernel.org/69db814cd178915cb5615334a29304678f960963.1777462630.git.d@ilvokhin.com Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com> Suggested-by: Steven Rostedt <rostedt@goodmis.org> Acked-by: Michal Hocko <mhocko@suse.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: David Hildenbrand <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: use zone lock guard in unset_migratetype_isolate()
Use spinlock_irqsave zone lock guard in unset_migratetype_isolate() to
replace the explicit lock/unlock and goto pattern with automatic
scope-based cleanup.
Link: https://lore.kernel.org/815c0905ea77828ed32bf56ff0a6d3c6548eb3a2.1777462630.git.d@ilvokhin.com Signed-off-by: Dmitry Ilvokhin <d@ilvokhin.com> Suggested-by: Steven Rostedt <rostedt@goodmis.org> Acked-by: Michal Hocko <mhocko@suse.com> Acked-by: Zi Yan <ziy@nvidia.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: David Hildenbrand <david@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: use zone lock guard in reserve_highatomic_pageblock()
Patch series "mm: use spinlock guards for zone lock", v3.
This series uses spinlock guard for zone lock across several mm functions
to replace explicit lock/unlock patterns with automatic scope-based
cleanup.
This simplifies the control flow by removing 'flags' variables, goto
labels, and redundant unlock calls.
Patches are ordered by decreasing value. The first six patches simplify
the control flow by removing gotos, multiple unlock paths, or 'ret'
variables. The last two are simpler lock/unlock pair conversions that
only remove 'flags' and can be dropped if considered unnecessary churn.
Binary size increase is +39 bytes, with Peter Zijlstra's fix for guards
[1] applied. This is due to the compiler not being able to deduplicate
epilogue and eliminate redundant NULL check. See discussion [2] for more
details. I proposed a patch [3] that fixes this, but until it is merged
we need to assume +39 bytes will stay (though it is compiler dependent).
This patch (of 8):
Use the spinlock_irqsave zone lock guard in reserve_highatomic_pageblock()
to replace the explicit lock/unlock and goto out_unlock pattern with
automatic scope-based cleanup.
SeongJae Park [Wed, 29 Apr 2026 15:03:06 +0000 (08:03 -0700)]
Docs/ABI/damon: mark schemes/<S>/filters/ deprecated
Now the 'filters/' directory is deprecated. Update ABI document to also
announce the fact. Also update the descriptions of the files to be based
on 'core_filter/' directory, to make the old descriptions ready to be
removed when the time arrives.
Link: https://lore.kernel.org/20260429150309.82282-3-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Wed, 29 Apr 2026 15:03:05 +0000 (08:03 -0700)]
Docs/admin-guide/mm/damon/usage: mark scheme filters sysfs dir as deprecated
Patch series "mm/damon/sysfs: document filters/ directory as deprecated".
Commit ab71d2d30121 ("mm/damon/sysfs-schemes: let
damon_sysfs_scheme_set_filters() be used for different named directories")
introduced alternatives of 'filters' directory, namely core_filters/ and
'ops_filters/ directories. Now the alternatives are well stabilized and
ready for all users. All filters/ directory use cases are expected to be
able to be migrated to the alternatives. An LTS kernel having the
alternatives, namely 6.18.y, is also released. Existence of filters/
directory is only confusing.
It would be better not immediately removing the directory, though. There
could be users that need time before migrating to the alternatives. There
might be unexpected use cases that the alternatives cannot support. Doing
the deprecation step by step across multiple years like DAMON debugfs
deprecation would be safer. Start the deprecation changes by announcing
the deprecation on the documents.
Every year, one more action for completely removing the directory will be
followed, like DAMON debugfs deprecation did. Following yearly actions
are currently expected. In 2027, deprecation warning kernel messages will
be printed once, for use of filters/ directory. In 2028, filters/
directory will be renamed to filters_DEPRECATED/. In 2029,
filters_DEPRECATED/ directory will be removed.
This patch (of 2):
The alternatives of 'filters/' directory, namely 'core_filters/' and
'ops_filters/', can fully support all the features 'filters/' directory
can do, and provide better user experience. Having 'filters/' directory
is only confusing to users. Announce it as deprecated on the usage
document.
Link: https://lore.kernel.org/20260429150309.82282-1-sj@kernel.org Link: https://lore.kernel.org/20260429150309.82282-2-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/khugepaged: return -EAGAIN for SCAN_PAGE_HAS_PRIVATE in MADV_COLLAPSE
MADV_COLLAPSE uses errno values to provide actionable feedback to
userspace. Temporary resource constraints are mapped to -EAGAIN so the
caller may retry, while intrinsic failures of the specified range are
mapped to -EINVAL.
collapse_file() returns SCAN_PAGE_HAS_PRIVATE when filemap_release_folio()
fails while isolating file-backed folios for collapse. This currently
falls through the default case in madvise_collapse_errno() and is reported
to userspace as -EINVAL.
However, filemap_release_folio() failure commonly reflects temporary folio
state rather than a permanently uncollapsible range.
For example, ext4 returns false when a folio still has dirty journalled
data, btrfs returns false for dirty or writeback folios before extent
state release, and NFS may return false while reclaiming
filesystem-private folio state.
In such cases, retrying MADV_COLLAPSE after writeback, reclaim or journal
progress may succeed. This matches the existing -EAGAIN handling for
SCAN_PAGE_DIRTY_OR_WRITEBACK and other transient collapse failures more
closely than -EINVAL.
Therefore, map SCAN_PAGE_HAS_PRIVATE to -EAGAIN so userspace receives
retryable feedback for this temporary failure path.
Link: https://lore.kernel.org/20260429140434.439456-1-agarwal.vineet2006@gmail.com Signed-off-by: Vineet Agarwal <agarwal.vineet2006@gmail.com> Reviewed-by: Dev Jain <dev.jain@arm.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
selftests/mm: khugepaged: initialize file contents via mmap
file_setup_area() currently allocates anonymous memory, fills it, and
writes it into the backing file used for collapse testing.
Instead of copying data through write(), resize the file with ftruncate(),
map it directly with MAP_SHARED, and initialize the mapped area in place.
This simplifies the setup path and avoids the need for explicit partial
write handling.
Link: https://lore.kernel.org/20260429115816.98824-1-agarwal.vineet2006@gmail.com Signed-off-by: Vineet Agarwal <agarwal.vineet2006@gmail.com> Reviewed-by: Zi Yan <ziy@nvidia.com> Tested-by: Zi Yan <ziy@nvidia.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Wed, 29 Apr 2026 04:12:25 +0000 (21:12 -0700)]
mm/damon/lru_sort: cover all system rams
DAMON_LRU_SORT allows users to set the physical address range to monitor
and do the work on. When users don't explicitly set the range, the
biggest system ram resource of the system is selected as the monitoring
target address range. The intention was to reduce the overhead from
monitoring non-System RAM areas because monitoring non-System RAM may be
meaningless. However, because of the sampling based access check and
adaptive regions adjustment, the overhead should be negligible. It makes
more sense to just cover all system rams of the system. Do so.
Link: https://lore.kernel.org/20260429041232.90257-4-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Wed, 29 Apr 2026 04:12:24 +0000 (21:12 -0700)]
mm/damon/reclaim: cover all system rams
DAMON_RECLAIM allows users to set the physical address range to monitor
and do the work on. When users don't explicitly set the range, the
biggest System RAM resource of the system is selected as the monitoring
target address range. The intention was to reduce the overhead from
monitoring non-System RAM areas because monitoring of non-System RAM may
be meaningless. However, because of the sampling based access check and
adaptive regions adjustment, the overhead should be negligible. It makes
more sense to just cover all system rams of the system. Do so.
Link: https://lore.kernel.org/20260429041232.90257-3-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm/damon/reclaim,lru_sort: monitor all system rams by
default".
DAMON_RECLAIM and DAMON_LRU_SORT set the biggest 'System RAM' resource of
the system as the default monitoring target address range. The main
intention behind the design is to minimize the overhead coming from
monitoring of non-System RAM areas.
This could result in an odd setup when there are multiple discrete System
RAMs of considerable sizes. For example, there are System RAMs each
having 500 GiB size. In this case, only the first 500 GiB will be set as
the monitoring region by default. This is particularly common on NUMA
systems. Hence the modules allow users to set the monitoring target
address range using the module parameters if the default setup doesn't
work for them. In other words, the current design trades ease of setup
for lower overhead.
However, because DAMON utilizes the sampling based access check and the
adaptive regions adjustment mechanisms, the overhead from the monitoring
of non-System RAM areas should be negligible in most setups. Meanwhile,
the setup complexity is causing real headaches for users who need to run
those modules on various types of systems. That is, the current tradeoff
is not a good deal.
Set the physical address range that can cover all System RAM areas of the
system as the default monitoring regions for DAMON_RECLAIM and
DAMON_LRU_SORT.
Technically speaking, this is changing documented behavior. However, it
makes no sense to believe there is a real use case that really depends on
the old weird default behavior. If the old default behavior was working
for them in the reasonable way, this change will only add a negligible
amount of monitoring overhead. If it didn't work, the users may already
be using manual monitoring regions setup, and they will not be affected by
this change.
Patches Sequence
================
Patch 1 introduces a new core function that will be used for the new
default monitoring target region setup. Patch 2 and 3 update
DAMON_RECLAIM and DAMON_LRU_SORT to use the new function instead of the
old one, respectively. Patch 4 removes the old core function that was
replaced by the new one, as there is no more user of it. Patch 5 updates
DAMON_STAT to use the new one instead of its in-house nearly-duplicate
self implementation of the functionality. Finally patches 6 and 7 update
the DAMON_RECLAIM and DAMON_LRU_SORT user documentation for the new
behaviors, respectively.
This patch (of 7):
damon_set_region_biggest_system_ram_default() sets the monitoring target
region as the caller requested. If the caller didn't specify the region,
it finds the biggest System RAM of the system and sets it as the target
region. When there are more than one considerable size of System RAM
resources in the system, the default target setup makes no sense.
Introduce a variant, namely damon_set_region_system_rams_default(). It
sets a physical address range that covers all System RAM resources as the
default target region.
Link: https://lore.kernel.org/20260429041232.90257-1-sj@kernel.org Link: https://lore.kernel.org/20260429041232.90257-2-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: skip KASAN tagging for page-allocated page tables
Page tables are always accessed via the linear mapping with a match-all
tag, so HW-tag KASAN never checks them. For page-allocated tables (PTEs
and PGDs etc), avoid the tag setup and poisoning overhead by using
__GFP_SKIP_KASAN. SLUB-backed page tables are unchanged for now. (They
aren't widely used and require more SLUB related skip logic. Leave it
later.)
Link: https://lore.kernel.org/20260429102704.680174-4-dev.jain@arm.com Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com> Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Ryan Roberts <ryan.roberts@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Ben Segall <bsegall@google.com> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Kees Cook <kees@kernel.org> Cc: K Prateek Nayak <kprateek.nayak@amd.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
kasan: skip HW tagging for all kernel thread stacks
HW-tag KASAN never checks kernel stacks because stack pointers carry the
match-all tag, so setting/poisoning tags is pure overhead.
- Add __GFP_SKIP_KASAN to THREADINFO_GFP so every stack allocator that
uses it skips tagging (fork path plus arch users)
- Add __GFP_SKIP_KASAN to GFP_VMAP_STACK for the fork-specific vmap
stacks.
- When reusing cached vmap stacks, skip kasan_unpoison_range() if HW tags
are enabled.
Software KASAN is unchanged; this only affects tag-based KASAN.
Link: https://lore.kernel.org/20260429102704.680174-3-dev.jain@arm.com Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com> Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Ben Segall <bsegall@google.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Kees Cook <kees@kernel.org> Cc: K Prateek Nayak <kprateek.nayak@amd.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "kasan: hw_tags: Disable tagging for stack and page-tables",
v4.
Stacks and page tables are always accessed with the match-all tag, so
assigning a new random tag every time at allocation and setting invalid
tag at deallocation time, just adds overhead without improving the
detection.
With __GFP_SKIP_KASAN the page keeps its poison tag and KASAN_TAG_KERNEL
(match-all tag) is stored in the page flags while keeping the poison tag
in the hardware. The benefit of it is that 256 tag setting instruction
per 4 kB page aren't needed at allocation and deallocation time.
Thus match-all pointers still work, while non-match tags (other than
poison tag) still fault.
__GFP_SKIP_KASAN only skips for KASAN_HW_TAGS mode, so coverage is
unchanged.
Benchmark:
The benchmark has two modes. In thread mode, the child process forks
and creates N threads. In pgtable mode, the parent maps and faults a
specified memory size and then forks repeatedly with children exiting
immediately.
Thread benchmark:
2000 iterations, 2000 threads: 2.575 s → 2.229 s (~13.4% faster)
The pgtable samples:
- 2048 MB, 2000 iters 19.08 s → 17.62 s (~7.6% faster)
This patch (of 3):
For allocations that will be accessed only with match-all pointers (e.g.,
kernel stacks), setting tags is wasted work. If the caller already set
__GFP_SKIP_KASAN, skip tag setting of vmalloc pages.
Before this patch, __GFP_SKIP_KASAN wasn't being used with vmalloc APIs.
So it wasn't being checked. Now its being checked and acted upon. Other
KASAN modes are unchanged because __GFP_SKIP_KASAN is ignored for them in
the page allocator, and in vmalloc too we ignore this flag for them.
This is a preparatory patch for optimizing kernel stack allocations.
Link: https://lore.kernel.org/20260429102704.680174-1-dev.jain@arm.com Link: https://lore.kernel.org/20260429102704.680174-2-dev.jain@arm.com Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com> Co-developed-by: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Ryan Roberts <ryan.roberts@arm.com> Co-developed-by: Dev Jain <dev.jain@arm.com> Signed-off-by: Dev Jain <dev.jain@arm.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Cc: Arnd Bergmann <arnd@arndb.de> Cc: Ben Segall <bsegall@google.com> Cc: David Hildenbrand <david@kernel.org> Cc: Dietmar Eggemann <dietmar.eggemann@arm.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: Juri Lelli <juri.lelli@redhat.com> Cc: Kees Cook <kees@kernel.org> Cc: K Prateek Nayak <kprateek.nayak@amd.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Mel Gorman <mgorman@suse.de> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: "Uladzislau Rezki (Sony)" <urezki@gmail.com> Cc: Valentin Schneider <vschneid@redhat.com> Cc: Vincent Guittot <vincent.guittot@linaro.org> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/memcontrol: hoist pstatc_pcpu assignment out of CPU loop
In mem_cgroup_alloc(), the assignment of pstatc_pcpu is invariant with
respect to the for_each_possible_cpu() loop: both the 'parent' pointer and
'parent->vmstats_percpu' remain constant throughout all iterations.
The original code redundantly re-evaluated the 'if (parent)' condition and
reassigned pstatc_pcpu on every CPU iteration, then repeated the same
ternary check 'parent ? pstatc_pcpu : NULL' when storing into
statc->parent_pcpu.
Move the single conditional assignment of pstatc_pcpu to before the loop,
resolving both the loop-invariant placement issue and the duplicated null
check. On systems with a large number of possible CPUs, this eliminates
repeated branch evaluation with no functional change.
No functional change intended.
Link: https://lore.kernel.org/20260429084216.186238-1-hui.zhu@linux.dev Signed-off-by: Hui Zhu <zhuhui@kylinos.cn> Reviewed-by: SeongJae Park <sj@kernel.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Roman Gushchin <roman.gushchin@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Shivank Garg [Tue, 24 Mar 2026 19:07:09 +0000 (19:07 +0000)]
mm/migrate: rename PAGE_ migration flags to FOLIO_
These flags only track folio-specific state during migration and are not
used for movable_ops pages. Rename the enum values and the old_page_state
variable to match.
No functional change.
Link: https://lore.kernel.org/20260324190706.964555-4-shivankg@amd.com Signed-off-by: Shivank Garg <shivankg@amd.com> Suggested-by: David Hildenbrand <david@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Huang Ying <ying.huang@linux.alibaba.com> Cc: Alistair Popple <apopple@nvidia.com> Cc: Byungchul Park <byungchul@sk.com> Cc: Gregory Price <gourry@gourry.net> Cc: Joshua Hahn <joshua.hahnjy@gmail.com> Cc: Matthew Brost <matthew.brost@intel.com> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Rakie Kim <rakie.kim@sk.com> Cc: Shivank Garg <shivankg@amd.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Mon, 27 Apr 2026 15:12:29 +0000 (08:12 -0700)]
selftests/damon/sysfs.py: pause DAMON before dumping status
The sysfs.py test commits DAMON parameters, dump the internal DAMON state,
and show if the parameters are committed as expected using the dumped
state. While the dumping is ongoing, DAMON is alive. It can make
internal changes including addition and removal of regions. It can
therefore make a race that can result in false test results. Pause DAMON
execution during the state dumping to avoid such races.
Link: https://lore.kernel.org/20260427151231.113429-11-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Mon, 27 Apr 2026 15:12:26 +0000 (08:12 -0700)]
selftests/damon/_damon_sysfs: support pause file staging
DAMON test-purpose sysfs interface control Python module, _damon_sysfs, is
not supporting the newly added pause file. Add the support of the file,
for future test and use of the feature.
Link: https://lore.kernel.org/20260427151231.113429-8-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Mon, 27 Apr 2026 15:12:21 +0000 (08:12 -0700)]
mm/damon/sysfs: add pause file under context dir
Add pause DAMON sysfs file under the context directory. It exposes the
damon_ctx->pause API parameter to the users so that they can use the
pause/resume feature.
Link: https://lore.kernel.org/20260427151231.113429-3-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Mon, 27 Apr 2026 15:12:20 +0000 (08:12 -0700)]
mm/damon/core: introduce damon_ctx->paused
Patch series "mm/damon: let DAMON be paused and resumed", v2.
DAMON utilizes a few mechanisms that enhance itself over time. Adaptive
regions adjustment, goal-based DAMOS quota auto-tuning and monitoring
intervals auto-tuning like self-training mechanisms are such examples. It
also adds access frequency stability information (age) to the monitoring
results, which makes it enhanced over time.
Sometimes users have to stop DAMON. In this case, DAMON internal state
that enhanced over the time of the last execution simply goes away.
Restarted DAMON have to train itself and enhance its output from the
scratch. This makes DAMON less useful in such cases. Introducing three
such use cases below.
Investigation of DAMON. It is best to do the investigation online,
especially when it is a production environment. DAMON therefore provides
features for such online investigations, including DAMOS stats, monitoring
result snapshot exposure, and multiple tracepoints. When those are
insufficient, and there are additional clues that could be interfered by
DAMON, users have to temporarily stop DAMON to collect the additional
clues. It is not very useful since many of DAMON internal clues are gone
when DAMON is stopped. The loss of the monitoring results that improved
over time is also problematic, especially in production environments.
Monitoring of workloads that have different user-known phases. For
example, in Android, applications are known to have very different access
patterns and behaviors when they are running on the foreground and the
background. It can therefore be useful to separate monitoring of apps
based on whether they are running on the foreground and on the background.
Having two DAMON threads per application that paused and resumed for the
apps foreground/background switches can be useful for the purpose. But
such pause/resume of the execution is not supported.
Tests of DAMON. A few DAMON selftests are using drgn to dump the internal
DAMON status. The tests show if the dumped status is the same as what the
test code expected. Because DAMON keeps running and modifying its
internal status, there are chances of data races that can cause false test
results. Stopping DAMON can avoid the race. But, since the internal
state of DAMON is dropped, the test coverage will be limited.
Let DAMON execution be paused and resumed without loss of the internal
state, to overhaul the limitations. For this, introduce a new DAMON
context parameter, namely 'pause'. API callers can update it while the
context is running, using the online parameters update functions
(damon_commit_ctx() and damon_call()). Once it is set, kdamond_fn() main
loop will do only limited works excluding the monitoring and DAMOS works,
while sleeping sampling intervals per the work. The limited works include
handling of the online parameters update. Hence users can unset the
'pause' parameter again. Once it is unset, kdamond_fn() main loop will do
all the work again (resumed). Under the paused state, it also does stop
condition checks and handling of it, so that paused DAMON can also be
stopped if needed. Expose the feature to the user space via DAMON sysfs
interface. Also, update existing drgn-based tests to test and use the
feature.
Tests
=====
I confirmed the feature functionality using real time tracing ('perf
trace' or 'trace-cmd stream') of damon:damon_aggregated DAMON tracepoint.
By pausing and resuming the DAMON execution, I was able to see the trace
stops and continued as expected. Note that the pause feature support is
added to DAMON user-space tool (damo) after v3.1.9. Users can use
'--pause_ctx' command line option of damo for that, and I actually used it
for my test. The extended drgn-based selftests are also testing a part of
the functionality.
Patches Sequence
================
Patch 1 introduces the new core API for the pause feature. Patch 2 extend
DAMON sysfs interface for the new parameter. Patches 3-5 update design,
usage and ABI documents for the new sysfs file, respectively. The
following five patches are for tests. Patch 6 implements a new kunit test
for the pause parameter online commitment. Patches 7 and 8 extend DAMON
selftest helpers to support the new feature. Patch 9 extends selftest to
test the commitment of the feature. Finally, patch 10 updates existing
selftest to be safe from the race condition using the pause/resume
feature.
This patch (of 10):
DAMON supports only start and stop of the execution. When it is stopped,
its internal data that it self-trained goes away. It will be useful if
the execution can be paused and resumed with the previous self-trained
data.
Introduce per-context API parameter, 'paused', for the purpose. The
parameter can be set and unset while DAMON is running and paused, using
the online parameters commit helper functions (damon_commit_ctx() and
damon_call()). Once 'paused' is set, the kdamond_fn() main loop does only
limited works with sampling interval sleep during the works. The limited
works include the handling of the online parameters update, so that users
can unset the 'pause' and resume the execution when they want. It also
keep checking DAMON stop conditions and handling of it, so that DAMON can
be stopped while paused if needed.
Link: https://lore.kernel.org/20260427151231.113429-1-sj@kernel.org Link: https://lore.kernel.org/20260427151231.113429-2-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Frederick Mayle [Mon, 27 Apr 2026 03:01:47 +0000 (20:01 -0700)]
mm: limit filemap_fault readahead to VMA boundaries
When a file mapping covers a strict subset of a file, an access to the
mapping can trigger readahead of file pages outside the mapped region.
Readahead is meant to prefetch pages likely to be accessed soon, but these
pages aren't accessible via the same means, so it fair to say we don't
have a good indicator they'll be accessed soon. Take an ELF file for
example: an access to the end of a program's read-only segment isn't a
sign that nearby file contents will be accessed next (they are likely to
be mapped discontiguously, or not at all). The pressure from loading
these pages into the cache can evict more useful pages.
To improve the behavior, make three changes:
* Introduce a new readahead_control field, max_index, as a hard limit on
the readahead. The existing file_ra_state->size can't be used as a
limit, it is more of a hint and can be increased by various
heuristics.
* Set readahead_control->max_index to the end of the VMA in all of the
readahead paths that can be triggered from a fault on a file mapping
(both "sync" and "async" readahead).
* Limit the read-around range start to the VMA's start.
Note that these changes only affect readahead triggered in the context of
a fault, they do not affect readahead triggered by read syscalls. If a
user mixes the two types of accesses, the behavior is expected to be the
following: if a fault causes readahead and places a PG_readahead marker
and then a read(2) syscall hits the PG_readahead marker, the resulting
async readahead *will not* be limited to the VMA end. Conversely, if a
read(2) syscall places a PG_readahead marker and then a fault hits the
marker, the async readahead *will* be limited to the VMA end.
There is an edge case that the above motivation glosses over: A single
file mapping might be backed by multiple VMAs. For example, a whole file
could be mapped RW, then part of the mapping made RO using mprotect. This
patch would hurt performance of a sequential faulted read of such a
mapping, the degree depending on how fragmented the VMAs are. A usage
pattern like that is likely rare and already suffering from sub-optimal
performance because, e.g., the fragmented VMAs limit the fault-around, so
each VMA boundary in a sequential faulted read would cause a minor fault.
Still, this patch would make it worse. See a previous discussion of this
topic at [1].
Tested by mapping and reading a small subset of a large file, then using
the cachestat syscall to verify the number of cached pages didn't exceed
the mapping size.
In practical scenarios, the effect depends on the specific file and usage.
Sometimes there is no effect at all, but, for some ELF files in Android,
we see ~20% fewer pages pulled into the cache.
A comprehensive performance evaluation hasn't been done, but, in addition
to the anecdontal memory savings mentioned above, a benchmark was run with
fio 3.38, showing neutral looking results:
fujunjie [Mon, 4 May 2026 10:39:57 +0000 (10:39 +0000)]
mm/madvise: reject invalid process_madvise() advice for zero-length vectors
process_madvise() used to validate the advice while walking each imported
iovec. If the vector has zero total length, vector_madvise() does not
enter the loop and can return success without checking whether the advice
value is valid.
For a local mm, such as process_madvise(PIDFD_SELF, ...), the remote-only
process_madvise_remote_valid() check is skipped. As a result, an invalid
advice can be reported as success when the vector has zero total length.
This differs from madvise(), which rejects an invalid advice before
returning success for a zero-length range.
Validate the generic madvise behavior at the syscall-facing entry points
before any vector walk. In process_madvise(), do this before the
remote-only advice restriction so unsupported advice is rejected with the
same priority for local and remote mm.
Use an errno-returning helper for address/length validation, and handle
zero-length ranges explicitly at the call sites. Requests with valid
advice and zero total length remain a noop and continue to return 0. Add
a selftest that covers invalid advice with a zero-length iovec and an
empty vector, while also checking that a request with valid advice and
zero length still succeeds.
Link: https://lore.kernel.org/tencent_C3AEB0E769C5F4F9370F9411B69B7F8B2907@qq.com Fixes: 021781b01275 ("mm/madvise: unrestrict process_madvise() for current process") Signed-off-by: fujunjie <fujunjie1@qq.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Christian Brauner <brauner@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
bpf: arena: use page_ref_count() instead of page_mapped() in arena_free_pages()
Pages that BPF arena code maps are allocated through
bpf_map_alloc_pages(), which does not allocate folios but pages.
In the future, pages will not have a mapcount, only folios will.
Converting the code to use folios and rely on folio_mapped() sounds like
the wrong approach.
Should BPF arena code allocate folios and use folio_mapped() here? But
likely we would not want to use folios here longterm, as we don't really
need folio information.
Hard to tell. But in the meantime, we can simply use the page refcount
instead, as a heuristic whether the page might be mapped to user space and
we would want to try zapping it, so we can get rid of page_mapped().
Page allocation will give us a page with a refcount of 1. Any user space
mapping adds a page reference. While there can be references from other
subsystems (e.g., GUP), in the common case for this test here relying on
the page count is good enough.
Link: https://lore.kernel.org/20260427-page_mapped-v1-2-e89c3592c74c@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Andrii Nakryiko <andrii@kernel.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Martin KaFai Lau <martin.lau@linux.dev> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Rich Felker <dalias@libc.org> Cc: Rik van Riel <riel@surriel.com> Cc: Song Liu <song@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Yonghong Song <yonghong.song@linux.dev> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
sh: use folio_mapped() instead of page_mapped() in sh4_flush_cache_page()
Patch series "mm: remove page_mapped()".
While preparing my slides for an LSF/MM talk, I realized that I did not
yet remove page_mapped().
So let's do that. In the BPF arena code it's unclear which memdesc we
would want to allocate in the future: certainly something with a refcount,
but likely none with a mapcount. So let's just rely on the page refcount
instead to decide whether we want to try zapping the page from user page
tables.
This patch (of 3):
We already have the folio in our hands, so let's just use folio_mapped().
Link: https://lore.kernel.org/20260427-page_mapped-v1-0-e89c3592c74c@kernel.org Link: https://lore.kernel.org/20260427-page_mapped-v1-1-e89c3592c74c@kernel.org Signed-off-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Andrii Nakryiko <andrii@kernel.org> Cc: Eduard Zingerman <eddyz87@gmail.com> Cc: Harry Yoo <harry@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Jiri Olsa <jolsa@kernel.org> Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> Cc: Kumar Kartikeya Dwivedi <memxor@gmail.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Martin KaFai Lau <martin.lau@linux.dev> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Rich Felker <dalias@libc.org> Cc: Rik van Riel <riel@surriel.com> Cc: Song Liu <song@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Yonghong Song <yonghong.song@linux.dev> Cc: Yoshinori Sato <ysato@users.sourceforge.jp> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/damon: support MADV_COLLAPSE via DAMOS_COLLAPSE scheme action
This patch set introces a new action: DAMOS_COLLAPSE.
For DAMOS_HUGEPAGE and DAMOS_NOHUGEPAGE to work, khugepaged should be
working, since it relies on hugepage_madvise to add a new slot. This slot
should be picked up by khugepaged and eventually collapse (or not, if we
are using DAMOS_NOHUGEPAGE) the pages. If THP is not enabled, khugepaged
will not be working, and therefore no collapse will happen.
DAMOS_COLLAPSE eventually calls madvise_collapse, which will collapse the
address range synchronously. In cases where there is a large VMA
(databases, for example), DAMOS_COLLAPSE allows us to collapse only the
hot region, and not the entire VMA.
This new action may be required to support autotuning with hugepage
as a goal[1].
=========
Benchmarks:
=========
MySQL
=====
Tests were performed in an ARM physical server with MariaDB 10.5 and
sysbench. Read only benchmark was perform with gaussian row hitting,
which follows a normal distribution.
T n, D h: THP set to never, DAMON action set to hugepage
T m, D h: THP set to madvise, DAMON action set to hugepage
T n, D c: THP set to never, DAMON action set to collapse
Memory consumption. Lower is better.
+------------------+----------+----------+----------+
| | T n, D h | T m, D h | T n, D c |
+------------------+----------+----------+----------+
| Total memory use | 2.13 | 2.20 | 2.20 |
| Huge pages | 0 | 1.3 | 1.27 |
+------------------+----------+----------+----------+
Performance in TPS (Transactions Per Second). Higher is better.
T n, D h: 18225.58
T m, D h 18252.93
T n, D c: 18270.21
Performance counter
I got the number of L1 D/I TLB accesses and the number a D/I TLB
accesses that triggered a page walk. I divided the second by the
first to get the percentage of page walkes per TLB access. The
lower the better.
I used masim with the "demo" configuration, but changing the times
to 100 seconds for the initial phase and 50 seconds for the rest of
the phases.
Memory consumption:
+------------------+----------+----------+----------+
| | T n, D h | T m, D h | T n, D c |
+------------------+----------+----------+----------+
| Total memory use | 2.38 GB | 2.36 GB | 2.37 GB |
| Huge pages | 0 | 190 MB | 188 MB |
+------------------+----------+----------+----------+
Performance:
THP never, DAMOS_HUGEPAGE
initial phase: 40,491 accesses/msec, 100001 msecs run
low phase 0: 39,658 accesses/msec, 50002 msecs run
high phase 0: 41,678 accesses/msec, 50000 msecs run
low phase 1: 39,625 accesses/msec, 50003 msecs run
high phase 1: 41,658 accesses/msec, 50002 msecs run
low phase 2: 39,642 accesses/msec, 50002 msecs run
high phase 2: 41,640 accesses/msec, 50001 msecs run
THP madvise, DAMOS_HUGEPAGE
initial phase: 51,977 accesses/msec, 100000 msecs run
low phase 0: 86,953 accesses/msec, 50000 msecs run
high phase 0: 94,812 accesses/msec, 50000 msecs run
low phase 1: 101,017 accesses/msec, 50000 msecs run
high phase 1: 94,841 accesses/msec, 50000 msecs run
low phase 2: 100,993 accesses/msec, 50000 msecs run
high phase 2: 94,791 accesses/msec, 50001 msecs run
THP never, DAMOS_COLLAPSE
initial phase: 93,678 accesses/msec, 100001 msecs run
low phase 0: 101,475 accesses/msec, 50000 msecs run
high phase 0: 98,589 accesses/msec, 50000 msecs run
low phase 1: 101,531 accesses/msec, 50001 msecs run
high phase 1: 98,506 accesses/msec, 50001 msecs run
low phase 2: 101,458 accesses/msec, 50001 msecs run
high phase 2: 98,555 accesses/msec, 50000 msecs run
- We can see that DAMOS "hugepage" action works only when THP is set
to madvise. "collapse" action works even when THP is set to never.
- Performance for "collapse" action is slightly lower than "hugepage"
action and THP madvise. This is due to the fact that collapases
occur synchronously. With "hugepage" they may occur during page
faults.
- Memory consumption is slighly lower for "collapse" than "hugepage"
with THP madvise. This is due to the khugepage collapses all VMAs,
while "collapse" action only collapses the VMAs in the hot region.
- There is an improvement in TLB utilization when collapse through
"hugepage" or "collapse" actions are triggered. The amount of
TLB misses is lower.
- "collapse" action is performance synchronously, which means that
page collapses happen earlier and more rapidly. This can be
useful or not, depending on the scenario.
- "hugepage" action may trigger a VMA split in some scenarios, since
it needs to change the flag of the VMA to THP enabled. This may
lead to additional overhead.
Collapse action just adds a new option to chose the correct system
balance.
Link: https://lore.kernel.org/20260426231619.107231-5-sj@kernel.org Link: https://lore.kernel.org/damon/20260313000816.79933-1-sj@kernel.org/ Signed-off-by: Asier Gutierrez <gutierrez.asier@huawei-partners.com> Signed-off-by: SeongJae Park <sj@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Cheng-Han Wu <hank20010209@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Liew Rui Yan <aethernet65535@gmail.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <skhan@linuxfoundation.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/damon: add synchronous commit for commit_inputs
Problem
=======
Writing invalid parameters to sysfs followed by 'commit_inputs=Y' fails
silently (no error returned to shell), because the validation happens
asynchronously in the kdamond.
Solution
========
To fix this, the commit_inputs_store() callback now uses damon_call() to
synchronously commit parameters in the kdamond thread's safe context.
This ensures that validation errors are returned immediately to
userspace, following the pattern used by DAMON_SYSFS.
Changes
=======
1. Added commit_inputs_store() and commit_inputs_fn() to commit
synchronously.
2. Removed handle_commit_inputs().
This change is motivated from another discussion [1].
Link: https://lore.kernel.org/20260426231619.107231-4-sj@kernel.org Link: https://lore.kernel.org/20260318153731.97470-1-aethernet65535@gmail.com Signed-off-by: Liew Rui Yan <aethernet65535@gmail.com> Reviewed-by: SeongJae Park <sj@kernel.org> Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Asier Gutierrez <gutierrez.asier@huawei-partners.com> Cc: Cheng-Han Wu <hank20010209@gmail.com> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <skhan@linuxfoundation.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/damon/ops-common: optimize damon_hot_score() using ilog2()
Patch series "mm/damon: repost non-hotfix reviewed patches in damon/next
tree", v2.
The first patch from Liew Rui Yan add a minor performance optimization
using ilog2() instead of inefficient manual implementation of the
functionality.
The second patch from Cheng-Han Wu fixes a minor typo:
s/parametrs/parameters/.
The third patch from Liew Rui Yan make commit_inputs operation of
DAMON_RECLAIM and DAMON_LRU_SORT synchronous to improve the user
experience.
The fourth patch from Asier Gutierrez adds a new DAMOS action,
DAMOS_COLLAPSE for deterministic DAMOS-based access-aware THP system.
This patch (of 4):
The current implementation of damon_hot_score() uses a manual for-loop to
calculate the value of 'age_in_log'. This can be efficiently replaced by
ilog2(), which is semantically more appropriate for calculating the
logarithmic value of age.
In a simulated-kernel-module performance test with 10,000,000 iterations,
this optimization showed a significant reduction in latency (average
latency reduced from ~12ns to ~1ns).
Muchun Song [Tue, 28 Apr 2026 08:18:55 +0000 (16:18 +0800)]
mm/mm_init: fix uninitialized struct pages for ZONE_DEVICE
If DAX memory is hotplugged into an unoccupied subsection of an early
section, section_activate() reuses the unoptimized boot memmap. However,
compound_nr_pages() still assumes that vmemmap optimization is in effect
and initializes only the reduced number of struct pages. As a result, the
remaining tail struct pages are left uninitialized, which can later lead
to unexpected behavior or crashes.
Fix this by treating early sections as unoptimized when calculating how
many struct pages to initialize.
Link: https://lore.kernel.org/20260428081855.1249045-7-songmuchun@bytedance.com Fixes: 6fd3620b3428 ("mm/page_alloc: reuse tail struct pages for compound devmaps") Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Liam R. Howlett <liam@infradead.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Joao Martins <joao.m.martins@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michal Hocko <mhocko@suse.com> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Tue, 28 Apr 2026 08:18:54 +0000 (16:18 +0800)]
mm/mm_init: fix pageblock migratetype for ZONE_DEVICE compound pages
The memmap_init_zone_device() function only initializes the migratetype of
the first pageblock of a compound page. If the compound page size exceeds
pageblock_nr_pages (e.g., 1GB hugepages with 2MB pageblocks), subsequent
pageblocks in the compound page remain uninitialized.
Move the migratetype initialization out of __init_zone_device_page() and
into a separate pageblock_migratetype_init_range() function. This
iterates over the entire PFN range of the memory, ensuring that all
pageblocks are correctly initialized.
Also remove the stale confusing comment about MEMINIT_HOTPLUG above the
migratetype setting since it is an obsolete relic from commit 966cf44f637e
("mm: defer ZONE_DEVICE page initialization to the point where we init
pgmap") and no longer makes sense here.
Link: https://lore.kernel.org/20260428081855.1249045-6-songmuchun@bytedance.com Fixes: c4386bd8ee3a ("mm/memremap: add ZONE_DEVICE support for compound pages") Signed-off-by: Muchun Song <songmuchun@bytedance.com> Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Oscar Salvador <osalvador@suse.de> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Liam R. Howlett <liam@infradead.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Joao Martins <joao.m.martins@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michal Hocko <mhocko@suse.com> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Tue, 28 Apr 2026 08:18:53 +0000 (16:18 +0800)]
mm/sparse-vmemmap: fix DAX vmemmap accounting with optimization
When vmemmap optimization is enabled for DAX, the nr_memmap_pages counter
in /proc/vmstat is incorrect. The current code always accounts for the
full, non-optimized vmemmap size, but vmemmap optimization reduces the
actual number of vmemmap pages by reusing tail pages. This causes the
system to overcount vmemmap usage, leading to inaccurate page statistics
in /proc/vmstat.
Fix this by introducing section_nr_vmemmap_pages(), which returns the
exact vmemmap page count for a given pfn range based on whether
optimization is in effect.
Link: https://lore.kernel.org/20260428081855.1249045-5-songmuchun@bytedance.com Fixes: 15995a352474 ("mm: report per-page metadata information") Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Oscar Salvador <osalvador@suse.de> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Liam R. Howlett <liam@infradead.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Joao Martins <joao.m.martins@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michal Hocko <mhocko@suse.com> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Tue, 28 Apr 2026 08:18:52 +0000 (16:18 +0800)]
mm/sparse-vmemmap: pass @pgmap argument to memory deactivation paths
Currently, the memory hot-remove call chain -- arch_remove_memory(),
__remove_pages(), sparse_remove_section() and section_deactivate() -- does
not carry the struct dev_pagemap pointer. This prevents the lower levels
from knowing whether the section was originally populated with vmemmap
optimizations (e.g., DAX with vmemmap optimization enabled).
Without this information, we cannot call vmemmap_can_optimize() to
determine if the vmemmap pages were optimized. As a result, the vmemmap
page accounting during teardown will mistakenly assume a non-optimized
allocation, leading to incorrect memmap statistics.
To lay the groundwork for fixing the vmemmap page accounting, we need to
pass the @pgmap pointer down to the deactivation location. Plumb the
@pgmap argument through the APIs of arch_remove_memory(), __remove_pages()
and sparse_remove_section(), mirroring the corresponding *_activate()
paths.
Link: https://lore.kernel.org/20260428081855.1249045-4-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Oscar Salvador <osalvador@suse.de> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Liam R. Howlett <liam@infradead.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Joao Martins <joao.m.martins@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michal Hocko <mhocko@suse.com> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Muchun Song [Tue, 28 Apr 2026 08:18:51 +0000 (16:18 +0800)]
mm/memory_hotplug: fix incorrect altmap passing in error path
In create_altmaps_and_memory_blocks(), when arch_add_memory() succeeds
with memmap_on_memory enabled, the vmemmap pages are allocated from
params.altmap. If create_memory_block_devices() subsequently fails, the
error path calls arch_remove_memory() with a NULL altmap instead of
params.altmap.
This is a bug that could lead to memory corruption. Since altmap is NULL,
vmemmap_free() falls back to freeing the vmemmap pages into the system
buddy allocator via free_pages() instead of the altmap.
arch_remove_memory() then immediately destroys the physical linear mapping
for this memory. This injects unowned pages into the buddy allocator,
causing machine checks or memory corruption if the system later attempts
to allocate and use those freed pages.
Fix this by passing params.altmap to arch_remove_memory() in the error
path.
Link: https://lore.kernel.org/20260428081855.1249045-3-songmuchun@bytedance.com Fixes: 6b8f0798b85a ("mm/memory_hotplug: split memmap_on_memory requests across memblocks") Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Liam R. Howlett <liam@infradead.org> Reviewed-by: Georgi Djakov <georgi.djakov@oss.qualcomm.com> Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Joao Martins <joao.m.martins@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Oscar Salvador <osalvador@suse.de> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Patch series "mm: Fix vmemmap optimization accounting and initialization",
v8.
The series fixes several bugs in vmemmap optimization, mainly around
incorrect page accounting and memmap initialization in DAX and memory
hotplug paths. It also fixes pageblock migratetype initialization and
struct page initialization for ZONE_DEVICE compound pages.
Patches 1-4 fix vmemmap accounting issues. Patch 1 fixes an accounting
underflow in the section activation failure path by moving vmemmap page
accounting into the lower-level allocation and freeing helpers. Patch 2
fixes incorrect altmap passing in the memory hotplug error path. Patch 3
passes pgmap through memory deactivation paths so the teardown side can
determine whether vmemmap optimization was in effect. Patch 4 uses that
information to account the optimized DAX vmemmap size correctly.
Patches 5-6 fix initialization issues in mm/mm_init. One makes sure all
pageblocks in ZONE_DEVICE compound pages get their migratetype
initialized. The other fixes a case where DAX memory hotplug reuses an
unoptimized early-section memmap while compound_nr_pages() still assumes
vmemmap optimization, leaving tail struct pages uninitialized.
This patch (of 6):
In section_activate(), if populate_section_memmap() fails, the error
handling path calls section_deactivate() to roll back the state. This
causes a vmemmap accounting imbalance.
Since commit c3576889d87b ("mm: fix accounting of memmap pages"), memmap
pages are accounted for only after populate_section_memmap() succeeds.
However, the failure path unconditionally calls section_deactivate(),
which decreases the vmemmap count. Consequently, a failure in
populate_section_memmap() leads to an accounting underflow, incorrectly
reducing the system's tracked vmemmap usage.
Fix this more thoroughly by moving all accounting calls into the lower
level functions that actually perform the vmemmap allocation and freeing:
- populate_section_memmap() accounts for newly allocated vmemmap pages -
depopulate_section_memmap() unaccounts when vmemmap is freed
This ensures proper accounting in all code paths, including error handling
and early section cases.
Link: https://lore.kernel.org/20260428081855.1249045-1-songmuchun@bytedance.com Link: https://lore.kernel.org/20260428081855.1249045-2-songmuchun@bytedance.com Fixes: c3576889d87b ("mm: fix accounting of memmap pages") Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: Oscar Salvador <osalvador@suse.de> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Liam R. Howlett <liam@infradead.org> Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com> Cc: Joao Martins <joao.m.martins@oracle.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Madhavan Srinivasan <maddy@linux.ibm.com> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Michal Hocko <mhocko@suse.com> Cc: Nicholas Piggin <npiggin@gmail.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Dev Jain [Wed, 15 Apr 2026 04:45:09 +0000 (10:15 +0530)]
selftests/mm: simplify byte pattern checking in mremap_test
The original version of mremap_test (7df666253f26: "kselftests: vm: add
mremap tests") validated remapped contents byte-by-byte and printed a
mismatch index in case the bytes streams didn't match. That was rather
inefficient, especially also if the test passed.
Later, commit 7033c6cc9620 ("selftests/mm: mremap_test: optimize execution
time from minutes to seconds using chunkwise memcmp") used memcmp() on
bigger chunks, to fallback to byte-wise scanning to detect the problematic
index only if it discovered a problem.
However, the implementation is overly complicated (e.g., get_sqrt() is
currently not optimal) and we don't really have to report the exact index:
whoever debugs the failing test can figure that out.
Let's simplify by just comparing both byte streams with memcmp() and not
detecting the exact failed index.
Link: https://lore.kernel.org/20260415044509.579428-1-dev.jain@arm.com Signed-off-by: Dev Jain <dev.jain@arm.com> Reported-by: Sarthak Sharma <sarthak.sharma@arm.com> Tested-by: Sarthak Sharma <sarthak.sharma@arm.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: David Laight <david.laight.linux@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Davidlohr Bueso [Mon, 23 Feb 2026 20:15:16 +0000 (12:15 -0800)]
dax/kmem: account for partial discontiguous resource upon removal
When dev_dax_kmem_probe() partially succeeds (at least one range is
mapped) but a subsequent range fails request_mem_region() or
add_memory_driver_managed(), the probe silently continues, ultimately
returning success, but with the corresponding range resource NULL'ed out.
dev_dax_kmem_remove() iterates over all dax_device ranges regardless of if
the underlying resource exists. When remove_memory() is called later, it
returns 0 because the memory was never added which causes
dev_dax_kmem_remove() to incorrectly assume the (nonexistent) resource can
be removed and attempts cleanup on a NULL pointer.
Fix this by skipping these ranges altogether, noting that these cases are
considered success, such that the cleanup is still reached when all
actually-added ranges are successfully removed.
Link: https://lore.kernel.org/20260223201516.1517657-1-dave@stgolabs.net Fixes: 60e93dc097f7 ("device-dax: add dis-contiguous resource support") Signed-off-by: Davidlohr Bueso <dave@stgolabs.net> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com> Reviewed-by: Alison Schofield <alison.schofield@intel.com> Reviewed-by: Jonathan Cameron <jonathan.cameron@huawei.com> Cc: Dan Williams <dan.j.williams@intel.com> Cc: Dave Jiang <dave.jiang@intel.com> Cc: Vishal Verma <vishal.l.verma@intel.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
In alloc_hugetlb_folio(), a single h_cg pointer is used for both the rsvd
and non-rsvd hugetlb cgroup charges. When map_chg is set,
hugetlb_cgroup_charge_cgroup_rsvd() stores the charged cgroup in h_cg, but
the immediately following hugetlb_cgroup_charge_cgroup() overwrites h_cg
with the non-rsvd cgroup pointer.
As a result, hugetlb_cgroup_commit_charge_rsvd() stores the wrong
(non-rsvd) cgroup pointer into the folio's rsvd slot.
When the folio is later freed, free_huge_folio() unconditionally calls
both hugetlb_cgroup_uncharge_folio() and
hugetlb_cgroup_uncharge_folio_rsvd(). The rsvd uncharge reads back the
wrong cgroup from the folio and decrements a counter that was never
charged for that cgroup, causing a page_counter underflow:
page_counter underflow: -512 nr_pages=512
WARNING: mm/page_counter.c:61 at page_counter_cancel
Fix this by introducing a separate h_cg_rsvd pointer exclusively for the
rsvd charge path, keeping the rsvd and non-rsvd charges fully independent
through their charge, commit, and error uncharge paths.
Link: https://lore.kernel.org/20260328065534.346053-1-kartikey406@gmail.com Fixes: 08cf9faf7558 ("hugetlb_cgroup: support noreserve mappings") Reported-by: syzbot+226c1f947186f8fef796@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=226c1f947186f8fef796 Signed-off-by: Deepanshu Kartikey <kartikey406@gmail.com> Reviewed-by: Muchun Song <muchun.song@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Oscar Salvador <osalvador@suse.de> Cc: Mina Almasry <almasrymina@google.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/lruvec: preemptively free dead folios during lru_add drain
Of all observable lruvec lock contention in our fleet, we find that ~24%
occurs when dead folios are present in lru_add batches at drain time.
This is wasteful in the sense that the folio is added to the LRU just to
be immediately removed via folios_put_refs(), incurring two unnecessary
lock acquisitions.
Eliminate this overhead by preemptively cleaning up dead folios before
they make it into the LRU. Use folio_ref_freeze() to filter folios whose
only remaining refcount is the batch ref. When dead folios are found,
move them off the add batch and onto a temporary batch to be freed.
PG_active may be set on a batched folio as well as PG_unevictable (via
migration path). Since filtered folios bypass the normal lru_add()
cleanup, both flags must be cleared before freeing.
During A/B testing on one of our prod instagram workloads (high-frequency
short-lived requests), the patch intercepted almost all dead folios before
they entered the LRU. Data collected using the mm_lru_insertion
tracepoint shows the effectiveness of the patch:
Per-host LRU add averages at 95% CPU load
(60 hosts each side, 3 x 60s intervals)
dead folios/min total folios/min dead %
unpatched: 1,297,785 19,341,986 6.7097%
patched: 14 19,039,996 0.0001%
Within this workload, we save ~2.6M lock acquisitions per minute per host
as a result.
System-wide memory stats improved on the patched side also at 95% CPU load:
- direct reclaim scanning reduced 7%
- allocation stalls reduced 5.2%
- compaction stalls reduced 12.3%
- page frees reduced 4.9%
No regressions were observed in requests served per second or request tail
latency (p99). Both metrics showed directional improvement at higher CPU
utilization (comparing 85% to 95%).
Note that tests were performed using classic LRU.
Link: https://lore.kernel.org/20260425053417.351146-1-jp.kobryn@linux.dev Signed-off-by: JP Kobryn (Meta) <jp.kobryn@linux.dev> Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Acked-by: Michal Hocko <mhocko@suse.com> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Baoquan He <bhe@redhat.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Rik van Riel <riel@surriel.com> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
David Rientjes [Tue, 31 Mar 2026 01:20:57 +0000 (18:20 -0700)]
mm, page_alloc: reintroduce page allocation stall warning
Previously, we had warnings when a single page allocation took longer than
reasonably expected. This was introduced in commit 63f53dea0c98 ("mm:
warn about allocations which stall for too long").
The warning was subsequently reverted in commit 400e22499dd9 ("mm: don't
warn about allocations which stall for too long") because it was possible
to generate memory pressure that would effectively stall further progress
through printk execution.
Page allocation stalls in excess of 10 seconds are always useful to debug
because they can result in severe userspace unresponsiveness. Adding this
artifact can be used to correlate with userspace going out to lunch and to
understand the state of memory at the time.
There should be a reasonable expectation that this warning will never
trigger given it is very passive, it will only be emitted when a page
allocation takes longer than 10 seconds. If it does trigger, this reveals
an issue that should be fixed: a single page allocation should never loop
for more than 10 seconds without oom killing to make memory available.
Unlike the original implementation, this implementation only reports
stalls once for the system every 10 seconds. Otherwise, many concurrent
reclaimers could spam the kernel log unnecessarily. Stalls are only
reported when calling into direct reclaim.
Link: https://lore.kernel.org/371c86c8-1d47-bd70-b74c-769842718b1f@google.com Signed-off-by: David Rientjes <rientjes@google.com> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev> Acked-by: Michal Hocko <mhocko@suse.com> Cc: Brendan Jackman <jackmanb@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Petr Mladek <pmladek@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Julian Braha [Tue, 31 Mar 2026 07:07:30 +0000 (08:07 +0100)]
mm/thp: dead code cleanup in Kconfig
There is already an 'if TRANSPARENT_HUGEPAGE' condition wrapping several
config options e.g. 'READ_ONLY_THP_FOR_FS', making the 'depends on'
statement for each of these a duplicate dependency (dead code).
I propose leaving the outer 'if TRANSPARENT_HUGEPAGE...endif' and removing
the individual 'depends on TRANSPARENT_HUGEPAGE' statement from each
option.
This dead code was found by kconfirm, a static analysis tool for Kconfig.
Link: https://lore.kernel.org/20260331070730.33915-1-julianbraha@gmail.com Signed-off-by: Julian Braha <julianbraha@gmail.com> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
David Carlier [Thu, 2 Apr 2026 07:26:50 +0000 (08:26 +0100)]
mm/page_io: rename swap_iocb fields for clarity
swap_iocb->pages tracks the number of bvec entries (folios), not base
pages. Rename the array from bvec to bvecs and the counter from pages to
nr_bvecs to accurately reflect their purpose.
Link: https://lore.kernel.org/20260402072650.48811-1-devnexen@gmail.com Signed-off-by: David Carlier <devnexen@gmail.com> Suggested-by: Matthew Wilcox (Oracle) <willy@infradead.org> Suggested-by: David Hildenbrand <david@redhat.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Chris Li <chrisl@kernel.org> Cc: Baoquan He <bhe@redhat.com> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: NeilBrown <neil@brown.name> Cc: Nhat Pham <nphamcs@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm/vmpressure: skip socket pressure for costly order reclaim
When reclaim is triggered by high order allocations on a fragmented
system, vmpressure() can report poor reclaim efficiency even though the
system has plenty of free memory. This is because many pages are scanned,
but few are found to actually reclaim - the pages are actively in use and
don't need to be freed. The resulting scan:reclaim ratio causes
vmpressure() to assert socket pressure, throttling TCP throughput
unnecessarily.
Costly order allocations (above PAGE_ALLOC_COSTLY_ORDER) rely heavily on
compaction to succeed, so poor reclaim efficiency at these orders does not
necessarily indicate memory pressure. The kernel already treats this
order as the boundary where reclaim is no longer expected to succeed and
compaction may take over.
Make vmpressure() order-aware through an additional parameter sourced from
scan_control at existing call sites. Socket pressure is now only asserted
when order <= PAGE_ALLOC_COSTLY_ORDER.
Memcg reclaim is unaffected since try_to_free_mem_cgroup_pages() always
uses order 0, which passes the filter unconditionally. Similarly,
vmpressure_prio() now passes order 0 internally when calling vmpressure(),
ensuring critical pressure from low reclaim priority is not suppressed by
the order filter.
The patch was motivated by a case of impacted net throughput in
production. On one affected host, the memory state at the time showed
~15GB available, zero cgroup pressure, and the following buddyinfo state:
mm: huge_memory: refactor defrag_show() to use defrag_flags[]
Replace the hardcoded if/else chain of test_bit() calls and string
literals in defrag_show() with a loop over defrag_flags[] and
defrag_mode_strings[] arrays introduced in the previous commit.
This makes defrag_show() consistent with defrag_store() and eliminates the
duplicated mode name strings.
Link: https://lore.kernel.org/20260408-thp_defrag-v2-2-bc544c1bde4e@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Tested-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Tested-by: Zi Yan <ziy@nvidia.com> Acked-by: Zi Yan <ziy@nvidia.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Liam Howlett <liam@infradead.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
mm: huge_memory: use sysfs_match_string() in defrag_store()
Patch series "mm: huge_memory: clean up defrag sysfs with shared", v2.
Refactor defrag_store() and defrag_show() to use shared data tables
instead of duplicated if/else chains.
Patch 1 introduces an enum defrag_mode, a defrag_mode_strings[] table, and
a defrag_flags[] mapping array, then rewrites defrag_store() to use
sysfs_match_string() with a loop over defrag_flags[].
Patch 2 refactors defrag_show() to use the same arrays, replacing its
hardcoded if/else chain of test_bit() calls and string literals.
This follows the same pattern applied to anon_enabled_store() in commit 522dfb4ba71f ("mm: huge_memory: refactor anon_enabled_store() with
change_anon_orders()").
This patch (of 2):
Replace the if/else chain of sysfs_streq() calls in defrag_store() with
sysfs_match_string() and a defrag_mode_strings[] table.
Introduce enum defrag_mode and defrag_flags[] array mapping each mode to
its corresponding transparent_hugepage_flag. The store function now loops
over defrag_flags[], setting the bit for the selected mode and clearing
the others. When mode is DEFRAG_NEVER (index 4), no index in the
4-element defrag_flags[] matches, so all flags are cleared.
Note that the enum ordering (always, defer, defer+madvise, madvise, never)
differs from the original if/else chain order in defrag_store() (always,
defer+madvise, defer, madvise, never). This is intentional to match the
display order used by defrag_show().
This is a follow-up cleanup to commit 522dfb4ba71f ("mm: huge_memory:
refactor anon_enabled_store() with change_anon_orders()") which applied
the same sysfs_match_string() pattern to anon_enabled_store().
Link: https://lore.kernel.org/20260408-thp_defrag-v2-0-bc544c1bde4e@debian.org Link: https://lore.kernel.org/20260408-thp_defrag-v2-1-bc544c1bde4e@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Tested-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Lance Yang <lance.yang@linux.dev> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Tested-by: Zi Yan <ziy@nvidia.com> Acked-by: Zi Yan <ziy@nvidia.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Liam Howlett <liam@infradead.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Ye Liu [Thu, 9 Apr 2026 01:43:22 +0000 (09:43 +0800)]
mm/khugepaged: use ALIGN helpers for PMD alignment
PMD alignment in khugepaged is currently implemented using a mix of
rounding helpers and open-coded bitmask operations.
Use ALIGN() and ALIGN_DOWN() consistently for PMD-sized address range
alignment, matching the preferred style for address and size handling.
No functional change intended.
Link: https://lore.kernel.org/20260409014323.2385982-1-ye.liu@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Reviewed-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Barry Song <baohua@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Dev Jain <dev.jain@arm.com> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam Howlett <liam@infradead.org> Cc: Liu Ye <liuye@kylinos.cn> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Ye Liu [Fri, 10 Apr 2026 07:47:39 +0000 (15:47 +0800)]
mm/memory-failure: use bool for forcekill state
'forcekill' is used as a boolean flag to control whether processes should
be forcibly killed. It is only assigned from boolean expressions and
never used in arithmetic or bitmask operations.
Convert it from int to bool.
No functional change intended.
Link: https://lore.kernel.org/20260410074740.2524718-1-ye.liu@linux.dev Signed-off-by: Ye Liu <liuye@kylinos.cn> Reviewed-by: SeongJae Park <sj@kernel.org> Acked-by: Miaohe Lin <linmiaohe@huawei.com> Cc: Liu Ye <liuye@kylinos.cn> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Commit 9bdac9142407 ("sparsemem: Put mem map for one node together.")
introduced a mechanism to pre-allocate a large memory block to hold all
memmaps for a NUMA node upfront.
However, the original commit message did not clearly state the actual
benefits or the necessity of explicitly pre-allocating a single chunk for
all memmap areas of a given node.
One of the concerns about removing this pre-allocation is that the
subsequent per-section memmap allocations could become scattered around,
and might turn too many memory blocks/sections into an "un-offlinable"
state. However, tests show that even without the explicit node-wide
pre-allocation, memblock still allocates memory closely and back-to-back.
When tracing vmemmap_set_pmd allocations, the physical chunks allocated by
memblock are strictly adjacent to each other in a single contiguous
physical range (mapped top-down). Because they are packed tightly
together naturally, they will at most consume or pollute the exact same
number of memory blocks as the explicit pre-allocation did.
Another concern is the boot performance impact of calling memmap_alloc()
multiple times compared to one large node-wide allocation. Tests on a
256GB VM showed that memmap allocation time increased from 199,555 ns to
741,292 ns. Even though it is 3.7x slower, on a 1TB machine, the entire
memory allocation time would only take a few milliseconds. This boot
performance difference is completely negligible.
Since no negative impact on memory offlining behavior or noticeable boot
performance regression was found, this patch proposes removing the
explicit node-wide memmap pre-allocation mechanism to reduce the
maintenance burden.
Link: https://lore.kernel.org/20260410092419.2446420-1-songmuchun@bytedance.com Signed-off-by: Muchun Song <songmuchun@bytedance.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Zhen Ni [Tue, 14 Apr 2026 07:58:13 +0000 (15:58 +0800)]
mm/page_owner: fix %pGp format specifier argument type
The %pGp format specifier expects an argument of type 'unsigned long *',
but page->flags is now of type 'memdesc_flags_t' (a struct containing an
unsigned long member 'f') after the introduction of memdesc_flags_t.
Fix the type mismatch by passing &page->flags.f instead of &page->flags,
which matches the expected type.
Link: https://lore.kernel.org/20260414075813.3425968-1-zhen.ni@easystack.cn Fixes: 53fbef56e07d ("mm: introduce memdesc_flags_t") Signed-off-by: Zhen Ni <zhen.ni@easystack.cn> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Anthony Yznaga [Thu, 16 Apr 2026 03:39:39 +0000 (20:39 -0700)]
selftests/mm: run the MAP_DROPPABLE selftest
The test was not being run by the selftest framework so it was never
noticed that it would fail with an assertion failure on configs without
support for MAP_DROPPABLE. Update the test so that it is skipped instead
when MAP_DROPPABLE is not supported, and add it to the mmap category so
that the test is run by the framework.
Link: https://lore.kernel.org/20260416033939.49981-4-anthony.yznaga@oracle.com Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Jason A. Donenfeld <jason@zx2c4.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Mark Brown <broonie@kernel.org> Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Anthony Yznaga [Thu, 16 Apr 2026 03:39:38 +0000 (20:39 -0700)]
selftests/mm: verify droppable mappings cannot be locked
For configs that support MAP_DROPPABLE verify that a mapping created with
MAP_DROPPABLE cannot be locked via mlock(), and that it will not be locked
if it's created after mlockall(MCL_FUTURE).
Link: https://lore.kernel.org/20260416033939.49981-3-anthony.yznaga@oracle.com Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Jason A. Donenfeld <jason@zx2c4.com> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Mark Brown <broonie@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Pedro Falcato <pfalcato@suse.de> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka (SUSE) <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Anthony Yznaga [Thu, 16 Apr 2026 03:39:37 +0000 (20:39 -0700)]
mm: fix mmap errno value when MAP_DROPPABLE is not supported
Patch series "fix MAP_DROPPABLE not supported errno", v4.
Mark Brown reported seeing a regression in -next on 32 bit arm with the
mlock selftests. Before exiting and marking the tests failed, the
following message was logged after an attempt to create a MAP_DROPPABLE
mapping:
Bail out! mmap error: Unknown error 524
It turns out error 524 is ENOTSUPP which is an error that userspace is not
supposed to see, but it indicates in this instance that MAP_DROPPABLE is
not supported.
The first patch changes the errno returned to EOPNOTSUPP. The second
patch is a second version of a prior patch to introduce selftests to
verify locking behavior with droppable mappings with the additional change
to skip the tests when MAP_DROPPABLE is not supported. The third patch
fixes the MAP_DROPPABLE selftest so that it is run by the framework and
skips if MAP_DROPPABLE is not supported.
This patch (of 3):
On configs where MAP_DROPPABLE is not supported (currently any 32-bit
config except for PPC32), mmap fails with errno set to ENOTSUPP. However,
ENOTSUPP is not a standard error value that userspace knows about. The
acceptable userspace-visible errno to use is EOPNOTSUPP. checkpatch.pl
has a warning to this effect.
Link: https://lore.kernel.org/20260416033939.49981-1-anthony.yznaga@oracle.com Link: https://lore.kernel.org/20260416033939.49981-2-anthony.yznaga@oracle.com Fixes: 9651fcedf7b9 ("mm: add MAP_DROPPABLE for designating always lazily freeable mappings") Signed-off-by: Anthony Yznaga <anthony.yznaga@oracle.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Reported-by: Mark Brown <broonie@kernel.org> Reviewed-by: Pedro Falcato <pfalcato@suse.de> Reviewed-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org> Cc: Jann Horn <jannh@google.com> Cc: Jason A. Donenfeld <jason@zx2c4.com> Cc: Liam Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: <stable@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
- Line 112: "zome_reclaim_mode" -> "zone_reclaim_mode"
- Line 6208: "prioities" -> "priorities"
- Line 7067: "that that high" -> "that the high" (duplicated word)
mm/sparse: remove unnecessary NULL check before allocating mem_section
Commit 850ed20539a4 ("mm: move array mem_section init code out of
memory_present()") moved mem_section allocation logic into
memblocks_present().
Before that move, memory_present() could be called multiple times, so
unlikely() matched the common case, where most calls found mem_section
already allocated.
After that move, memblocks_present() is called exactly once from
sparse_init(). Under CONFIG_SPARSEMEM_EXTREME, mem_section is always NULL
when it is called.
So remove unnecessary NULL check before allocating mem_section. No
functional change.
Link: https://lore.kernel.org/20260419144225.2875654-1-ekffu200098@gmail.com Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed by: Donet Tom <donettom@linux.ibm.com> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Li Wang [Fri, 24 Apr 2026 04:00:59 +0000 (12:00 +0800)]
selftests/cgroup: test_zswap: wait for asynchronous writeback
zswap writeback is asynchronous, but test_zswap.c checks writeback
counters immediately after reclaim/trigger paths. On some platforms (e.g.
ppc64le), this can race with background writeback and cause spurious
failures even when behavior is correct.
Add wait_for_writeback() to poll get_cg_wb_count() with a bounded
timeout, and use it in:
test_zswap_writeback_one() when writeback is expected
test_no_invasive_cgroup_shrink() for the wb_group check
This keeps the original before/after assertion style while making the
tests robust against writeback completion latency.
No test behavior change, selftest stability improvement only.
Link: https://lore.kernel.org/20260424040059.12940-9-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Koutný <mkoutny@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Tejun Heo <tj@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Yosry Ahmed <yosryahmed@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Waiman Long <longman@redhat.com> Cc: Yosry Ahmed <yosry@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Li Wang [Fri, 24 Apr 2026 04:00:58 +0000 (12:00 +0800)]
selftest/cgroup: fix zswap attempt_writeback() on 64K pagesize system
In attempt_writeback(), a memsize of 4M only covers 64 pages on 64K page
size systems. When memory.reclaim is called, the kernel prefers
reclaiming clean file pages (binary, libc, linker, etc.) over swapping
anonymous pages. With only 64 pages of anonymous memory, the reclaim
target can be largely or entirely satisfied by dropping file pages,
resulting in very few or zero anonymous pages being pushed into zswap.
This causes zswap_usage to be extremely small or zero, making
zswap_usage/4 insufficient to create meaningful writeback pressure. The
test then fails because no writeback is triggered.
On 4K page size systems this is not an issue because 4M covers 1024
pages, and file pages are a small fraction of the reclaim target.
Fix this by:
- Always allocating 1024 pages regardless of page size. This ensures
enough anonymous pages to reliably populate zswap and trigger
writeback, while keeping the original 4M allocation on 4K systems.
- Setting zswap.max to zswap_usage/4 instead of zswap_usage/2 to
create stronger writeback pressure, ensuring reclaim reliably
triggers writeback even on large page size systems.
Li Wang [Fri, 24 Apr 2026 04:00:57 +0000 (12:00 +0800)]
selftest/cgroup: fix zswap test_no_invasive_cgroup_shrink on large pagesize system
test_no_invasive_cgroup_shrink sets up two cgroups: wb_group, which is
expected to trigger zswap writeback, and a control group (renamed to
zw_group), which should only have pages sitting in zswap without any
writeback.
There are two problems with the current test:
1) The data patterns are reversed. wb_group uses allocate_bytes(), which
writes only a single byte per page — trivially compressible,
especially by zstd — so compressed pages fit within zswap.max and
writeback is never triggered. Meanwhile, the control group uses
getrandom() to produce hard-to-compress data, but it is the group
that does *not* need writeback.
2) The test uses fixed sizes (10K zswap.max, 10MB allocation) that are
too small on systems with large PAGE_SIZE (e.g. 64K), failing to
build enough memory pressure to trigger writeback reliably.
Fix both issues by:
- Swapping the data patterns: fill wb_group pages with partially
random data (getrandom for page_size/4 bytes) to resist compression
and trigger writeback, and fill zw_group pages with simple repeated
data to stay compressed in zswap.
- Making all size parameters PAGE_SIZE-aware: set allocation size to
PAGE_SIZE * 1024, memory.zswap.max to PAGE_SIZE, and memory.max to
allocation_size / 2 for both cgroups.
- Allocating memory inline instead of via cg_run() so the pages
remain resident throughout the test.
=== Error Log ===
# getconf PAGESIZE
65536
# ./test_zswap
TAP version 13
...
ok 5 test_zswap_writeback_disabled
ok 6 # SKIP test_no_kmem_bypass
not ok 7 test_no_invasive_cgroup_shrink
Link: https://lore.kernel.org/20260424040059.12940-7-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Koutný <mkoutny@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Tejun Heo <tj@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Yosry Ahmed <yosryahmed@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Waiman Long <longman@redhat.com> Cc: Yosry Ahmed <yosry@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Li Wang [Fri, 24 Apr 2026 04:00:56 +0000 (12:00 +0800)]
selftests/cgroup: replace hardcoded page size values in test_zswap
test_zswap uses hardcoded values of 4095 and 4096 throughout as page
stride and page size, which are only correct on systems with a 4K page
size. On architectures with larger pages (e.g., 64K on arm64 or ppc64),
these constants cause memory to be touched at sub-page granularity,
leading to inefficient access patterns and incorrect page count
calculations, which can cause test failures.
Replace all hardcoded 4095 and 4096 values with a global pagesize variable
initialized from sysconf(_SC_PAGESIZE) at startup, and remove the
redundant local sysconf() calls scattered across individual functions. No
functional change on 4K page size systems.
Link: https://lore.kernel.org/20260424040059.12940-6-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Acked-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Koutný <mkoutny@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Tejun Heo <tj@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Waiman Long <longman@redhat.com> Cc: Yosry Ahmed <yosryahmed@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Li Wang [Fri, 24 Apr 2026 04:00:55 +0000 (12:00 +0800)]
selftests/cgroup: rename PAGE_SIZE to BUF_SIZE in cgroup_util
The cgroup utility code defines a local PAGE_SIZE macro hardcoded to 4096,
which is used primarily as a generic buffer size for reading cgroup and
proc files. This naming is misleading because the value has nothing to do
with the actual page size of the system. On architectures with larger
pages (e.g., 64K on arm64 or ppc64), the name suggests a relationship that
does not exist. Additionally, the name can shadow or conflict with
PAGE_SIZE definitions from system headers, leading to confusion or subtle
bugs.
To resolve this, rename the macro to BUF_SIZE to accurately reflect its
purpose as a general I/O buffer size.
Furthermore, test_memcontrol currently relies on this hardcoded 4K value
to stride through memory and trigger page faults. Update this logic to
use the actual system page size dynamically. This micro-optimizes the
memory faulting process by ensuring it iterates correctly and efficiently
based on the underlying architecture's true page size. (This part from
Waiman)
Link: https://lore.kernel.org/20260424040059.12940-5-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Signed-off-by: Waiman Long <longman@redhat.com> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Koutný <mkoutny@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Tejun Heo <tj@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Yosry Ahmed <yosryahmed@google.com> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Yosry Ahmed <yosry@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Li Wang [Fri, 24 Apr 2026 04:00:54 +0000 (12:00 +0800)]
selftests/cgroup: use runtime page size for zswpin check
test_zswapin compares memory.stat:zswpin (counted in pages) against a byte
threshold converted with PAGE_SIZE. In cgroup selftests, PAGE_SIZE is
hardcoded to 4096, which makes the conversion wrong on systems with non-4K
base pages (e.g. 64K).
As a result, the test requires too many pages to pass and fails spuriously
even when zswap is working.
Use sysconf(_SC_PAGESIZE) for the zswpin threshold conversion so the check
matches the actual system page size.
Link: https://lore.kernel.org/20260424040059.12940-4-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Reviewed-by: Yosry Ahmed <yosry@kernel.org> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Koutný <mkoutny@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Tejun Heo <tj@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Waiman Long <longman@redhat.com> Cc: Yosry Ahmed <yosryahmed@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Li Wang [Fri, 24 Apr 2026 04:00:53 +0000 (12:00 +0800)]
selftests/cgroup: avoid OOM in test_swapin_nozswap
test_swapin_nozswap can hit OOM before reaching its assertions on some
setups. The test currently sets memory.max=8M and then allocates/reads
32M with memory.zswap.max=0, which may over-constrain reclaim and kill the
workload process.
This keeps the test pressure model intact (allocate/read beyond memory.max
to force swap-in/out) while making it more robust across different
environments.
The test intent is unchanged: confirm that swapping occurs while zswap remains
unused when memory.zswap.max=0.
=== Error Logs ===
# ./test_zswap
TAP version 13
1..7
ok 1 test_zswap_usage
not ok 2 test_swapin_nozswap
...
Li Wang [Fri, 24 Apr 2026 04:00:52 +0000 (12:00 +0800)]
selftests/cgroup: skip test_zswap if zswap is globally disabled
Patch series "selftests/cgroup: improve zswap tests robustness and support
large page sizes", v7.
This patchset aims to fix various spurious failures and improve the
overall robustness of the cgroup zswap selftests.
The primary motivation is to make the tests compatible with architectures
that use non-4K page sizes (such as 64K on ppc64le and arm64). Currently,
the tests rely heavily on hardcoded 4K page sizes and fixed memory limits.
On 64K page size systems, these hardcoded values lead to sub-page
granularity accesses, incorrect page count calculations, and insufficient
memory pressure to trigger zswap writeback, ultimately causing the tests
to fail.
Additionally, this series addresses OOM kills occurring in
test_swapin_nozswap by dynamically scaling memory limits, and prevents
spurious test failures when zswap is built into the kernel but globally
disabled.
This patch (of 8):
test_zswap currently only checks whether zswap is present by testing
/sys/module/zswap. This misses the runtime global state exposed in
/sys/module/zswap/parameters/enabled.
When zswap is built/loaded but globally disabled, the zswap cgroup
selftests run in an invalid environment and may fail spuriously.
Check the runtime enabled state before running the tests:
- skip if zswap is not configured,
- fail if the enabled knob cannot be read,
- skip if zswap is globally disabled.
Also print a hint in the skip message on how to enable zswap.
Link: https://lore.kernel.org/20260424040059.12940-1-li.wang@linux.dev Link: https://lore.kernel.org/20260424040059.12940-2-li.wang@linux.dev Signed-off-by: Li Wang <li.wang@linux.dev> Acked-by: Yosry Ahmed <yosry@kernel.org> Acked-by: Nhat Pham <nphamcs@gmail.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Koutný <mkoutny@suse.com> Cc: Muchun Song <muchun.song@linux.dev> Cc: Tejun Heo <tj@kernel.org> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Chengming Zhou <chengming.zhou@linux.dev> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Waiman Long <longman@redhat.com> Cc: Yosry Ahmed <yosryahmed@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Tue, 28 Apr 2026 01:33:59 +0000 (18:33 -0700)]
selftests/damon/drgn_dump_damon_status: support failed region quota charge ratio
Extend drgn_dump_damon_status.py to dump DAMON internal state for DAMOS
action failed regions quota charge ratio, to be able to show if the
internal state for the feature is working, with future DAMON selftests.
Link: https://lore.kernel.org/20260428013402.115171-11-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Implement the user-space ABI for the DAMOS action failed region
quota-charge ratio setup. For this, add two new sysfs files under the
DAMON sysfs interface for DAMOS quotas. Names of the files are
fail_charge_num and fail_charge_denom, and work for reading and setting
the numerator and denominator of the failed regions charge ratio.
Link: https://lore.kernel.org/20260428013402.115171-5-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Tue, 28 Apr 2026 01:33:52 +0000 (18:33 -0700)]
mm/damon/core: introduce failed region quota charge ratio
DAMOS quota is charged to all DAMOS action application attempted memory,
regardless of how much of the memory the action was successful and failed.
This makes understanding quota behavior without DAMOS stat but only with
end level metrics (e.g., increased amount of free memory for DAMOS_PAGEOUT
action) difficult. Also, charging action-failed memory same as
action-successful memory is somewhat unfair, as successful action
application will induce more overhead in most cases.
Introduce DAMON core API for setting the charge ratio for such
action-failed memory. It allows API callers to specify the ratio in a
flexible way, by setting the numerator and the denominator.
Link: https://lore.kernel.org/20260428013402.115171-4-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Tue, 28 Apr 2026 01:33:51 +0000 (18:33 -0700)]
mm/damon/core: merge regions after applying DAMOS schemes
damos_apply_scheme() could split the given region if applying the scheme's
action to the entire region can result in violating the quota-set upper
limit. Keeping regions that are created by such split operations is
unnecessary overhead.
The overhead would be negligible in the common case because such split
operations could happen only up to the number of installed schemes per
scheme apply interval. The following commit could make the impact larger,
though. The following commit will allow the action-failed region to be
charged in a different ratio. If both the ratio and the remaining quota
is quite small while the region to apply the scheme is quite large and the
action is nearly always failing, a high number of split operations could
happen.
Remove the unnecessary overhead by merging regions after applying schemes
is done for each region. The merge operation is made only if it will not
lose monitoring information and keep min_nr_regions constraint. In the
worst case, the max_nr_regions could still be violated until the next
per-aggregation interval merge operation is made.
Link: https://lore.kernel.org/20260428013402.115171-3-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Tue, 28 Apr 2026 01:33:50 +0000 (18:33 -0700)]
mm/damon/core: handle <min_region_sz remaining quota as empty
Patch series "mm/damon: introduce DAMOS failed region quota charge ratio".
Let users set different DAMOS quota charge ratios for DAMOS action failed
regions, for deterministic and consistent DAMOS action progress.
Common Reports: Unexpectedly Slow DAMOS
=======================================
One common issue report that we get from DAMON users is that DAMOS action
applying progress speed is sometimes much slower than expected. And one
common root cause is that the DAMOS quota is exceeded by the action
applying failed memory regions.
For example, a group of users tried to run DAMOS-based proactive memory
reclamation (DAMON_RECLAIM) with 100 MiB per second DAMOS quota. They ran
it on a system having no active workload which means all memory of the
system is cold. The expectation was that the system will show 100 MiB per
second reclamation until (nearly) all memory is reclaimed. But what they
found is that the speed is quite inconsistent and sometimes it becomes
very slower than the expectation, sometimes even no reclamation at all for
about tens of seconds. The upper limit of the speed (100 MiB per second)
was being kept as expected, though.
By monitoring the qt_exceeds (number of DAMOS quota exceed events) DAMOS
stat, we found DAMOS quota is always exceeded when the speed is slow. By
monitoring sz_tried and sz_applied (the total amount of DAMOS action tried
memory and succeeded memory) DAMOS stats together, we found the
reclamation attempts nearly always failed when the speed is slow.
DAMOS quota charges DAMOS action tried regions regardless of the
successfulness of the try. Hence in the example reported case, there was
unreclaimable memory spread around the system memory. Sometimes nearly
100 MiB of memory that DAMOS tried to reclaim in the given quota interval
was reclaimable, and therefore showed nearly 100 MiB per second speed.
Sometimes nearly 99 MiB of memory that DAMOS was trying to reclaim in the
given quota interval was unreclaimable, and therefore showing only about 1
MiB per second reclaim speed.
We explained it is an expected behavior of the feature rather than a bug,
as DAMOS quota is there for only the upper-limit of the speed. The users
agreed and later reported a huge win from the adoption of DAMON_RECLAIM on
their products.
It is Not a Bug but a Feature; But...
=====================================
So nothing is broken. DAMOS quota is working as intended, as the upper
limit of the speed. It also provides its behavior observability via DAMOS
stat. In the real world production environment that runs long term active
workloads and matters stability, the speed sometimes being slow is not a
real problem.
But, the non-deterministic behavior is sometimes annoying, especially in
lab environments. Even in a realistic production environment, when there
is a huge amount of DAMOS action unapplicable memory, the speed could be
problematically slow. Let's suppose a virtual machines provider that
setup 99% of the host memory as hugetlb pages that cannot be reclaimed, to
give it to virtual machines. Also, when aim-oriented DAMOS auto-tuning is
applied, this could also make the internal feedback loop confused.
The intention of the current behavior was that trying DAMOS action to
regions would anyway impose some overhead, and therefore somehow be
charged. But in the real world, the overhead for failed action is much
lighter than successful action. Charging those at the same ratio may be
unfair, or at least suboptimum in some environments.
DAMOS Action Failed Region Quota Charge Ratio
=============================================
Let users set the charge ratio for the action-failed memory, for more
optimal and deterministic use of DAMOS. It allows users to specify the
numerator and the denominator of the ratio for flexible setup. For
example, let's suppose the numerator and the denominator are set to 1 and
4,096, respectively. The ratio is 1 / 4,096. A DAMOS scheme action is
applied to 5 GiB memory. For 1 GiB of the memory, the action is
succeeded. For the rest (4 GiB), the action is failed. Then, only 1 GiB
and 1 MiB quota is charged.
The optimal charge ratio will depend on the use case and system/workload.
I'd recommend starting from setting the nominator as 1 and the denominator
as PAGE_SIZE and tune based on the results, because many DAMOS actions are
applied at page level.
Tests
=====
I tested this feature in the steps below.
1. Allocate 50% of system memory and mlock() it using a test program.
2. Fill up the page cache to exhaust nearly all free memory.
3. Start DAMON-based proactive reclamation with 100 MiB/second DAMOS
hard-quota. Auto-tune the DAMOS soft-quota under the hard-quota for
achieving 40% free memory of the system with 'temporal' tuner.
For step 1, I run a simple C program that is written by Gemini. It is
quite straightforward, so I'm not sharing the code here.
For step 3, I use the latest version of DAMON user-space tool (damo) like
below.
sudo damo start --damos_action pageout \
` # Do the pageout only up to 100 MiB per second ` \
--damos_quota_space 100M --damos_quota_interval 1s \
` # Auto-tune the quota below the hard quota aiming` \
` # 40% free memory of the node 0 ` \
` # (entire node of the test system)` \
--damos_quota_goal node_mem_free_bp 40% 0 \
` # use temporal tuner, which is easy to understnd ` \
--damos_quota_goal_tuner temporal
As expected, the progress of the reclamation is not consistent, because
the quota is exceeded for the failed reclamation of the unreclaimable
memory.
I do this again, but with the failed region charge ratio feature. For
this, the above 'damo' command is used, after appending command line
option for setup of the charge ratio like below. Note that the option was
added to 'damo' after v3.1.9.
sudo ./damo start --damos_action pageout \
[...]
` # quota-charge only 1/4096 for pageout-failed regions ` \
--damos_quota_fail_charge_ratio 1 4096
The progress of the reclamation was nearly 100 MiB per second until the
goal was achieved, meeting the expectation.
Patches Sequence
================
The first two patches make preparational changes. Patch 1 updates fully
charged quota check to handle <min_region_sz remaining quota, which will
be able to exist after this series is applied. Patch 2 merges regions
after applying schemes is done as long as it is ok to do, since regions
split operations for quota could happen much more frequently under a
corner case that this series will make available.
Patch 3 implements the feature and exposes it via DAMON core API. Patch 4
implements DAMON sysfs ABI for the feature. Three following patches (5-7)
document the feature and ABI on design, usage, and ABI documents,
respectively. Four patches for testing of the new feature follow. Patch
8 implements a kunit test for the feature. Patches 9 and 10 extend DAMON
selftest helpers for DAMON sysfs control and internal state dumping for
adding a new selftest for the feature. Patch 11 extends existing DAMON
sysfs interface selftest to test the new feature using the extended helper
scripts.
This patch (of 11):
Less than min_region_sz remaining quota effectively means the quota is
fully charged. In other words, no remaining quota. This is because DAMOS
actions are applied in the region granularity, and each region should have
min_region_sz or larger size. However the existing fully charged quota
check, which is also used for setting charge_target_from and
charge_addr_from of the quota, is not aware of the case. For the reason,
charge_target_from and charge_addr_from of the quota will not be updated
in the case. This can result in DAMOS action being applied more
frequently to a specific area of the memory.
The case is unreal because quota charging is also made in the region
granularity. It could be changed in future, though. Actually, the
following commit will make the change, by allowing users to set arbitrary
quota charging ratio for action-failed regions. To be prepared for the
change, update the fully charged quota checks to treat having less than
min_region_sz remaining quota as fully charged.
Link: https://lore.kernel.org/20260428013402.115171-1-sj@kernel.org Link: https://lore.kernel.org/20260428013402.115171-2-sj@kernel.org Signed-off-by: SeongJae Park <sj@kernel.org> Cc: Brendan Higgins <brendan.higgins@linux.dev> Cc: David Hildenbrand <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Background and Motivation
=========================
In heterogeneous memory systems, controlling memory distribution across
NUMA nodes is essential for performance optimization. This patch enables
system-wide page distribution with target-state goals such as "maintain
60% of scheme-eligible memory on DRAM" using PA-mode DAMON schemes.
Rather than using absolute thresholds, this metric tracks the ratio of
memory that matches each scheme's access pattern filters on a target node,
enabling the quota system to automatically adjust migration aggressiveness
to maintain the desired distribution.
What This Metric Measures
=========================
Two-Scheme Setup for Hot Page Distribution
==========================================
For maintaining 60% of hot memory on DRAM (node 0) and 40% on CXL
(node 1):
PULL scheme: migrate_hot to node 0
goal: node_eligible_mem_bp, nid=0, target=6000
addr filter: node 1 address range (only migrate FROM CXL)
"Move hot pages to DRAM if less than 60% of hot data is in DRAM"
PUSH scheme: migrate_hot to node 1
goal: node_eligible_mem_bp, nid=1, target=4000
addr filter: node 0 address range (only migrate FROM DRAM)
"Move hot pages to CXL if less than 40% of hot data is in CXL"
Each scheme independently measures its own eligible memory and adjusts its
quota to achieve its target ratio. The schemes work in concert through
DAMON's unified monitoring context, with the quota autotuner balancing
their relative aggressiveness.
Implementation Details
======================
The implementation adds a new quota goal metric type
DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP to the existing DAMOS quota goal
framework. When this metric is configured for a scheme:
1. During each quota adjustment cycle, damos_get_node_eligible_mem_bp()
is called to calculate the current memory distribution.
2. The function iterates through all regions that match the scheme's
access pattern (via __damos_valid_target()) and calculates:
- Total eligible bytes across all nodes
- Eligible bytes specifically on the target node (goal->nid)
3. For each eligible region, damos_calc_eligible_bytes() walks through
the physical address range, using damon_get_folio() to look up
each folio and determine its NUMA node via folio_nid().
4. Large folios are handled by calculating the exact overlap between
the region boundaries and folio boundaries, ensuring accurate
byte counts even when regions partially span folios.
5. The ratio (node_eligible / total_eligible * 10000) is returned
as basis points, which the quota autotuner uses to adjust the
scheme's effective quota size (esz).
The implementation requires CONFIG_DAMON_PADDR since damon_get_folio()
is only available for physical address space monitoring.
Testing Results
===============
Functionally tested on a two-node heterogeneous memory system with DRAM
(node 0) and CXL memory (node 1). A PUSH+PULL scheme configuration using
migrate_hot actions was used to reach a target hot memory ratio between
the two tiers.
With the TEMPORAL tuner, the system converges quickly to the target
distribution. The tuner drives esz to maximum when under goal and to zero
once the goal is met, forming a simple on/off feedback loop that
stabilizes at the desired ratio.
With the CONSIST tuner, the scheme still converges but more slowly, as it
migrates and then throttles itself based on quota feedback. The time to
reach the goal varies depending on workload intensity.
Note: This metric works with both TEMPORAL and CONSIST goal tuners.
Link: https://lore.kernel.org/20260428030520.701-1-ravis.opensrc@gmail.com Signed-off-by: Ravi Jonnalagadda <ravis.opensrc@gmail.com> Suggested-by: SeongJae Park <sj@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Honggyu Kim <honggyu.kim@sk.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Yunjeong Mun <yunjeong.mun@sk.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
SeongJae Park [Tue, 28 Apr 2026 04:29:40 +0000 (21:29 -0700)]
mm/damon/core: make charge_addr_from aware of end-address exclusivity
DAMON region end address is exclusive one, but charge_addr_from is
assigned assuming the end address is inclusive. As a result, DAMOS action
to next up to min_region_sz memory can be skipped. This is quite
negligible user impact. But, the bug is a bug that can be very simply
fixed. Fix the wrong assignment to respect the exclusiveness of the
address.
mm/memory: update stale locking comments for fault handlers
Update the comments for wp_page_copy(), do_wp_page(), do_swap_page(),
do_anonymous_page(), __do_fault(), do_fault(), handle_pte_fault(),
__handle_mm_fault(), and handle_mm_fault() to concisely clarify that they
can be entered holding either the mmap_lock or the VMA lock, and that the
lock may be released upon returning VM_FAULT_RETRY.
Additionally, make the following corrections:
- In do_anonymous_page(), correct the outdated claim that the function
is entered with the PTE "mapped but not yet locked". Since
handle_pte_fault() unmaps the empty PTE before routing to
do_pte_missing(), the comment now correctly states it is entered
with the PTE unmapped and unlocked.
- In __do_fault(), update the stale reference from __lock_page_retry()
to __folio_lock_or_retry().
Link: https://lore.kernel.org/20260424092217.263648-1-adi.sharma@zohomail.in Signed-off-by: Aditya Sharma <adi.sharma@zohomail.in> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
projects / thirdparty / kernel / linux.git / log
