mm/page_alloc: batch page freeing in decay_pcp_high

[ Upstream commit fc4b909c368f3a7b08c895dd5926476b58e85312 ]

It is possible for pcp->count - pcp->high to exceed pcp->batch by a lot.
When this happens, we should perform batching to ensure that
free_pcppages_bulk isn't called with too many pages to free at once and
starve out other threads that need the pcp or zone lock.

Since we are still only freeing the difference between the initial
pcp->count and pcp->high values, there should be no change to how many
pages are freed.

Link: https://lkml.kernel.org/r/20251014145011.3427205-3-joshua.hahnjy@gmail.com
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Suggested-by: Chris Mason <clm@fb.com>
Suggested-by: Andrew Morton <akpm@linux-foundation.org>
Co-developed-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: "Kirill A. Shutemov" <kirill@shutemov.name>
Cc: Michal Hocko <mhocko@suse.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Stable-dep-of: 038a102535eb ("mm/page_alloc: prevent pcp corruption with SMP=n")
Signed-off-by: Sasha Levin <sashal@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index b79d20e8c4a07616fb70050620a875f79c58014a..9ae204e6dfe80c1f8f87c9e57e296c39f540de80 100644 (file)
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2365,7 +2365,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
  */
 bool decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp)
 {
-       int high_min, to_drain, batch;
+       int high_min, to_drain, to_drain_batched, batch;
        bool todo = false;
 
        high_min = READ_ONCE(pcp->high_min);
@@ -2383,11 +2383,14 @@ bool decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp)
        }
 
        to_drain = pcp->count - pcp->high;
-       if (to_drain > 0) {
+       while (to_drain > 0) {
+               to_drain_batched = min(to_drain, batch);
                spin_lock(&pcp->lock);
-               free_pcppages_bulk(zone, to_drain, pcp, 0);
+               free_pcppages_bulk(zone, to_drain_batched, pcp, 0);
                spin_unlock(&pcp->lock);
                todo = true;
+
+               to_drain -= to_drain_batched;
        }
 
        return todo;