mm/page_vma_mapped_walk: use ptep_get_lockless() for lockless access

When not holding the lock, there is a chance that the pte gets modified
under our feet, so we need to use the lockless API to make sure that the
entries remain consistent during the read."

Switch from ptep_get() to ptep_get_lockless() accessor for PTE reads when
no lock is taken.

[osalvador@suse.de: changelog addition]
Link: https://lore.kernel.org/ahhNq0pFKvSKZQbR@localhost.localdomain
Link: https://lore.kernel.org/20260528075507.1821939-1-agordeev@linux.ibm.com
Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
Reviewed-by: Oscar Salvador (SUSE) <osalvador@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Harry Yoo <harry@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Liam Howlett <liam@infradead.org>
Cc: Rik van Riel <riel@surriel.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Yang <richard.weiyang@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c
index a4d52fdb3056d5e9d562a3ca5d4c2208ae341172..2ccbabfb2cc17a6ab9c14f2df0753cbbbfbae199 100644 (file)
--- a/mm/page_vma_mapped.c
+++ b/mm/page_vma_mapped.c
@@ -41,7 +41,7 @@ again:
        if (!pvmw->pte)
                return false;
 
-       ptent = ptep_get(pvmw->pte);
+       ptent = ptep_get_lockless(pvmw->pte);
 
        if (pte_none(ptent)) {
                return false;
@@ -183,6 +183,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
        struct mm_struct *mm = vma->vm_mm;
        unsigned long end;
        spinlock_t *ptl;
+       pte_t pteval;
        pgd_t *pgd;
        p4d_t *p4d;
        pud_t *pud;
@@ -310,7 +311,11 @@ next_pte:
                                goto restart;
                        }
                        pvmw->pte++;
-               } while (pte_none(ptep_get(pvmw->pte)));
+                       if (!pvmw->ptl)
+                               pteval = ptep_get_lockless(pvmw->pte);
+                       else
+                               pteval = ptep_get(pvmw->pte);
+               } while (pte_none(pteval));
 
                if (!pvmw->ptl) {
                        spin_lock(ptl);