userfaultfd: introduce mfill_copy_folio_locked() helper

Patch series "mm, kvm: allow uffd support in guest_memfd", v4.

These patches enable support for userfaultfd in guest_memfd.

As the groundwork I refactored userfaultfd handling of PTE-based memory
types (anonymous and shmem) and converted them to use vm_uffd_ops for
allocating a folio or getting an existing folio from the page cache.
shmem also implements callbacks that add a folio to the page cache after
the data passed in UFFDIO_COPY was copied and remove the folio from the
page cache if page table update fails.

In order for guest_memfd to notify userspace about page faults, there are
new VM_FAULT_UFFD_MINOR and VM_FAULT_UFFD_MISSING that a ->fault() handler
can return to inform the page fault handler that it needs to call
handle_userfault() to complete the fault.

Nikita helped to plumb these new goodies into guest_memfd and provided
basic tests to verify that guest_memfd works with userfaultfd.  The
handling of UFFDIO_MISSING in guest_memfd requires ability to remove a
folio from page cache, the best way I could find was exporting
filemap_remove_folio() to KVM.

I deliberately left hugetlb out, at least for the most part.  hugetlb
handles acquisition of VMA and more importantly establishing of parent
page table entry differently than PTE-based memory types.  This is a
different abstraction level than what vm_uffd_ops provides and people
objected to exposing such low level APIs as a part of VMA operations.

Also, to enable uffd in guest_memfd refactoring of hugetlb is not needed
and I prefer to delay it until the dust settles after the changes in this
set.

This patch (of 4):

Split copying of data when locks held from mfill_atomic_pte_copy() into a
helper function mfill_copy_folio_locked().

This makes improves code readability and makes complex
mfill_atomic_pte_copy() function easier to comprehend.

No functional change.

Link: https://lore.kernel.org/20260402041156.1377214-1-rppt@kernel.org
Link: https://lore.kernel.org/20260402041156.1377214-2-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Peter Xu <peterx@redhat.com>
Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Andrei Vagin <avagin@google.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: James Houghton <jthoughton@google.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Harry Yoo <harry.yoo@oracle.com>
Cc: Nikita Kalyazin <kalyazin@amazon.com>
Cc: David Carlier <devnexen@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 89879c3ba344560213e2040c4d90b80ab9149ca6..795bafb2c6cce2cd6227e3f18e731708d66651a7 100644 (file)
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -238,6 +238,40 @@ out:
        return ret;
 }
 
+static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
+{
+       void *kaddr;
+       int ret;
+
+       kaddr = kmap_local_folio(folio, 0);
+       /*
+        * The read mmap_lock is held here.  Despite the
+        * mmap_lock being read recursive a deadlock is still
+        * possible if a writer has taken a lock.  For example:
+        *
+        * process A thread 1 takes read lock on own mmap_lock
+        * process A thread 2 calls mmap, blocks taking write lock
+        * process B thread 1 takes page fault, read lock on own mmap lock
+        * process B thread 2 calls mmap, blocks taking write lock
+        * process A thread 1 blocks taking read lock on process B
+        * process B thread 1 blocks taking read lock on process A
+        *
+        * Disable page faults to prevent potential deadlock
+        * and retry the copy outside the mmap_lock.
+        */
+       pagefault_disable();
+       ret = copy_from_user(kaddr, (const void __user *) src_addr,
+                            PAGE_SIZE);
+       pagefault_enable();
+       kunmap_local(kaddr);
+
+       if (ret)
+               return -EFAULT;
+
+       flush_dcache_folio(folio);
+       return ret;
+}
+
 static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
                                 struct vm_area_struct *dst_vma,
                                 unsigned long dst_addr,
@@ -245,7 +279,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
                                 uffd_flags_t flags,
                                 struct folio **foliop)
 {
-       void *kaddr;
        int ret;
        struct folio *folio;
 
@@ -256,27 +289,7 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
                if (!folio)
                        goto out;
 
-               kaddr = kmap_local_folio(folio, 0);
-               /*
-                * The read mmap_lock is held here.  Despite the
-                * mmap_lock being read recursive a deadlock is still
-                * possible if a writer has taken a lock.  For example:
-                *
-                * process A thread 1 takes read lock on own mmap_lock
-                * process A thread 2 calls mmap, blocks taking write lock
-                * process B thread 1 takes page fault, read lock on own mmap lock
-                * process B thread 2 calls mmap, blocks taking write lock
-                * process A thread 1 blocks taking read lock on process B
-                * process B thread 1 blocks taking read lock on process A
-                *
-                * Disable page faults to prevent potential deadlock
-                * and retry the copy outside the mmap_lock.
-                */
-               pagefault_disable();
-               ret = copy_from_user(kaddr, (const void __user *) src_addr,
-                                    PAGE_SIZE);
-               pagefault_enable();
-               kunmap_local(kaddr);
+               ret = mfill_copy_folio_locked(folio, src_addr);
 
                /* fallback to copy_from_user outside mmap_lock */
                if (unlikely(ret)) {
@@ -285,8 +298,6 @@ static int mfill_atomic_pte_copy(pmd_t *dst_pmd,
                        /* don't free the page */
                        goto out;
                }
-
-               flush_dcache_folio(folio);
        } else {
                folio = *foliop;
                *foliop = NULL;