mm: add vm_ops->mapped hook

Previously, when a driver needed to do something like establish a
reference count, it could do so in the mmap hook in the knowledge that the
mapping would succeed.

With the introduction of f_op->mmap_prepare this is no longer the case, as
it is invoked prior to actually establishing the mapping.

mmap_prepare is not appropriate for this kind of thing as it is called
before any merge might take place, and after which an error might occur
meaning resources could be leaked.

To take this into account, introduce a new vm_ops->mapped callback which
is invoked when the VMA is first mapped (though notably - not when it is
merged - which is correct and mirrors existing mmap/open/close behaviour).

We do better that vm_ops->open() here, as this callback can return an
error, at which point the VMA will be unmapped.

Note that vm_ops->mapped() is invoked after any mmap action is complete
(such as I/O remapping).

We intentionally do not expose the VMA at this point, exposing only the
fields that could be used, and an output parameter in case the operation
needs to update the vma->vm_private_data field.

In order to deal with stacked filesystems which invoke inner filesystem's
mmap() invocations, add __compat_vma_mapped() and invoke it on vfs_mmap()
(via compat_vma_mmap()) to ensure that the mapped callback is handled when
an mmap() caller invokes a nested filesystem's mmap_prepare() callback.

Update the mmap_prepare documentation to describe the mapped hook and make
it clear what its intended use is.

The vm_ops->mapped() call is handled by the mmap complete logic to ensure
the same code paths are handled by both the compatibility and VMA layers.

Additionally, update VMA userland test headers to reflect the change.

Link: https://lkml.kernel.org/r/4c5e98297eb0aae9565c564e1c296a112702f144.1774045440.git.ljs@kernel.org
Signed-off-by: Lorenzo Stoakes (Oracle) <ljs@kernel.org>
Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Alexandre Torgue <alexandre.torgue@foss.st.com>
Cc: Al Viro <viro@zeniv.linux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Bodo Stroesser <bostroesser@gmail.com>
Cc: Christian Brauner <brauner@kernel.org>
Cc: Clemens Ladisch <clemens@ladisch.de>
Cc: David Hildenbrand <david@kernel.org>
Cc: David Howells <dhowells@redhat.com>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Liam Howlett <liam.howlett@oracle.com>
Cc: Long Li <longli@microsoft.com>
Cc: Marc Dionne <marc.dionne@auristor.com>
Cc: "Martin K. Petersen" <martin.petersen@oracle.com>
Cc: Maxime Coquelin <mcoquelin.stm32@gmail.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Miquel Raynal <miquel.raynal@bootlin.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Richard Weinberger <richard@nod.at>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vignesh Raghavendra <vigneshr@ti.com>
Cc: Wei Liu <wei.liu@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/Documentation/filesystems/mmap_prepare.rst b/Documentation/filesystems/mmap_prepare.rst
index ae484d3718614ba631e99814fc84708cd33646e8..f14b35ee11d591653603a52d8c5cab5e7a5a8cee 100644 (file)
--- a/Documentation/filesystems/mmap_prepare.rst
+++ b/Documentation/filesystems/mmap_prepare.rst
@@ -25,6 +25,21 @@ That is - no resources should be allocated nor state updated to reflect that a
 mapping has been established, as the mapping may either be merged, or fail to be
 mapped after the callback is complete.
 
+Mapped callback
+---------------
+
+If resources need to be allocated per-mapping, or state such as a reference
+count needs to be manipulated, this should be done using the ``vm_ops->mapped``
+hook, which itself should be set by the >mmap_prepare hook.
+
+This callback is only invoked if a new mapping has been established and was not
+merged with any other, and is invoked at a point where no error may occur before
+the mapping is established.
+
+You may return an error to the callback itself, which will cause the mapping to
+become unmapped and an error returned to the mmap() caller. This is useful if
+resources need to be allocated, and that allocation might fail.
+
 How To Use
 ==========
 

diff --git a/include/linux/fs.h b/include/linux/fs.h
index a2628a12bd2bc59e59ab3fd367c915ff83c86874..c390f5c667e3eae601ff3758ea65e103968db51d 100644 (file)
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -2059,13 +2059,20 @@ static inline bool can_mmap_file(struct file *file)
 }
 
 int compat_vma_mmap(struct file *file, struct vm_area_struct *vma);
+int __vma_check_mmap_hook(struct vm_area_struct *vma);
 
 static inline int vfs_mmap(struct file *file, struct vm_area_struct *vma)
 {
+       int err;
+
        if (file->f_op->mmap_prepare)
                return compat_vma_mmap(file, vma);
 
-       return file->f_op->mmap(file, vma);
+       err = file->f_op->mmap(file, vma);
+       if (err)
+               return err;
+
+       return __vma_check_mmap_hook(vma);
 }
 
 static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 21a2eef5f8fee5014b5bee0854d94eb6685e344b..81fbcfed44dd4bcb87c783bfe9984181e52ee312 100644 (file)
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -775,6 +775,23 @@ struct vm_operations_struct {
         * Context: User context.  May sleep.  Caller holds mmap_lock.
         */
        void (*close)(struct vm_area_struct *vma);
+       /**
+        * @mapped: Called when the VMA is first mapped in the MM. Not called if
+        * the new VMA is merged with an adjacent VMA.
+        *
+        * The @vm_private_data field is an output field allowing the user to
+        * modify vma->vm_private_data as necessary.
+        *
+        * ONLY valid if set from f_op->mmap_prepare. Will result in an error if
+        * set from f_op->mmap.
+        *
+        * Returns %0 on success, or an error otherwise. On error, the VMA will
+        * be unmapped.
+        *
+        * Context: User context.  May sleep.  Caller holds mmap_lock.
+        */
+       int (*mapped)(unsigned long start, unsigned long end, pgoff_t pgoff,
+                     const struct file *file, void **vm_private_data);
        /* Called any time before splitting to check if it's allowed */
        int (*may_split)(struct vm_area_struct *vma, unsigned long addr);
        int (*mremap)(struct vm_area_struct *vma);

diff --git a/mm/util.c b/mm/util.c
index e272efca8c0e971e407f7f67478f29d935678f0e..98fe67e59ec38c4a7fb82f2981770ec9f4594d87 100644 (file)
--- a/mm/util.c
+++ b/mm/util.c
@@ -1163,33 +1163,7 @@ void flush_dcache_folio(struct folio *folio)
 EXPORT_SYMBOL(flush_dcache_folio);
 #endif
 
-/**
- * compat_vma_mmap() - Apply the file's .mmap_prepare() hook to an
- * existing VMA and execute any requested actions.
- * @file: The file which possesss an f_op->mmap_prepare() hook.
- * @vma: The VMA to apply the .mmap_prepare() hook to.
- *
- * Ordinarily, .mmap_prepare() is invoked directly upon mmap(). However, certain
- * stacked filesystems invoke a nested mmap hook of an underlying file.
- *
- * Until all filesystems are converted to use .mmap_prepare(), we must be
- * conservative and continue to invoke these stacked filesystems using the
- * deprecated .mmap() hook.
- *
- * However we have a problem if the underlying file system possesses an
- * .mmap_prepare() hook, as we are in a different context when we invoke the
- * .mmap() hook, already having a VMA to deal with.
- *
- * compat_vma_mmap() is a compatibility function that takes VMA state,
- * establishes a struct vm_area_desc descriptor, passes to the underlying
- * .mmap_prepare() hook and applies any changes performed by it.
- *
- * Once the conversion of filesystems is complete this function will no longer
- * be required and will be removed.
- *
- * Returns: 0 on success or error.
- */
-int compat_vma_mmap(struct file *file, struct vm_area_struct *vma)
+static int __compat_vma_mmap(struct file *file, struct vm_area_struct *vma)
 {
        struct vm_area_desc desc = {
                .mm = vma->vm_mm,
@@ -1221,8 +1195,49 @@ int compat_vma_mmap(struct file *file, struct vm_area_struct *vma)
        set_vma_from_desc(vma, &desc);
        return mmap_action_complete(vma, action);
 }
+
+/**
+ * compat_vma_mmap() - Apply the file's .mmap_prepare() hook to an
+ * existing VMA and execute any requested actions.
+ * @file: The file which possesss an f_op->mmap_prepare() hook.
+ * @vma: The VMA to apply the .mmap_prepare() hook to.
+ *
+ * Ordinarily, .mmap_prepare() is invoked directly upon mmap(). However, certain
+ * stacked filesystems invoke a nested mmap hook of an underlying file.
+ *
+ * Until all filesystems are converted to use .mmap_prepare(), we must be
+ * conservative and continue to invoke these stacked filesystems using the
+ * deprecated .mmap() hook.
+ *
+ * However we have a problem if the underlying file system possesses an
+ * .mmap_prepare() hook, as we are in a different context when we invoke the
+ * .mmap() hook, already having a VMA to deal with.
+ *
+ * compat_vma_mmap() is a compatibility function that takes VMA state,
+ * establishes a struct vm_area_desc descriptor, passes to the underlying
+ * .mmap_prepare() hook and applies any changes performed by it.
+ *
+ * Once the conversion of filesystems is complete this function will no longer
+ * be required and will be removed.
+ *
+ * Returns: 0 on success or error.
+ */
+int compat_vma_mmap(struct file *file, struct vm_area_struct *vma)
+{
+       return __compat_vma_mmap(file, vma);
+}
 EXPORT_SYMBOL(compat_vma_mmap);
 
+int __vma_check_mmap_hook(struct vm_area_struct *vma)
+{
+       /* vm_ops->mapped is not valid if mmap() is specified. */
+       if (vma->vm_ops && WARN_ON_ONCE(vma->vm_ops->mapped))
+               return -EINVAL;
+
+       return 0;
+}
+EXPORT_SYMBOL(__vma_check_mmap_hook);
+
 static void set_ps_flags(struct page_snapshot *ps, const struct folio *folio,
                         const struct page *page)
 {
@@ -1311,11 +1326,32 @@ again:
        }
 }
 
+static int call_vma_mapped(struct vm_area_struct *vma)
+{
+       const struct vm_operations_struct *vm_ops = vma->vm_ops;
+       void *vm_private_data = vma->vm_private_data;
+       int err;
+
+       if (!vm_ops || !vm_ops->mapped)
+               return 0;
+
+       err = vm_ops->mapped(vma->vm_start, vma->vm_end, vma->vm_pgoff,
+                            vma->vm_file, &vm_private_data);
+       if (err)
+               return err;
+
+       if (vm_private_data != vma->vm_private_data)
+               vma->vm_private_data = vm_private_data;
+       return 0;
+}
+
 static int mmap_action_finish(struct vm_area_struct *vma,
                              struct mmap_action *action, int err)
 {
        size_t len;
 
+       if (!err)
+               err = call_vma_mapped(vma);
        if (!err && action->success_hook)
                err = action->success_hook(vma);
 

diff --git a/mm/vma.c b/mm/vma.c
index e1950ae048e22f7528019505544383c81891ca99..a43f3c5d4b3dd9669de2c9271f5193ce59bd1ba7 100644 (file)
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -2781,7 +2781,6 @@ static unsigned long __mmap_region(struct file *file, unsigned long addr,
 
        if (have_mmap_prepare && allocated_new) {
                error = mmap_action_complete(vma, &desc.action);
-
                if (error)
                        return error;
        }

diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h
index a95a4b07f68bec17129f38fd7150d4df43435b28..1fb7bcae4f318940fcd4be369040186209549427 100644 (file)
--- a/tools/testing/vma/include/dup.h
+++ b/tools/testing/vma/include/dup.h
@@ -643,6 +643,23 @@ struct vm_operations_struct {
         * Context: User context.  May sleep.  Caller holds mmap_lock.
         */
        void (*close)(struct vm_area_struct *vma);
+       /**
+        * @mapped: Called when the VMA is first mapped in the MM. Not called if
+        * the new VMA is merged with an adjacent VMA.
+        *
+        * The @vm_private_data field is an output field allowing the user to
+        * modify vma->vm_private_data as necessary.
+        *
+        * ONLY valid if set from f_op->mmap_prepare. Will result in an error if
+        * set from f_op->mmap.
+        *
+        * Returns %0 on success, or an error otherwise. On error, the VMA will
+        * be unmapped.
+        *
+        * Context: User context.  May sleep.  Caller holds mmap_lock.
+        */
+       int (*mapped)(unsigned long start, unsigned long end, pgoff_t pgoff,
+                     const struct file *file, void **vm_private_data);
        /* Called any time before splitting to check if it's allowed */
        int (*may_split)(struct vm_area_struct *vma, unsigned long addr);
        int (*mremap)(struct vm_area_struct *vma);