execmem: add fallback for failures in vmalloc(VM_ALLOW_HUGE_VMAP)

When execmem populates ROX cache it uses vmalloc(VM_ALLOW_HUGE_VMAP).
Although vmalloc falls back to allocating base pages if high order
allocation fails, it may happen that it still cannot allocate enough
memory.

Right now ROX cache is only used by modules and in majority of cases the
allocations happen at boot time when there's plenty of free memory, but
upcoming enabling ROX cache for ftrace and kprobes would mean that execmem
allocations can happen when the system is under memory pressure and a
failure to allocate large page worth of memory becomes more likely.

Fallback to regular vmalloc() if vmalloc(VM_ALLOW_HUGE_VMAP) fails.

Link: https://lkml.kernel.org/r/20250713071730.4117334-6-rppt@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Daniel Gomez <da.gomez@samsung.com>
Cc: Masami Hiramatsu (Google) <mhiramat@kernel.org>
Cc: Petr Pavlu <petr.pavlu@suse.com>
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/mm/execmem.c b/mm/execmem.c
index c99b299b113c41617c947da007a1a33234086a8b..9abf76a63a791434879911345c979564573a4502 100644 (file)
--- a/mm/execmem.c
+++ b/mm/execmem.c
@@ -291,6 +291,11 @@ static int execmem_cache_populate(struct execmem_range *range, size_t size)
 
        alloc_size = round_up(size, PMD_SIZE);
        p = execmem_vmalloc(range, alloc_size, PAGE_KERNEL, vm_flags);
+       if (!p) {
+               alloc_size = size;
+               p = execmem_vmalloc(range, alloc_size, PAGE_KERNEL, vm_flags);
+       }
+
        if (!p)
                return err;
 
@@ -462,7 +467,7 @@ void *execmem_alloc(enum execmem_type type, size_t size)
        bool use_cache = range->flags & EXECMEM_ROX_CACHE;
        vm_flags_t vm_flags = VM_FLUSH_RESET_PERMS;
        pgprot_t pgprot = range->pgprot;
-       void *p;
+       void *p = NULL;
 
        size = PAGE_ALIGN(size);