mm: remove the branch from compound_head()

The compound_head() function is a hot path.  For example, the zap path
calls it for every leaf page table entry.

Rewrite the helper function in a branchless manner to eliminate the risk
of CPU branch misprediction.

Link: https://lkml.kernel.org/r/20260227194302.274384-17-kas@kernel.org
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Albert Ou <aou@eecs.berkeley.edu>
Cc: Alexandre Ghiti <alex@ghiti.fr>
Cc: Baoquan He <bhe@redhat.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Frank van der Linden <fvdl@google.com>
Cc: Harry Yoo <harry.yoo@oracle.com>
Cc: Huacai Chen <chenhuacai@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Lorenzo Stoakes <lorenzo.stoakes@oracle.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Paul Walmsley <paul.walmsley@sifive.com>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Usama Arif <usamaarif642@gmail.com>
Cc: WANG Xuerui <kernel@xen0n.name>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index f361bd6c814c20d099c590e5be2b2306a41d2fcd..7223f6f4e2b4035494e8fb14f2b35fe5c33f8eda 100644 (file)
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -224,25 +224,32 @@ static __always_inline bool compound_info_has_mask(void)
 static __always_inline unsigned long _compound_head(const struct page *page)
 {
        unsigned long info = READ_ONCE(page->compound_info);
+       unsigned long mask;
 
-       /* Bit 0 encodes PageTail() */
-       if (!(info & 1))
-               return (unsigned long)page;
+       if (!compound_info_has_mask()) {
+               /* Bit 0 encodes PageTail() */
+               if (info & 1)
+                       return info - 1;
 
-       /*
-        * If compound_info_has_mask() is false, the rest of compound_info is
-        * the pointer to the head page.
-        */
-       if (!compound_info_has_mask())
-               return info - 1;
+               return (unsigned long)page;
+       }
 
        /*
         * If compound_info_has_mask() is true the rest of the info encodes
         * the mask that converts the address of the tail page to the head page.
         *
         * No need to clear bit 0 in the mask as 'page' always has it clear.
+        *
+        * Let's do it in a branchless manner.
         */
-       return (unsigned long)page & info;
+
+       /* Non-tail: -1UL, Tail: 0 */
+       mask = (info & 1) - 1;
+
+       /* Non-tail: -1UL, Tail: info */
+       mask |= info;
+
+       return (unsigned long)page & mask;
 }
 
 #define compound_head(page)    ((typeof(page))_compound_head(page))