x86/mm: Revert INVLPGB optimization for set_memory code

tl;dr: Revert an INVLPGB optimization that did not properly handle
discontiguous virtual addresses.

Full story:

I got a report from some graphics (i915) folks that bisected a
regression in their test suite to 86e6815b316e ("x86/mm: Change
cpa_flush() to call flush_kernel_range() directly").  There was a bit
of flip-flopping on the exact bisect, but the code here does seem
wrong to me. The i915 folks were calling set_pages_array_wc(), so
using the CPA_PAGES_ARRAY mode.

Basically, the 'struct cpa_data' can wrap up all kinds of page table
changes.  Some of these are virtually contiguous, but some are very
much not which is one reason why there are ->vaddr and ->pages arrays.

86e6815b316e made the mistake of assuming that the virtual addresses
in the cpa_data are always contiguous. It got things right when neither
CPA_ARRAY/CPA_PAGES_ARRAY is used, but theoretically wrong when either
of those is used.

In the i915 case, it probably failed to flush some WB TLB entries and
install WC ones, leaving some data in the caches and not flushing it
out to where the device could see it. That eventually caused graphics
problems.

Revert the INVLPGB optimization. It can be reintroduced later, but it
will need to be a bit careful about the array modes.

Fixes: 86e6815b316ec ("x86/mm: Change cpa_flush() to call flush_kernel_range()")
Reported-by: Cui, Ling <ling.cui@intel.com>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Link: https://patch.msgid.link/20260421151909.6B3281C6@davehans-spike.ostc.intel.com

diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c
index cba907c397186866d106a2b5442c28db52798350..d023a40a1e034519ebcb7abd3290de0bd18da999 100644 (file)
--- a/arch/x86/mm/pat/set_memory.c
+++ b/arch/x86/mm/pat/set_memory.c
@@ -399,6 +399,15 @@ static void cpa_flush_all(unsigned long cache)
        on_each_cpu(__cpa_flush_all, (void *) cache, 1);
 }
 
+static void __cpa_flush_tlb(void *data)
+{
+       struct cpa_data *cpa = data;
+       unsigned int i;
+
+       for (i = 0; i < cpa->numpages; i++)
+               flush_tlb_one_kernel(fix_addr(__cpa_addr(cpa, i)));
+}
+
 static int collapse_large_pages(unsigned long addr, struct list_head *pgtables);
 
 static void cpa_collapse_large_pages(struct cpa_data *cpa)
@@ -435,7 +444,6 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa)
 
 static void cpa_flush(struct cpa_data *cpa, int cache)
 {
-       unsigned long start, end;
        unsigned int i;
 
        BUG_ON(irqs_disabled() && !early_boot_irqs_disabled);
@@ -445,12 +453,10 @@ static void cpa_flush(struct cpa_data *cpa, int cache)
                goto collapse_large_pages;
        }
 
-       start = fix_addr(__cpa_addr(cpa, 0));
-       end =   start + cpa->numpages * PAGE_SIZE;
-       if (cpa->force_flush_all)
-               end = TLB_FLUSH_ALL;
-
-       flush_tlb_kernel_range(start, end);
+       if (cpa->force_flush_all || cpa->numpages > tlb_single_page_flush_ceiling)
+               flush_tlb_all();
+       else
+               on_each_cpu(__cpa_flush_tlb, cpa, 1);
 
        if (!cache)
                goto collapse_large_pages;