x86/mm/cpa: Move flush_tlb_all()

There is an atom errata, where we do a local TLB invalidate right
before we return and then do a global TLB invalidate.

Move the global invalidate up a little bit and avoid the local
invalidate entirely.

This does put the global invalidate under pgd_lock, but that shouldn't
matter.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Dave Hansen <dave.hansen@intel.com>
Cc: Bin Yang <bin.yang@intel.com>
Cc: Mark Gross <mark.gross@intel.com>
Link: https://lkml.kernel.org/r/20180919085947.882287392@infradead.org

diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index a22f6b71a308473e45253dabfa65aaa439b1bd0d..b6a4c638f086cd9fb11a25083305ae09b959d2bb 100644 (file)
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -999,14 +999,24 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address,
        __set_pmd_pte(kpte, address, mk_pte(base, __pgprot(_KERNPG_TABLE)));
 
        /*
-        * Intel Atom errata AAH41 workaround.
+        * Do a global flush tlb after splitting the large page
+        * and before we do the actual change page attribute in the PTE.
         *
-        * The real fix should be in hw or in a microcode update, but
-        * we also probabilistically try to reduce the window of having
-        * a large TLB mixed with 4K TLBs while instruction fetches are
-        * going on.
+        * Without this, we violate the TLB application note, that says:
+        * "The TLBs may contain both ordinary and large-page
+        *  translations for a 4-KByte range of linear addresses. This
+        *  may occur if software modifies the paging structures so that
+        *  the page size used for the address range changes. If the two
+        *  translations differ with respect to page frame or attributes
+        *  (e.g., permissions), processor behavior is undefined and may
+        *  be implementation-specific."
+        *
+        * We do this global tlb flush inside the cpa_lock, so that we
+        * don't allow any other cpu, with stale tlb entries change the
+        * page attribute in parallel, that also falls into the
+        * just split large page entry.
         */
-       __flush_tlb_all();
+       flush_tlb_all();
        spin_unlock(&pgd_lock);
 
        return 0;
@@ -1531,28 +1541,8 @@ repeat:
         * We have to split the large page:
         */
        err = split_large_page(cpa, kpte, address);
-       if (!err) {
-               /*
-                * Do a global flush tlb after splitting the large page
-                * and before we do the actual change page attribute in the PTE.
-                *
-                * With out this, we violate the TLB application note, that says
-                * "The TLBs may contain both ordinary and large-page
-                *  translations for a 4-KByte range of linear addresses. This
-                *  may occur if software modifies the paging structures so that
-                *  the page size used for the address range changes. If the two
-                *  translations differ with respect to page frame or attributes
-                *  (e.g., permissions), processor behavior is undefined and may
-                *  be implementation-specific."
-                *
-                * We do this global tlb flush inside the cpa_lock, so that we
-                * don't allow any other cpu, with stale tlb entries change the
-                * page attribute in parallel, that also falls into the
-                * just split large page entry.
-                */
-               flush_tlb_all();
+       if (!err)
                goto repeat;
-       }
 
        return err;
 }