armv8: mmu: fix and optimise explicitly unmapping regions

As more platforms start ensuring they explicitly unmap reserved-memory
regions a few issues have appeared with how the existing dynamic mapping
code works. Fix these and get a small optimisation as well.

1. Teach pte_type() to actually respect the PTE_TYPE_VALID bit
2. Don't walk the TLB a second time if we call mmu_change_region_attr()
   with PTE_TYPE_FAULT (since it would just be a slow nop)
3. Fix how set_one_region() decides to split blocks.

Today set_one_region() will always split blocks until it reaches the
smallest granule size (4k) and then update all of these pages. This
appears to be due to a big in how is_aligned() is implemented, since
it only evaluates to true if addr and size are both multiples of the
current granule size, so a mapping aligned to 2M which is 4M in size
will cleanly result in 2 blocks being set, but a mapping aligned to
2M which is 4M + 8k in size will result in blocks being split and 1026
individual pages being set.

While for the address it is correct to enforce that it is aligned to
the current granule size, we only need to check if the region size is
greater than the current granule size. This allows us to simplify our
second example above to only 4 entries being updated (assuming no blocks
have to be split) since we only need to update 2 blocks to map the first
4M, drastically improving the best-case performance.

In the case where the address is 4k aligned rather than 2M aligned we
will still be restricted to mapping 4k pages until we reach 2M alignment
where we could then map a larger 2M granule which previously would never
happen.

Signed-off-by: Casey Connolly <casey.connolly@linaro.org>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>

diff --git a/arch/arm/cpu/armv8/cache_v8.c b/arch/arm/cpu/armv8/cache_v8.c
index 01f72c3d0ec6405330652bab333b49fbdc2d743a..6c85022556ad441ee54fc1783ab3be662dbad3ad 100644 (file)
--- a/arch/arm/cpu/armv8/cache_v8.c
+++ b/arch/arm/cpu/armv8/cache_v8.c
@@ -163,7 +163,7 @@ u64 get_tcr(u64 *pips, u64 *pva_bits)
 
 static int pte_type(u64 *pte)
 {
-       return *pte & PTE_TYPE_MASK;
+       return *pte & PTE_TYPE_VALID ? *pte & PTE_TYPE_MASK : PTE_TYPE_FAULT;
 }
 
 /* Returns the LSB number for a PTE on level <level> */
@@ -991,9 +991,10 @@ u64 *__weak arch_get_page_table(void) {
        return NULL;
 }
 
+/* Checks if the current PTE is an aligned subset of the region */
 static bool is_aligned(u64 addr, u64 size, u64 align)
 {
-       return !(addr & (align - 1)) && !(size & (align - 1));
+       return !(addr & (align - 1)) && size >= align;
 }
 
 /* Use flag to indicate if attrs has more than d-cache attributes */
@@ -1003,9 +1004,14 @@ static u64 set_one_region(u64 start, u64 size, u64 attrs, bool flag, int level)
        u64 levelsize = 1ULL << levelshift;
        u64 *pte = find_pte(start, level);
 
-       /* Can we can just modify the current level block PTE? */
+       /* Can we can just modify the current level block/page? */
        if (is_aligned(start, size, levelsize)) {
-               if (flag) {
+               if (attrs == PTE_TYPE_FAULT) {
+                       if (pte_type(pte) == PTE_TYPE_TABLE && level < 3)
+                               *pte = 0;
+                       else
+                               *pte &= ~(PTE_TYPE_MASK);
+               } else if (flag) {
                        *pte &= ~PMD_ATTRMASK;
                        *pte |= attrs & PMD_ATTRMASK;
                } else {
@@ -1107,6 +1113,10 @@ void mmu_change_region_attr(phys_addr_t addr, size_t size, u64 attrs)
                           gd->arch.tlb_addr + gd->arch.tlb_size);
        __asm_invalidate_tlb_all();
 
+       /* If we were unmapping a region then we have nothing to make and can return. */
+       if (attrs == PTE_TYPE_FAULT)
+               return;
+
        mmu_change_region_attr_nobreak(addr, size, attrs);
 }