queue-4.4/hugetlbfs-fix-races-and-page-leaks-during-migration.patch

   1 From cb6acd01e2e43fd8bad11155752b7699c3d0fb76 Mon Sep 17 00:00:00 2001
   2 From: Mike Kravetz <mike.kravetz@oracle.com>
   3 Date: Thu, 28 Feb 2019 16:22:02 -0800
   4 Subject: hugetlbfs: fix races and page leaks during migration
   5
   6 From: Mike Kravetz <mike.kravetz@oracle.com>
   7
   8 commit cb6acd01e2e43fd8bad11155752b7699c3d0fb76 upstream.
   9
  10 hugetlb pages should only be migrated if they are 'active'.  The
  11 routines set/clear_page_huge_active() modify the active state of hugetlb
  12 pages.
  13
  14 When a new hugetlb page is allocated at fault time, set_page_huge_active
  15 is called before the page is locked.  Therefore, another thread could
  16 race and migrate the page while it is being added to page table by the
  17 fault code.  This race is somewhat hard to trigger, but can be seen by
  18 strategically adding udelay to simulate worst case scheduling behavior.
  19 Depending on 'how' the code races, various BUG()s could be triggered.
  20
  21 To address this issue, simply delay the set_page_huge_active call until
  22 after the page is successfully added to the page table.
  23
  24 Hugetlb pages can also be leaked at migration time if the pages are
  25 associated with a file in an explicitly mounted hugetlbfs filesystem.
  26 For example, consider a two node system with 4GB worth of huge pages
  27 available.  A program mmaps a 2G file in a hugetlbfs filesystem.  It
  28 then migrates the pages associated with the file from one node to
  29 another.  When the program exits, huge page counts are as follows:
  30
  31   node0
  32   1024    free_hugepages
  33   1024    nr_hugepages
  34
  35   node1
  36   0       free_hugepages
  37   1024    nr_hugepages
  38
  39   Filesystem                         Size  Used Avail Use% Mounted on
  40   nodev                              4.0G  2.0G  2.0G  50% /var/opt/hugepool
  41
  42 That is as expected.  2G of huge pages are taken from the free_hugepages
  43 counts, and 2G is the size of the file in the explicitly mounted
  44 filesystem.  If the file is then removed, the counts become:
  45
  46   node0
  47   1024    free_hugepages
  48   1024    nr_hugepages
  49
  50   node1
  51   1024    free_hugepages
  52   1024    nr_hugepages
  53
  54   Filesystem                         Size  Used Avail Use% Mounted on
  55   nodev                              4.0G  2.0G  2.0G  50% /var/opt/hugepool
  56
  57 Note that the filesystem still shows 2G of pages used, while there
  58 actually are no huge pages in use.  The only way to 'fix' the filesystem
  59 accounting is to unmount the filesystem
  60
  61 If a hugetlb page is associated with an explicitly mounted filesystem,
  62 this information in contained in the page_private field.  At migration
  63 time, this information is not preserved.  To fix, simply transfer
  64 page_private from old to new page at migration time if necessary.
  65
  66 There is a related race with removing a huge page from a file and
  67 migration.  When a huge page is removed from the pagecache, the
  68 page_mapping() field is cleared, yet page_private remains set until the
  69 page is actually freed by free_huge_page().  A page could be migrated
  70 while in this state.  However, since page_mapping() is not set the
  71 hugetlbfs specific routine to transfer page_private is not called and we
  72 leak the page count in the filesystem.
  73
  74 To fix that, check for this condition before migrating a huge page.  If
  75 the condition is detected, return EBUSY for the page.
  76
  77 Link: http://lkml.kernel.org/r/74510272-7319-7372-9ea6-ec914734c179@oracle.com
  78 Link: http://lkml.kernel.org/r/20190212221400.3512-1-mike.kravetz@oracle.com
  79 Fixes: bcc54222309c ("mm: hugetlb: introduce page_huge_active")
  80 Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
  81 Reviewed-by: Naoya Horiguchi <n-horiguchi@ah.jp.nec.com>
  82 Cc: Michal Hocko <mhocko@kernel.org>
  83 Cc: Andrea Arcangeli <aarcange@redhat.com>
  84 Cc: "Kirill A . Shutemov" <kirill.shutemov@linux.intel.com>
  85 Cc: Mel Gorman <mgorman@techsingularity.net>
  86 Cc: Davidlohr Bueso <dave@stgolabs.net>
  87 Cc: <stable@vger.kernel.org>
  88 [mike.kravetz@oracle.com: v2]
  89   Link: http://lkml.kernel.org/r/7534d322-d782-8ac6-1c8d-a8dc380eb3ab@oracle.com
  90 [mike.kravetz@oracle.com: update comment and changelog]
  91   Link: http://lkml.kernel.org/r/420bcfd6-158b-38e4-98da-26d0cd85bd01@oracle.com
  92 Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
  93 Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
  94 Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
  95
  96
  97 ---
  98  fs/hugetlbfs/inode.c |   12 ++++++++++++
  99  mm/hugetlb.c         |   14 ++++++++++++--
 100  mm/migrate.c         |   11 +++++++++++
 101  3 files changed, 35 insertions(+), 2 deletions(-)
 102
 103 --- a/fs/hugetlbfs/inode.c
 104 +++ b/fs/hugetlbfs/inode.c
 105 @@ -869,6 +869,18 @@ static int hugetlbfs_migrate_page(struct
 106         rc = migrate_huge_page_move_mapping(mapping, newpage, page);
 107         if (rc != MIGRATEPAGE_SUCCESS)
 108                 return rc;
 109 +
 110 +       /*
 111 +        * page_private is subpool pointer in hugetlb pages.  Transfer to
 112 +        * new page.  PagePrivate is not associated with page_private for
 113 +        * hugetlb pages and can not be set here as only page_huge_active
 114 +        * pages can be migrated.
 115 +        */
 116 +       if (page_private(page)) {
 117 +               set_page_private(newpage, page_private(page));
 118 +               set_page_private(page, 0);
 119 +       }
 120 +
 121         migrate_page_copy(newpage, page);
 122
 123         return MIGRATEPAGE_SUCCESS;
 124 --- a/mm/hugetlb.c
 125 +++ b/mm/hugetlb.c
 126 @@ -3472,7 +3472,6 @@ retry_avoidcopy:
 127         copy_user_huge_page(new_page, old_page, address, vma,
 128                             pages_per_huge_page(h));
 129         __SetPageUptodate(new_page);
 130 -       set_page_huge_active(new_page);
 131
 132         mmun_start = address & huge_page_mask(h);
 133         mmun_end = mmun_start + huge_page_size(h);
 134 @@ -3494,6 +3493,7 @@ retry_avoidcopy:
 135                                 make_huge_pte(vma, new_page, 1));
 136                 page_remove_rmap(old_page);
 137                 hugepage_add_new_anon_rmap(new_page, vma, address);
 138 +               set_page_huge_active(new_page);
 139                 /* Make the old page be freed below */
 140                 new_page = old_page;
 141         }
 142 @@ -3575,6 +3575,7 @@ static int hugetlb_no_page(struct mm_str
 143         struct page *page;
 144         pte_t new_pte;
 145         spinlock_t *ptl;
 146 +       bool new_page = false;
 147
 148         /*
 149          * Currently, we are forced to kill the process in the event the
 150 @@ -3608,7 +3609,7 @@ retry:
 151                 }
 152                 clear_huge_page(page, address, pages_per_huge_page(h));
 153                 __SetPageUptodate(page);
 154 -               set_page_huge_active(page);
 155 +               new_page = true;
 156
 157                 if (vma->vm_flags & VM_MAYSHARE) {
 158                         int err = huge_add_to_page_cache(page, mapping, idx);
 159 @@ -3680,6 +3681,15 @@ retry:
 160         }
 161
 162         spin_unlock(ptl);
 163 +
 164 +       /*
 165 +        * Only make newly allocated pages active.  Existing pages found
 166 +        * in the pagecache could be !page_huge_active() if they have been
 167 +        * isolated for migration.
 168 +        */
 169 +       if (new_page)
 170 +               set_page_huge_active(page);
 171 +
 172         unlock_page(page);
 173  out:
 174         return ret;
 175 --- a/mm/migrate.c
 176 +++ b/mm/migrate.c
 177 @@ -1056,6 +1056,16 @@ static int unmap_and_move_huge_page(new_
 178                 lock_page(hpage);
 179         }
 180
 181 +       /*
 182 +        * Check for pages which are in the process of being freed.  Without
 183 +        * page_mapping() set, hugetlbfs specific move page routine will not
 184 +        * be called and we could leak usage counts for subpools.
 185 +        */
 186 +       if (page_private(hpage) && !page_mapping(hpage)) {
 187 +               rc = -EBUSY;
 188 +               goto out_unlock;
 189 +       }
 190 +
 191         if (PageAnon(hpage))
 192                 anon_vma = page_get_anon_vma(hpage);
 193
 194 @@ -1086,6 +1096,7 @@ put_anon:
 195                 put_new_page = NULL;
 196         }
 197
 198 +out_unlock:
 199         unlock_page(hpage);
 200  out:
 201         if (rc != -EAGAIN)