BUG/MINOR: debug/pools: properly handle out-of-memory when building with DEBUG_UAF

Commit 158fa75 ("MINOR: pools: implement DEBUG_UAF to detect use after free")
implemented pool use-after-free detection, but the mmap() return value isn't
properly checked, preventing the call to pool_alloc_area() from returning
NULL. So on out-of-memory a mangled pointer is returned, causing a crash on
the pool_alloc() site instead of forcing a GC. It doesn't affect regular
operations however, just complicates complex bug investigations.

This fix should be backported to 1.8 and to 1.7.

diff --git a/include/common/memory.h b/include/common/memory.h
index bf77f95263895ba6d2040c86d301230c104d57d2..a305a8c63fffc5d1c078ba8652306804e8124733 100644 (file)
--- a/include/common/memory.h
+++ b/include/common/memory.h
@@ -303,8 +303,10 @@ static inline void pool_free_area(void *area, size_t __maybe_unused size)
 static inline void *pool_alloc_area(size_t size)
 {
        size_t pad = (4096 - size) & 0xFF0;
+       void *ret;
 
-       return mmap(NULL, (size + 4095) & -4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) + pad;
+       ret = mmap(NULL, (size + 4095) & -4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0);
+       return ret == MAP_FAILED ? NULL : ret + pad;
 }
 
 /* frees an area <area> of size <size> allocated by pool_alloc_area(). The