harden REALLOC_ARRAY and xcalloc against size_t overflow

REALLOC_ARRAY inherently involves a multiplication which can
overflow size_t, resulting in a much smaller buffer than we
think we've allocated. We can easily harden it by using
st_mult() to check for overflow.  Likewise, we can add
ALLOC_ARRAY to do the same thing for xmalloc calls.

xcalloc() should already be fine, because it takes the two
factors separately, assuming the system calloc actually
checks for overflow. However, before we even hit the system
calloc(), we do our memory_limit_check, which involves a
multiplication. Let's check for overflow ourselves so that
this limit cannot be bypassed.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>

diff --git a/git-compat-util.h b/git-compat-util.h
index 0c6503348189e0396bd853a1f615aaa8e707f48e..81f2347b27ba39d8129679038602587807fa37f9 100644 (file)
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -779,7 +779,8 @@ extern int odb_pack_keep(char *name, size_t namesz, const unsigned char *sha1);
 extern char *xgetcwd(void);
 extern FILE *fopen_for_writing(const char *path);
 
-#define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), (alloc) * sizeof(*(x)))
+#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))
+#define REALLOC_ARRAY(x, alloc) (x) = xrealloc((x), st_mult(sizeof(*(x)), (alloc)))
 
 static inline char *xstrdup_or_null(const char *str)
 {

diff --git a/wrapper.c b/wrapper.c
index 52001789ded0372becd3f55dda4cd1226c4b6dae..32c6c609da0fbc3a40d17b34de61bffa7c43700c 100644 (file)
--- a/wrapper.c
+++ b/wrapper.c
@@ -152,6 +152,9 @@ void *xcalloc(size_t nmemb, size_t size)
 {
        void *ret;
 
+       if (unsigned_mult_overflows(nmemb, size))
+               die("data too large to fit into virtual memory space");
+
        memory_limit_check(size * nmemb, 0);
        ret = calloc(nmemb, size);
        if (!ret && (!nmemb || !size))