Fix selectivity estimation integer overflow in contrib/intarray

This fixes a poorly written integer comparison function which was
performing subtraction in an attempt to return a negative value when
a < b and a positive value when a > b, and 0 when the values were equal.
Unfortunately that didn't always work correctly due to two's complement
having the INT_MIN 1 further from zero than INT_MAX.  This could result
in an overflow and cause the comparison function to return an incorrect
result, which would result in the binary search failing to find the
value being searched for.

This could cause poor selectivity estimates when the statistics stored
the value of INT_MAX (2147483647) and the value being searched for was
large enough to result in the binary search doing a comparison with that
INT_MAX value.

Author: Chao Li <li.evan.chao@gmail.com>
Reviewed-by: David Rowley <dgrowleyml@gmail.com>
Discussion: https://postgr.es/m/CAEoWx2ng1Ot5LoKbVU-Dh---dFTUZWJRH8wv2chBu29fnNDMaQ@mail.gmail.com
Backpatch-through: 14

diff --git a/contrib/intarray/_int_selfuncs.c b/contrib/intarray/_int_selfuncs.c
index 88ce0f8601a70ae22c411748892d28d3f7f83f09..c3e19cdf27f2b7f7eabf4af9cc35ad474ce7d32d 100644 (file)
--- a/contrib/intarray/_int_selfuncs.c
+++ b/contrib/intarray/_int_selfuncs.c
@@ -329,7 +329,12 @@ static int
 compare_val_int4(const void *a, const void *b)
 {
        int32           key = *(int32 *) a;
-       const Datum *t = (const Datum *) b;
+       int32           value = DatumGetInt32(*(const Datum *) b);
 
-       return key - DatumGetInt32(*t);
+       if (key < value)
+               return -1;
+       else if (key > value)
+               return 1;
+       else
+               return 0;
 }