From: David Rowley Date: Sun, 4 Jan 2026 07:34:01 +0000 (+1300) Subject: Fix selectivity estimation integer overflow in contrib/intarray X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=54f82c4aae768037fe14d909b2b535cfbe89f900;p=thirdparty%2Fpostgresql.git Fix selectivity estimation integer overflow in contrib/intarray This fixes a poorly written integer comparison function which was performing subtraction in an attempt to return a negative value when a < b and a positive value when a > b, and 0 when the values were equal. Unfortunately that didn't always work correctly due to two's complement having the INT_MIN 1 further from zero than INT_MAX. This could result in an overflow and cause the comparison function to return an incorrect result, which would result in the binary search failing to find the value being searched for. This could cause poor selectivity estimates when the statistics stored the value of INT_MAX (2147483647) and the value being searched for was large enough to result in the binary search doing a comparison with that INT_MAX value. Author: Chao Li Reviewed-by: David Rowley Discussion: https://postgr.es/m/CAEoWx2ng1Ot5LoKbVU-Dh---dFTUZWJRH8wv2chBu29fnNDMaQ@mail.gmail.com Backpatch-through: 14 --- diff --git a/contrib/intarray/_int_selfuncs.c b/contrib/intarray/_int_selfuncs.c index 17dc6768ea5..43146feb18f 100644 --- a/contrib/intarray/_int_selfuncs.c +++ b/contrib/intarray/_int_selfuncs.c @@ -327,7 +327,12 @@ static int compare_val_int4(const void *a, const void *b) { int32 key = *(int32 *) a; - const Datum *t = (const Datum *) b; + int32 value = DatumGetInt32(*(const Datum *) b); - return key - DatumGetInt32(*t); + if (key < value) + return -1; + else if (key > value) + return 1; + else + return 0; }