ldb: reduce non-transitive comparisons in ldb_msg_element_compare()

We can still have inconsistent comparisons, because two elements with
the same number of values will always return -1 if they are unequal,
which means they will sort differently depending on the order in which
they are compared.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=15625

Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
(cherry picked from commit 21a071e4864dd739840c2ad4adb0c71ec33f8427)

diff --git a/lib/ldb/common/ldb_msg.c b/lib/ldb/common/ldb_msg.c
index 5c5f6a7c2b485d0eaa1d3780ceb4322d52185d1e..e8164474882da44a5cde74670c356e4bacbc6093 100644 (file)
--- a/lib/ldb/common/ldb_msg.c
+++ b/lib/ldb/common/ldb_msg.c
@@ -749,9 +749,16 @@ int ldb_msg_element_compare(struct ldb_message_element *el1,
        unsigned int i;
 
        if (el1->num_values != el2->num_values) {
-               return el1->num_values - el2->num_values;
+               return NUMERIC_CMP(el1->num_values, el2->num_values);
        }
-
+       /*
+        * Note this is an inconsistent comparison, unsuitable for
+        * sorting. If A has values {a, b} and B has values {b, c},
+        * then
+        *
+        * ldb_msg_element_compare(A, B) returns -1, meaning A < B
+        * ldb_msg_element_compare(B, A) returns -1, meaning B < A
+        */
        for (i=0;i<el1->num_values;i++) {
                if (!ldb_msg_find_val(el2, &el1->values[i])) {
                        return -1;