c++: Optimize away NULLPTR_TYPE comparisons [PR101443]

Comparisons of NULLPTR_TYPE operands cause all kinds of problems in the
middle-end and in fold-const.c, various optimizations assume that if they
see e.g. a non-equality comparison with one of the operands being
INTEGER_CST and it is not INTEGRAL_TYPE_P (which has TYPE_{MIN,MAX}_VALUE),
they can build_int_cst (type, 1) to find a successor.

The following patch fixes it by making sure they don't appear in the IL,
optimize them away at cp_fold time as all can be folded.

Though, I've just noticed that clang++ rejects the non-equality comparisons
instead, foo () > 0 with
invalid operands to binary expression ('decltype(nullptr)' (aka 'nullptr_t') and 'int')
and foo () > nullptr with
invalid operands to binary expression ('decltype(nullptr)' (aka 'nullptr_t') and 'nullptr_t')

Shall we reject those too, in addition or instead of parts of this patch?
If so, wouldn't this patch be still useful for backports, I bet we don't
want to start reject it on the release branches when we used to accept it.

2021-07-15  Jakub Jelinek  <jakub@redhat.com>

	PR c++/101443
	* cp-gimplify.c (cp_fold): For comparisons with NULLPTR_TYPE
	operands, fold them right away to true or false.

	* g++.dg/cpp0x/nullptr46.C: New test.

diff --git a/gcc/cp/cp-gimplify.c b/gcc/cp/cp-gimplify.c
index de37f2cdfdca79414a1816153de3cdde79286b28..ff0bff771df69fa2092ea63571e6f45537bd7c7e 100644 (file)
--- a/gcc/cp/cp-gimplify.c
+++ b/gcc/cp/cp-gimplify.c
@@ -2423,6 +2423,32 @@ cp_fold (tree x)
       op0 = cp_fold_maybe_rvalue (TREE_OPERAND (x, 0), rval_ops);
       op1 = cp_fold_rvalue (TREE_OPERAND (x, 1));
 
+      /* decltype(nullptr) has only one value, so optimize away all comparisons
+        with that type right away, keeping them in the IL causes troubles for
+        various optimizations.  */
+      if (COMPARISON_CLASS_P (org_x)
+         && TREE_CODE (TREE_TYPE (op0)) == NULLPTR_TYPE
+         && TREE_CODE (TREE_TYPE (op1)) == NULLPTR_TYPE)
+       {
+         switch (code)
+           {
+           case EQ_EXPR:
+           case LE_EXPR:
+           case GE_EXPR:
+             x = constant_boolean_node (true, TREE_TYPE (x));
+             break;
+           case NE_EXPR:
+           case LT_EXPR:
+           case GT_EXPR:
+             x = constant_boolean_node (false, TREE_TYPE (x));
+             break;
+           default:
+             gcc_unreachable ();
+           }
+         return omit_two_operands_loc (loc, TREE_TYPE (x), x,
+                                       op0, op1);
+       }
+
       if (op0 != TREE_OPERAND (x, 0) || op1 != TREE_OPERAND (x, 1))
        {
          if (op0 == error_mark_node || op1 == error_mark_node)

diff --git a/gcc/testsuite/g++.dg/cpp0x/nullptr46.C b/gcc/testsuite/g++.dg/cpp0x/nullptr46.C
new file mode 100644 (file)
index 0000000..1514cee
--- /dev/null
+++ b/gcc/testsuite/g++.dg/cpp0x/nullptr46.C
@@ -0,0 +1,11 @@
+// PR c++/101443
+// { dg-do compile { target c++11 } }
+// { dg-options "-O2" }
+
+decltype(nullptr) foo ();
+
+bool
+bar ()
+{
+  return foo () > nullptr || foo () < nullptr;
+}