arm: avoid invalid shift in arm_canonicalize_comparison [PR122999]

There was UB in arm_canonicalize_comparison if it is called with
both operands of type VOIDmode.  Avoid this by first handling
floating-point types, then returning if we are left with anything
other than an integer mode.  For belt-and-braces also check that
the mode does not require a mask larger than HOST_WIDE_INT.

gcc/ChangeLog:

	PR target/122999
	* config/arm/arm.cc (arm_canonicalize_comparison): Defer
	initializing maxval until we know we are dealing with an
	integer mode.

diff --git a/gcc/config/arm/arm.cc b/gcc/config/arm/arm.cc
index 20d3f1f4578b63b1b948f635c01bd7812548e5c8..1f413b61d5f6d1ab0e249f14b2b63db574d8deaa 100644 (file)
--- a/gcc/config/arm/arm.cc
+++ b/gcc/config/arm/arm.cc
@@ -5700,8 +5700,6 @@ arm_canonicalize_comparison (int *code, rtx *op0, rtx *op1,
   if (mode == VOIDmode)
     mode = GET_MODE (*op1);
 
-  maxval = (HOST_WIDE_INT_1U << (GET_MODE_BITSIZE (mode) - 1)) - 1;
-
   /* For floating-point comparisons, prefer >= and > over <= and < since
      the former are supported by VSEL on some architectures.  Only do this
      if both operands are registers.  */
@@ -5718,6 +5716,13 @@ arm_canonicalize_comparison (int *code, rtx *op0, rtx *op1,
       return;
     }
 
+  /* Everything below assumes an integer mode.  */
+  if (GET_MODE_CLASS (mode) != MODE_INT
+      || GET_MODE_BITSIZE (mode) > HOST_BITS_PER_WIDE_INT)
+    return;
+
+  maxval = (HOST_WIDE_INT_1U << (GET_MODE_BITSIZE (mode) - 1)) - 1;
+
   /* For DImode, we have GE/LT/GEU/LTU comparisons (with cmp/sbc).  In
      ARM mode we can also use cmp/cmpeq for GTU/LEU.  GT/LE must be
      either reversed or (for constant OP1) adjusted to GE/LT.