bswap: Fix up ubsan detected UB in find_bswap_or_nop [PR121322]

The following testcase results in compiler UB as detected by ubsan.
find_bswap_or_nop first checks is_bswap_or_nop_p and if that fails
on the tmp_n value, tries some rotation of that if possible.
The discovery what rotate count to use ignores zero bytes from
the least significant end (those mean zero bytes and so can be masked
away) and on the first non-zero non-0xff byte (0xff means don't know),
1-8 means some particular byte of the original computes count (the rotation
count) from that byte + the byte index.
Now, on the following testcase we have tmp_n 0x403020105060700, i.e.
the least significant byte is zero, then the msb from the original value,
byte below it, another one below it, then the low 32 bits of the original
value.  So, we stop at count 7 with i 1, it wraps around and we get count
0.
Then we invoke UB on
          tmp_n = tmp_n >> count | tmp_n << (range - count);
because count is 0 and range is 64.
Now, of course I could fix it up by doing tmp_n << ((range - count) % range)
or something similar, but that is just wasted compile time, if count is 0,
we already know that is_bswap_or_nop_p failed on that tmp_n value and
so it will fail again if the value is the same.  So I think better
just return NULL (i.e. punt).

2025-08-01  Jakub Jelinek  <jakub@redhat.com>

	PR middle-end/121322
	* gimple-ssa-store-merging.cc (find_bswap_or_nop): Return NULL if
	count is 0.

	* gcc.dg/pr121322.c: New test.

(cherry picked from commit d8fc7bce9cf722aa73782ba66a74646c8ae545cc)

diff --git a/gcc/gimple-ssa-store-merging.cc b/gcc/gimple-ssa-store-merging.cc
index 90f449ce82f489628e9b4b819f100aab61e2e930..8eedb2a7ce4a6e4b5f03c13b87f4c72df76b9cba 100644 (file)
--- a/gcc/gimple-ssa-store-merging.cc
+++ b/gcc/gimple-ssa-store-merging.cc
@@ -1057,6 +1057,8 @@ find_bswap_or_nop (gimple *stmt, struct symbolic_number *n, bool *bswap,
                      if (count <= range / BITS_PER_MARKER)
                        {
                          count = (count + i) * BITS_PER_MARKER % range;
+                         if (!count)
+                           return NULL;
                          break;
                        }
                      else

diff --git a/gcc/testsuite/gcc.dg/pr121322.c b/gcc/testsuite/gcc.dg/pr121322.c
new file mode 100644 (file)
index 0000000..2fad5b5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/pr121322.c
@@ -0,0 +1,14 @@
+/* PR middle-end/121322 */
+/* { dg-do compile } */
+/* { dg-options "-O2" } */
+
+unsigned long long
+foo (unsigned long long *p)
+{
+  unsigned long long a = *p;
+  unsigned long long b = __builtin_bswap64 (a);
+  return ((b << 32)
+         | ((b >> 8) & 0xff000000ULL)
+         | ((b >> 24) & 0xff0000ULL)
+         | ((b >> 40) & 0xff00ULL));
+}