Handle non-grouped stores as single-lane SLP

The following enables single-lane loop SLP discovery for non-grouped stores
and adjusts vectorizable_store to properly handle those.

For gfortran.dg/vect/vect-8.f90 we vectorize one additional loop,
not running into the "not falling back to strided accesses" bail-out.
I have not investigated in detail.

There is a set of i386 target assembler test FAILs,
gcc.target/i386/pr88531-2[bc].c in particular fail because the
target cannot identify SLP emulated gathers, see another mail from me.
Others need adjustment, I've adjusted one with this patch only.
In particular there are gcc.target/i386/cond_op_fma_*-1.c FAILs
that are because we no longer fold a VEC_COND_EXPR during the
region value-numbering we do after vectorization since we
code-generate a { 0.0, ... } constant in the VEC_COND_EXPR now
instead of having a separate statement which gets forwarded
and then triggers folding.  This leads to sligtly different
code generation.  The solution is probably to use gimple_build
when building stmts or, in this case, directly emit .COND_FMA
instead of .FMA and a VEC_COND_EXPR.

gcc.dg/vect/slp-19a.c mixes contiguous 8-lane SLP with a single
lane contiguous store from one lane of the 8-lane load and we
expect to use load-lanes for this reason but the heuristic for
forcing single-lane rediscovery as implemented doesn't trigger
here as it treats both SLP instances separately.  FAILs on RISC-V

gcc.dg/vect/slp-19c.c shows we fail to implement an interleaving
scheme for group_size 12 (by extension using the group_size 3
scheme to reduce to 4 lanes and then continue with a pow2 scheme
would work);  we are also not considering load-lanes because of
the above reason, but aarch64 cannot do ld12.  FAILs on AARCH64
(load requires three vectors) and x86_64.

gcc.dg/vect/slp-19c.c FAILs with variable-length vectors because
of "SLP induction not supported for variable-length vectors".

gcc.target/aarch64/pr110449.c will FAIL because the (contested)
optimization in r14-2367-g224fd59b2dc8a5 was only applied to
loop-vect but not SLP vect.  I'll leave it to target maintainers
to either XFAIL (the optimization is bad) or remove the test.

	* tree-vect-slp.cc (vect_analyze_slp): Perform single-lane
	loop SLP discovery for non-grouped stores.  Move check on the root
	for re-doing SLP analysis with a single lane for load/store-lanes
	earlier and make sure we are dealing with a grouped access.
	* tree-vect-stmts.cc (vectorizable_store): Always set
	vec_num for SLP.

	* gcc.dg/vect/O3-pr39675-2.c: Adjust expected number of SLP.
	* gcc.dg/vect/fast-math-vect-call-1.c: Likewise.
	* gcc.dg/vect/no-scevccp-slp-31.c: Likewise.
	* gcc.dg/vect/slp-12b.c: Likewise.
	* gcc.dg/vect/slp-12c.c: Likewise.
	* gcc.dg/vect/slp-19a.c: Likewise.
	* gcc.dg/vect/slp-19b.c: Likewise.
	* gcc.dg/vect/slp-4-big-array.c: Likewise.
	* gcc.dg/vect/slp-4.c: Likewise.
	* gcc.dg/vect/slp-5.c: Likewise.
	* gcc.dg/vect/slp-7.c: Likewise.
	* gcc.dg/vect/slp-perm-7.c: Likewise.
	* gcc.dg/vect/slp-37.c: Likewise.
	* gcc.dg/vect/fast-math-vect-call-2.c: Likewise.
	* gcc.dg/vect/slp-26.c: RISC-V can now SLP two instances.
	* gcc.dg/vect/vect-outer-slp-3.c: Disable vectorization of
	initialization loop.
	* gcc.dg/vect/slp-reduc-5.c: Likewise.
	* gcc.dg/vect/no-scevccp-outer-12.c: Un-XFAIL.  SLP can handle
	inner loop inductions with multiple vector stmt copies.
	* gfortran.dg/vect/vect-8.f90: Adjust expected number of
	vectorized loops.
	* gcc.target/i386/vectorize1.c: Adjust what we scan for.

diff --git a/gcc/testsuite/gcc.dg/vect/O3-pr39675-2.c b/gcc/testsuite/gcc.dg/vect/O3-pr39675-2.c
index c3f0f6dc1beb7359e9edc4806e8617ab42bac2b5..ddaac56cc0b9b046e925278e458bc8b44254e1c0 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/O3-pr39675-2.c
+++ b/gcc/testsuite/gcc.dg/vect/O3-pr39675-2.c
@@ -27,5 +27,5 @@ foo ()
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect"  { target vect_strided4 } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target vect_strided4 } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target vect_strided4 } } } */
   

diff --git a/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-1.c b/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-1.c
index ad22f6e82b3c3312c9f10522377c4749e87ce3aa..6c9b7c37b6ec4aae79402d1a97f800ab310753e3 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-1.c
+++ b/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-1.c
@@ -101,4 +101,4 @@ main ()
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" { target { vect_call_copysignf && vect_call_sqrtf } } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 3 "vect" { target { { vect_call_copysignf && vect_call_sqrtf } && vect_perm3_int } } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" { target { { vect_call_copysignf && vect_call_sqrtf } && vect_perm3_int } } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-2.c b/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-2.c
index d51e17ff656b7cc7ef3d87d207f78aae8eec9373..ed42a21cedc16adc4b3e92b0dc24bfaedc6a9a00 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-2.c
+++ b/gcc/testsuite/gcc.dg/vect/fast-math-vect-call-2.c
@@ -132,4 +132,4 @@ main ()
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 6 "vect" { target vect_call_lrint } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" { target vect_call_lrint } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 6 "vect" { target vect_call_lrint } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/no-scevccp-outer-12.c b/gcc/testsuite/gcc.dg/vect/no-scevccp-outer-12.c
index c2d3031bc0cee278b2b066a3b9b8cd130bd0c9ac..6ace6ad022ee288e38ea3603cae6f96588071761 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/no-scevccp-outer-12.c
+++ b/gcc/testsuite/gcc.dg/vect/no-scevccp-outer-12.c
@@ -46,5 +46,4 @@ int main (void)
   return 0;
 }
 
-/* Until we support multiple types in the inner loop  */
-/* { dg-final { scan-tree-dump-times "OUTER LOOP VECTORIZED." 1 "vect" { xfail { ! { aarch64*-*-* riscv*-*-* } } } } } */
+/* { dg-final { scan-tree-dump-times "OUTER LOOP VECTORIZED." 1 "vect" } } */

diff --git a/gcc/testsuite/gcc.dg/vect/no-scevccp-slp-31.c b/gcc/testsuite/gcc.dg/vect/no-scevccp-slp-31.c
index 22817a57ef81398cc018a78597755397d20e0eb9..f6ac5f6029836d392592a73c10e38b4f5fac8e53 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/no-scevccp-slp-31.c
+++ b/gcc/testsuite/gcc.dg/vect/no-scevccp-slp-31.c
@@ -53,6 +53,7 @@ int main (void)
   return 0;
 }
 
+/* We cannot handle grouped accesses in outer loops.  */
+/* { dg-final { scan-tree-dump-not "OUTER LOOP VECTORIZED" "vect" } } */
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect"  } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect"  } } */
-  
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect"  } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-12b.c b/gcc/testsuite/gcc.dg/vect/slp-12b.c
index e2ea24d6c535c60ba903ce2411290e603414009a..8e06e3bfa93ee99a1c29777ff81e1c77b9923c23 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-12b.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-12b.c
@@ -47,6 +47,6 @@ int main (void)
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect"  { target { vect_strided2 && vect_int_mult } } } } */
 /* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect"  { target { ! { vect_strided2 && vect_int_mult } } } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect"  { target { vect_strided2 && vect_int_mult } } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect"  { target { vect_strided2 && vect_int_mult } } } } */
 /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect"  { target { ! { vect_strided2 && vect_int_mult } } } } } */
   

diff --git a/gcc/testsuite/gcc.dg/vect/slp-12c.c b/gcc/testsuite/gcc.dg/vect/slp-12c.c
index 9c48dff3bf486a8cd1843876975dfba40a055a23..a3536e3053bd8a8833c8086d1b0ba063b2f9e0a8 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-12c.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-12c.c
@@ -49,5 +49,5 @@ int main (void)
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect"  { target { vect_int_mult } } } } */
 /* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect"  { target { ! vect_int_mult } } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target vect_int_mult } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target vect_int_mult } } } */
 /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect" { target { ! vect_int_mult } } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-19a.c b/gcc/testsuite/gcc.dg/vect/slp-19a.c
index ca7a0a8e456b1b787ad82e910ea5e3c5e5048c80..6c21416046d12d41163d893c57c0e6a0cc812804 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-19a.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-19a.c
@@ -57,5 +57,5 @@ int main (void)
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target vect_strided8 } } } */
 /* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect" { target { ! vect_strided8 } } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target vect_strided8 } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target vect_strided8 } } } */
 /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect" { target { ! vect_strided8} } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-19b.c b/gcc/testsuite/gcc.dg/vect/slp-19b.c
index 4d53ac698dbd164d20271c4fe9ccc2c20f3c4eaa..10b84aab3b5618c11b36682f6ac3e02c991ff1c8 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-19b.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-19b.c
@@ -54,5 +54,5 @@ int main (void)
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target vect_strided4 } } } */
 /* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect" { target { ! vect_strided4 } } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target vect_strided4 } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target vect_strided4 } } } */
 /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect" { target { ! vect_strided4 } } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-26.c b/gcc/testsuite/gcc.dg/vect/slp-26.c
index cfb763bf519bd7e6b17e3387d677f6a360be2ba2..cdb5d9c694beb9aab06403813b898b31173dc8ff 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-26.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-26.c
@@ -50,4 +50,5 @@ int main (void)
 /* { dg-final { scan-tree-dump-times "vectorized 0 loops" 1 "vect" { target { ! { mips_msa || { amdgcn-*-* || { riscv_v || loongarch_sx } } } } } } } */
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { mips_msa || { amdgcn-*-* || { riscv_v || loongarch_sx } } } } } } */
 /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 0 "vect" { target { ! { mips_msa || { amdgcn-*-* || { riscv_v || loongarch_sx } } } } } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target { mips_msa || { amdgcn-*-* || { riscv_v || loongarch_sx } } } } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target { mips_msa || { amdgcn-*-* || loongarch_sx } } } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target riscv_v } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-37.c b/gcc/testsuite/gcc.dg/vect/slp-37.c
index caee2bb508f1824fa549568dd09911c8624222f4..8a430e63847a0b575b2f40f5607606130c7d599d 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-37.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-37.c
@@ -60,4 +60,4 @@ int main (void)
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target vect_hw_misalign } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target vect_hw_misalign } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target vect_hw_misalign } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-4-big-array.c b/gcc/testsuite/gcc.dg/vect/slp-4-big-array.c
index fcda45ff368511b350b25857f21b2eaeb721561a..f738a613324b2e40cd75ca43ee92c91ec1c525f2 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-4-big-array.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-4-big-array.c
@@ -131,5 +131,5 @@ int main (void)
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 3 loops" 1 "vect"  } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 3 "vect"  } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 6 "vect"  } } */
 

diff --git a/gcc/testsuite/gcc.dg/vect/slp-4.c b/gcc/testsuite/gcc.dg/vect/slp-4.c
index 29e741df02ba0ef6874cde2a4410b79d1d7608ee..1ecad7415ef0fb6a7d479db6997f571c91abddee 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-4.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-4.c
@@ -125,5 +125,5 @@ int main (void)
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 3 loops" 1 "vect"  } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 3 "vect"  } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 6 "vect"  } } */
   

diff --git a/gcc/testsuite/gcc.dg/vect/slp-5.c b/gcc/testsuite/gcc.dg/vect/slp-5.c
index 6d51f6a73234ac41eb2cc4d2fcedc8928d9932b2..484898c2afd4e73ea2eb8826a7a49404fe0cee60 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-5.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-5.c
@@ -124,5 +124,5 @@ int main (void)
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 3 loops" 1 "vect" } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 3 "vect"  } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 5 "vect"  } } */
   

diff --git a/gcc/testsuite/gcc.dg/vect/slp-7.c b/gcc/testsuite/gcc.dg/vect/slp-7.c
index 2845a99dedf5c99032b099a136acd96f37fc5295..f83fdc96d163d6f603024e31fd34b5b5c6bf981a 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-7.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-7.c
@@ -125,6 +125,6 @@ int main (void)
 
 /* { dg-final { scan-tree-dump-times "vectorized 3 loops" 1 "vect"  { target vect_short_mult } } }*/
 /* { dg-final { scan-tree-dump-times "vectorized 2 loops" 1 "vect"  { target { ! { vect_short_mult } } } } }*/
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 3 "vect"  { target vect_short_mult } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect"  { target { ! { vect_short_mult } } } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 5 "vect"  { target vect_short_mult } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect"  { target { ! { vect_short_mult } } } } } */
  

diff --git a/gcc/testsuite/gcc.dg/vect/slp-perm-7.c b/gcc/testsuite/gcc.dg/vect/slp-perm-7.c
index f15736ef729ddc379d8d32f69920849c2b3d986a..9c522ba47059f97cb23c2a060a562c463e535f6c 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-perm-7.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-perm-7.c
@@ -97,6 +97,6 @@ int main (int argc, const char* argv[])
 }
 
 /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect"  { target vect_perm } } } */
-/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target { vect_perm3_int || vect_load_lanes } } } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" { target { vect_perm3_int || vect_load_lanes } } } } */
 /* { dg-final { scan-tree-dump "LOAD_LANES" "vect" { target vect_load_lanes } } } */
 /* { dg-final { scan-tree-dump "STORE_LANES" "vect" { target vect_load_lanes } } } */

diff --git a/gcc/testsuite/gcc.dg/vect/slp-reduc-5.c b/gcc/testsuite/gcc.dg/vect/slp-reduc-5.c
index 11f5a7414cf30eda182bfdddbb52f6f3730954f3..0cde79d9e496396a160681fd3bc738a7b03ebf7a 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/slp-reduc-5.c
+++ b/gcc/testsuite/gcc.dg/vect/slp-reduc-5.c
@@ -36,6 +36,7 @@ int main (void)
 
   check_vect ();
 
+#pragma GCC novector
   for (i = 0; i < N; i++)
     c[i] = (i+3) * -1;
 
@@ -44,6 +45,6 @@ int main (void)
   return 0;
 }
 
-/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 2 "vect" { xfail vect_no_int_min_max } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { xfail vect_no_int_min_max } } } */
 /* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { xfail vect_no_int_min_max } } } */
 /* { dg-final { scan-tree-dump-times "VEC_PERM_EXPR" 0 "vect" } } */

diff --git a/gcc/testsuite/gcc.dg/vect/vect-outer-slp-3.c b/gcc/testsuite/gcc.dg/vect/vect-outer-slp-3.c
index 3dce51426b5b83d85bc93aaaa67bca3e4c29bc44..d315db5632ba6764d0cf23cdedec2e7b6db01ab2 100644 (file)
--- a/gcc/testsuite/gcc.dg/vect/vect-outer-slp-3.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-outer-slp-3.c
@@ -30,6 +30,7 @@ int main ()
 {
   check_vect ();
 
+#pragma GCC novector
   for (int i = 0; i < 40; ++i)
     image[i] = 1.;
 

diff --git a/gcc/testsuite/gcc.target/i386/vectorize1.c b/gcc/testsuite/gcc.target/i386/vectorize1.c
index f3b9bfba38294f0083a1d023b293b423b69d9262..14a8c5f28b3a316085658357279d20632b95b1da 100644 (file)
--- a/gcc/testsuite/gcc.target/i386/vectorize1.c
+++ b/gcc/testsuite/gcc.target/i386/vectorize1.c
@@ -1,6 +1,6 @@
 /* PR middle-end/28915 */
 /* { dg-do compile } */
-/* { dg-options "-msse -O2 -ftree-vectorize -fdump-tree-vect" } */
+/* { dg-options "-msse -O2 -ftree-vectorize -fdump-tree-vect-optimized" } */
 
 extern char lanip[3][40];
 typedef struct
@@ -17,4 +17,4 @@ int set_names (void)
       tt1.t[ln] = lanip[1];
 }
 
-/* { dg-final { scan-tree-dump "vect_cst" "vect" } } */
+/* { dg-final { scan-tree-dump "optimized: loop vectorized" "vect" } } */

diff --git a/gcc/testsuite/gfortran.dg/vect/vect-8.f90 b/gcc/testsuite/gfortran.dg/vect/vect-8.f90
index 557a523e2bd8bfaeb5eda35a1d8904a8d16fb0fc..2a3fa90740e3ee788435888370f60b8bdede813d 100644 (file)
--- a/gcc/testsuite/gfortran.dg/vect/vect-8.f90
+++ b/gcc/testsuite/gfortran.dg/vect/vect-8.f90
@@ -708,5 +708,5 @@ END SUBROUTINE kernel
 
 ! { dg-final { scan-tree-dump-times "vectorized 2\[56\] loops" 1 "vect" { target aarch64_sve } } }
 ! { dg-final { scan-tree-dump-times "vectorized 2\[45\] loops" 1 "vect" { target { aarch64*-*-* && { ! aarch64_sve } } } } }
-! { dg-final { scan-tree-dump-times "vectorized 2\[234\] loops" 1 "vect" { target { vect_intdouble_cvt && { ! aarch64*-*-* } } } } }
+! { dg-final { scan-tree-dump-times "vectorized 2\[345\] loops" 1 "vect" { target { vect_intdouble_cvt && { ! aarch64*-*-* } } } } }
 ! { dg-final { scan-tree-dump-times "vectorized 17 loops" 1 "vect" { target { { ! vect_intdouble_cvt } && { ! aarch64*-*-* } } } } }

diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index b6839c7707b70d040a7d8b5918475773b9dbff0d..3d2973698e27dd78253ebe071de8b0291b64a6ec 100644 (file)
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -4548,6 +4548,7 @@ vect_lower_load_permutations (loop_vec_info loop_vinfo,
 opt_result
 vect_analyze_slp (vec_info *vinfo, unsigned max_tree_size)
 {
+  loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
   unsigned int i;
   stmt_vec_info first_element;
   slp_instance instance;
@@ -4564,6 +4565,28 @@ vect_analyze_slp (vec_info *vinfo, unsigned max_tree_size)
     vect_analyze_slp_instance (vinfo, bst_map, first_element,
                               slp_inst_kind_store, max_tree_size, &limit);
 
+  /* For loops also start SLP discovery from non-grouped stores.  */
+  if (loop_vinfo)
+    {
+      data_reference_p dr;
+      FOR_EACH_VEC_ELT (vinfo->shared->datarefs, i, dr)
+       if (DR_IS_WRITE (dr))
+         {
+           stmt_vec_info stmt_info = vinfo->lookup_dr (dr)->stmt;
+           /* Grouped stores are already handled above.  */
+           if (STMT_VINFO_GROUPED_ACCESS (stmt_info))
+             continue;
+           vec<stmt_vec_info> stmts;
+           vec<stmt_vec_info> roots = vNULL;
+           vec<tree> remain = vNULL;
+           stmts.create (1);
+           stmts.quick_push (stmt_info);
+           vect_build_slp_instance (vinfo, slp_inst_kind_store,
+                                    stmts, roots, remain, max_tree_size,
+                                    &limit, bst_map, NULL);
+         }
+    }
+
   if (bb_vec_info bb_vinfo = dyn_cast <bb_vec_info> (vinfo))
     {
       for (unsigned i = 0; i < bb_vinfo->roots.length (); ++i)
@@ -4750,6 +4773,18 @@ vect_analyze_slp (vec_info *vinfo, unsigned max_tree_size)
          int group_size = SLP_TREE_LANES (slp_root);
          tree vectype = SLP_TREE_VECTYPE (slp_root);
 
+         stmt_vec_info rep_info = SLP_TREE_REPRESENTATIVE (slp_root);
+         gimple *rep = STMT_VINFO_STMT (rep_info);
+         bool masked = (is_gimple_call (rep)
+                        && gimple_call_internal_p (rep)
+                        && internal_fn_mask_index
+                             (gimple_call_internal_fn (rep)) != -1);
+         if (!STMT_VINFO_GROUPED_ACCESS (rep_info)
+             || slp_root->ldst_lanes
+             || (vect_store_lanes_supported (vectype, group_size, masked)
+                 == IFN_LAST))
+           continue;
+
          auto_vec<slp_tree> loads;
          hash_set<slp_tree> visited;
          vect_gather_slp_loads (loads, slp_root, visited);
@@ -4773,17 +4808,9 @@ vect_analyze_slp (vec_info *vinfo, unsigned max_tree_size)
                  }
            }
 
-         gimple *rep = STMT_VINFO_STMT (SLP_TREE_REPRESENTATIVE (slp_root));
-         bool masked = (is_gimple_call (rep)
-                        && gimple_call_internal_p (rep)
-                        && internal_fn_mask_index
-                             (gimple_call_internal_fn (rep)) != -1);
          /* If the loads and stores can use load/store-lanes force re-discovery
             with single lanes.  */
-         if (loads_permuted
-             && !slp_root->ldst_lanes
-             && vect_store_lanes_supported (vectype, group_size, masked)
-             != IFN_LAST)
+         if (loads_permuted)
            {
              bool can_use_lanes = true;
              FOR_EACH_VEC_ELT (loads, j, load_node)

diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 25b120c158e678247666bc37245df2347d80ecc9..f6c5b7a7e8727f54fdf28e3b63bf5e95e0537a04 100644 (file)
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -8355,10 +8355,12 @@ vectorizable_store (vec_info *vinfo,
       return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies);
     }
 
-  if (grouped_store)
+  if (grouped_store || slp)
     {
       /* FORNOW */
-      gcc_assert (!loop || !nested_in_vect_loop_p (loop, stmt_info));
+      gcc_assert (!grouped_store
+                 || !loop
+                 || !nested_in_vect_loop_p (loop, stmt_info));
 
       if (slp)
         {
@@ -8367,8 +8369,9 @@ vectorizable_store (vec_info *vinfo,
              group.  */
           vec_num = SLP_TREE_NUMBER_OF_VEC_STMTS (slp_node);
          first_stmt_info = SLP_TREE_SCALAR_STMTS (slp_node)[0];
-         gcc_assert (DR_GROUP_FIRST_ELEMENT (first_stmt_info)
-                     == first_stmt_info);
+         gcc_assert (!STMT_VINFO_GROUPED_ACCESS (first_stmt_info)
+                     || (DR_GROUP_FIRST_ELEMENT (first_stmt_info)
+                         == first_stmt_info));
          first_dr_info = STMT_VINFO_DR_INFO (first_stmt_info);
          op = vect_get_store_rhs (first_stmt_info);
         }