From f0563f6658b9185727ef5a125a08f5dec9502cde Mon Sep 17 00:00:00 2001 From: Victor Do Nascimento Date: Tue, 30 Sep 2025 13:47:27 +0100 Subject: [PATCH] vect: Make all exit conditions early breaks for uncounted loops For uncounted loops, given how no information can be derived from the max number of iterations, we wish to make no distinction between the "main" exit and any additional exits. That is, an epilogue is required across all exits to establish when the exit condition was met within the final vectorized loop iteration. This can be accomplished via a two-fold approach: 1. The need for all exits to go to the epilogue is shared with counted loops with early-break exits after the IV-counting exit. Such counted loops have the `LOOP_VINFO_EARLY_BREAKS_VECT_PEELED' flag set. By modifying the flag's definition to encompass uncounted loops, we can make considerable use of the code written for this category of counted loops. 2. Currently, when populating the `loop_vinfo' struct for a given loop, there is an assumption that the first exit condition in the `vect_loop_form_info' struct is the IV condition and any subsequent conditions are early break conditions. This assumption breaks down for uncounted loops where _all_ exits should be treated as being "early break" exits. This is fixed by populating `LOOP_VINFO_LOOP_IV_COND (loop_vinfo)' conditionally on the false evaluation of the `LOOP_VINFO_NITERS_UNCOUNTED_P (loop_vinfo)' predicate, such that if we do have an uncounted loop we leave this field unpopulated, storing all conditions in `LOOP_VINFO_LOOP_CONDS (loop_vinfo)'. This approach has a further benefit in that, give how `LOOP_VINFO_EARLY_BREAKS' is defined by a non-empty list of early break exit conditions (`LOOP_VINFO_LOOP_CONDS (loop_vinfo)'), having LOOP_VINFO_LOOP_CONDS populated even for single-exit uncounted loops means that `LOOP_VINFO_EARLY_BREAKS' evaluates to true for all uncounted loops, irrespective of the number of exits it has. gcc/ChangeLog: * tree-vectorizer.h (LOOP_VINFO_EARLY_BREAKS_VECT_PEELED): OR its current definition with `LOOP_VINFO_NITERS_UNCOUNTED_P(L)' * tree-vect-loop.cc (vect_create_loop_vinfo): Don't populate `LOOP_VINFO_LOOP_IV_COND' for uncounted loops. --- gcc/tree-vect-loop.cc | 9 ++++++--- gcc/tree-vectorizer.h | 3 ++- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index eb89d8e6ebe..3a38732e746 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -1653,9 +1653,12 @@ vect_create_loop_vinfo (class loop *loop, vec_info_shared *shared, STMT_VINFO_DEF_TYPE (loop_cond_info) = vect_condition_def; } - for (unsigned i = 1; i < info->conds.length (); i ++) - LOOP_VINFO_LOOP_CONDS (loop_vinfo).safe_push (info->conds[i]); - LOOP_VINFO_LOOP_IV_COND (loop_vinfo) = info->conds[0]; + unsigned cond_id = 0; + if (!LOOP_VINFO_NITERS_UNCOUNTED_P (loop_vinfo)) + LOOP_VINFO_LOOP_IV_COND (loop_vinfo) = info->conds[cond_id++]; + + for (; cond_id < info->conds.length (); cond_id ++) + LOOP_VINFO_LOOP_CONDS (loop_vinfo).safe_push (info->conds[cond_id]); LOOP_VINFO_IV_EXIT (loop_vinfo) = info->loop_exit; diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index 5b68c3b46fe..c9dee90c470 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -1313,7 +1313,8 @@ public: #define LOOP_VINFO_EARLY_BREAKS(L) (L)->early_breaks #define LOOP_VINFO_EARLY_BRK_STORES(L) (L)->early_break_stores #define LOOP_VINFO_EARLY_BREAKS_VECT_PEELED(L) \ - (single_pred ((L)->loop->latch) != (L)->vec_loop_iv_exit->src) + ((single_pred ((L)->loop->latch) != (L)->vec_loop_iv_exit->src) \ + || LOOP_VINFO_NITERS_UNCOUNTED_P (L)) #define LOOP_VINFO_EARLY_BREAKS_LIVE_IVS(L) \ (L)->early_break_live_ivs #define LOOP_VINFO_EARLY_BRK_DEST_BB(L) (L)->early_break_dest_bb -- 2.47.3