VECT: Apply LEN_MASK_GATHER_LOAD/SCATTER_STORE into vectorizer
Hi, Richard and Richi.
Address comments from Richi.
Make gs_info.ifn = LEN_MASK_GATHER_LOAD/LEN_MASK_SCATTER_STORE.
I have fully tested these 4 format:
length = vf is a dummpy length,
mask = {-1,-1, ... } is a dummy mask.
1. no length, no mask
LEN_MASK_GATHER_LOAD (..., length = vf, mask = {-1,-1,...})
2. exist length, no mask
LEN_MASK_GATHER_LOAD (..., len, mask = {-1,-1,...})
3. exist mask, no length
LEN_MASK_GATHER_LOAD (..., length = vf, mask)
4. both mask and length exist
LEN_MASK_GATHER_LOAD (..., length, mask)
All of these work fine in this patch.
Here is the example:
void
f (int *restrict a,
int *restrict b, int n,
int base, int step,
int *restrict cond)
{
for (int i = 0; i < n; ++i)
{
if (cond[i])
a[i * 4] = b[i];
}
}
Gimple IR:
<bb 3> [local count:
105119324]:
_58 = (unsigned long) n_13(D);
<bb 4> [local count:
630715945]:
# vectp_cond.7_45 = PHI <vectp_cond.7_46(4), cond_14(D)(3)>
# vectp_b.11_51 = PHI <vectp_b.11_52(4), b_15(D)(3)>
# vectp_a.14_55 = PHI <vectp_a.14_56(4), a_16(D)(3)>
# ivtmp_59 = PHI <ivtmp_60(4), _58(3)>
_61 = .SELECT_VL (ivtmp_59, POLY_INT_CST [2, 2]);
ivtmp_44 = _61 * 4;
vect__4.9_47 = .LEN_MASK_LOAD (vectp_cond.7_45, 32B, _61, 0, { -1, ... });
mask__24.10_49 = vect__4.9_47 != { 0, ... };
vect__8.13_53 = .LEN_MASK_LOAD (vectp_b.11_51, 32B, _61, 0, mask__24.10_49);
ivtmp_54 = _61 * 16;
.LEN_MASK_SCATTER_STORE (vectp_a.14_55, { 0, 16, 32, ... }, 1, vect__8.13_53, _61, 0, mask__24.10_49);
vectp_cond.7_46 = vectp_cond.7_45 + ivtmp_44;
vectp_b.11_52 = vectp_b.11_51 + ivtmp_44;
vectp_a.14_56 = vectp_a.14_55 + ivtmp_54;
ivtmp_60 = ivtmp_59 - _61;
if (ivtmp_60 != 0)
goto <bb 4>; [83.33%]
else
goto <bb 5>; [16.67%]
Ok for trunk ?
gcc/ChangeLog:
* internal-fn.cc (internal_fn_len_index): Apply
LEN_MASK_GATHER_LOAD/SCATTER_STORE into vectorizer.
(internal_fn_mask_index): Ditto.
* optabs-query.cc (supports_vec_gather_load_p): Ditto.
(supports_vec_scatter_store_p): Ditto.
* tree-vect-data-refs.cc (vect_gather_scatter_fn_p): Ditto.
* tree-vect-patterns.cc (vect_recog_gather_scatter_pattern): Ditto.
* tree-vect-stmts.cc (check_load_store_for_partial_vectors): Ditto.
(vect_get_strided_load_store_ops): Ditto.
(vectorizable_store): Ditto.
(vectorizable_load): Ditto.