rtlanal: Determine nonzero bits of popcount from operand [PR123501].

rtlanal: Determine nonzero bits of popcount from operand [PR123501].

The PR involves large mask vectors (e.g. V128BI) from which we take
the popcount.  Currently a (popcount:DI (V128BI)) is assumed to have
at most 8 set bits as we assume the popcount operand also has DImode.

This patch uses the operand mode for unary operations and thus
calculates a proper nonzero-bits mask.

We could do the same estimate for ctz and clz but they use nonzero in a
non-poly way and I didn't want to change more than necessary.  Therefore
the patch just returns -1 when we have a different operand mode for
ctz/clz.

	PR rtl-optimization/123501
	PR rtl-optimization/123444

gcc/ChangeLog:

	* rtlanal.cc (nonzero_bits1): Use operand mode instead of
	operation mode.

gcc/testsuite/ChangeLog:

	* gcc.target/riscv/rvv/autovec/reduc/pr123501.c: New test.