i386: return 256/512-bit vectors in registers for x86_64 MS ABI [PR89597]
On x86_64 Windows targets using MS ABI, GCC classified 256-bit and
512-bit vector returns as memory returns. That caused hidden sret
pointer returns where YMM0/ZMM0 returns are expected.
Teach MS ABI return classification to keep 32-byte and 64-byte vector
returns in registers when AVX/AVX512F is enabled, matching the return
register selection path.
Also extend function_value_ms_64 so 32-byte and 64-byte eligible vector
returns are mapped to the SSE register class (YMM0/ZMM0 lanes).
Add tests for x86_64-*-mingw* that verify 256-bit and 512-bit vector
returns use YMM0/ZMM0 codegen.
gcc:
PR target/89597
* config/i386/i386.cc (function_value_ms_64): Handle 32-byte and
64-byte vector returns in registers when supported.
(ix86_return_in_memory): Do not force 32-byte/64-byte eligible
vector returns to memory for MS ABI.
gcc/testsuite:
* gcc.target/i386/pr89597-1.c: New test.
* gcc.target/i386/pr89597-2.c: New test.
Signed-off-by: Oleg Tolmatcev <oleg.tolmatcev@gmail.com> Signed-off-by: Jonathan Yong <10walls@gmail.com>