Turns out for consistency with LLVM the +nofp option shouldn't remove ALL of FP and MVE, just the FP part of MVE.
This requires more surgery with feature bits so for GCC 10 I'd rather just not support +nofp for -mcpu=cortex-m55
and implement it properly for GCC 11.