KVM: arm64: nv: Don't save/restore FP register during a nested ERET or exception

When switching between L1 and L2, we save the old state using
kvm_arch_vcpu_put(), mutate the state in memory, then load the new
state using kvm_arch_vcpu_load(). Any live FPSIMD/SVE state is saved
and unbound, such that it can be lazily restored on a subsequent trap.

The FPSIMD/SVE state is shared by exception levels, and only a handful
of related control registers need to be changed when transitioning
between L1 and L2. The save/restore of the common state is needless
overhead, especially as trapping becomes exponentially more expensive
with nesting.

Avoid this overhead by leaving the common FPSIMD/SVE state live on the
CPU, and only switching the state that is distinct for L1 and L2:

- the trap controls: the effective values are recomputed on each entry
  into the guest to take the EL into account and merge the L0 and L1
  configuration if in a nested context, or directly use the L0 configuration
  in non-nested context (see __activate_traps()).

- the VL settings: the effective values are are also recomputed on each
  entry into the guest (see fpsimd_lazy_switch_to_guest()).

Since we appear to cover all bases, use the vcpu flags indicating the
handling of a nested ERET or exception delivery to avoid the whole FP
save/restore shenanigans. SME will have to be similarly dealt with when
it eventually gets supported.

For an EL1 L3 guest where L1 and L2 have this optimisation, this
results in at least a 10% wall clock reduction when running an I/O
heavy workload, generating a high rate of nested exceptions.

Reviewed-by: Joey Gouly <joey.gouly@arm.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://patch.msgid.link/20260520085036.541666-3-maz@kernel.org

diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c
index 15e17aca1dec0eb16311593c0a283aa28976b817..3f6b1e29cd6b9754fb79700419cccb37388003a5 100644 (file)
--- a/arch/arm64/kvm/fpsimd.c
+++ b/arch/arm64/kvm/fpsimd.c
@@ -28,6 +28,20 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu)
        if (!system_supports_fpsimd())
                return;
 
+       /*
+        * Avoid needless save/restore of the guest's common
+        * FPSIMD/SVE/SME regs during transitions between L1/L2.
+        *
+        * These transitions only happens in a non-preemptible context
+        * where the host regs have already been saved and unbound. The
+        * live registers are either free or owned by the guest.
+        */
+       if (vcpu_get_flag(vcpu, IN_NESTED_ERET) ||
+           vcpu_get_flag(vcpu, IN_NESTED_EXCEPTION)) {
+               WARN_ON_ONCE(host_owns_fp_regs());
+               return;
+       }
+
        /*
         * Ensure that any host FPSIMD/SVE/SME state is saved and unbound such
         * that the host kernel is responsible for restoring this state upon
@@ -102,6 +116,18 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)
 {
        unsigned long flags;
 
+       /*
+        * See comment in kvm_arch_vcpu_load_fp(). Note that we also rely on
+        * the guest's max VL to have been set by fpsimd_lazy_switch_to_host()
+        * so that any intervening kernel-mode SIMD (NEON or otherwise)
+        * operation sees the full guest state that needs saving.
+        */
+       if (vcpu_get_flag(vcpu, IN_NESTED_ERET) ||
+           vcpu_get_flag(vcpu, IN_NESTED_EXCEPTION)) {
+               WARN_ON_ONCE(host_owns_fp_regs());
+               return;
+       }
+
        local_irq_save(flags);
 
        if (guest_owns_fp_regs()) {