KVM: SVM: Handle #MCs in guest outside of fastpath

Handle Machine Checks (#MC) that happen in the guest (by forwarding them
to the host) outside of KVM's fastpath so that as much host state as
possible is re-loaded before invoking the kernel's #MC handler.  The only
requirement is that KVM invokes the #MC handler before enabling IRQs (and
even that could _probably_ be relaxed to handling #MCs before enabling
preemption).

Waiting to handle #MCs until "more" host state is loaded hardens KVM
against flaws in the #MC handler, which has historically been quite
brittle. E.g. prior to commit 5567d11c21a1 ("x86/mce: Send #MC singal from
task work"), the #MC code could trigger a schedule() with IRQs and
preemption disabled.  That led to a KVM hack-a-fix in commit 1811d979c716
("x86/kvm: move kvm_load/put_guest_xcr0 into atomic context").

Note, except for #MCs on VM-Enter, VMX already handles #MCs outside of the
fastpath.

Reviewed-by: Rick Edgecombe <rick.p.edgecombe@intel.com>
Reviewed-by: Jon Kohler <jon@nutanix.com>
Link: https://patch.msgid.link/20251118222328.2265758-2-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 59d896322855ce15d2a2e7d9bd1ee0bf4407b379..ff6ddba16a2608e6caf034dd1bca3bee586e9d76 100644 (file)
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4325,14 +4325,6 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
        vcpu->arch.regs_avail &= ~SVM_REGS_LAZY_LOAD_SET;
 
-       /*
-        * We need to handle MC intercepts here before the vcpu has a chance to
-        * change the physical cpu
-        */
-       if (unlikely(svm->vmcb->control.exit_code ==
-                    SVM_EXIT_EXCP_BASE + MC_VECTOR))
-               svm_handle_mce(vcpu);
-
        trace_kvm_exit(vcpu, KVM_ISA_SVM);
 
        svm_complete_interrupts(vcpu);
@@ -4621,8 +4613,16 @@ out:
 
 static void svm_handle_exit_irqoff(struct kvm_vcpu *vcpu)
 {
-       if (to_svm(vcpu)->vmcb->control.exit_code == SVM_EXIT_INTR)
+       switch (to_svm(vcpu)->vmcb->control.exit_code) {
+       case SVM_EXIT_EXCP_BASE + MC_VECTOR:
+               svm_handle_mce(vcpu);
+               break;
+       case SVM_EXIT_INTR:
                vcpu->arch.at_instruction_boundary = true;
+               break;
+       default:
+               break;
+       }
 }
 
 static void svm_setup_mce(struct kvm_vcpu *vcpu)