KVM: SVM: Refresh vcpu->arch.cr{0,3} prior to invoking fastpath handler

Refresh KVM's copies of CR0 and CR3 from the VMCB prior to (potentially)
invoking a fastpath handler to ensure that KVM doesn't consume stale
state.  While it's unlikely KVM will ever consume CR3 or CR0.{TS,MP} in
the fastpath, grabbing the values from the VMCB is inexpensive, i.e. the
risk of subtle bugs far outweighs the reward of deferring reads for a
small subset of VM-Exits.

Note, KVM doesn't currently consume CR3 or CR0.{TS,MP} in the fastpath,
as KVM requires next_rip to be valid (i.e. KVM doesn't read CR3 to decode
the instruction), CR0.MP is never consumed, and CR0.TS is only consumed by
the full emulator.

Reviewed-by: Nikunj A. Dadhania <nikunj@amd.com>
Link: https://patch.msgid.link/20260423162628.490962-3-seanjc@google.com
Signed-off-by: Sean Christopherson <seanjc@google.com>

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index fd0362874756e2e13b817976c86b462bd2dd5655..95c411da6f2ceeec0147febb43d8c1e6fd2c1103 100644 (file)
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3663,14 +3663,6 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
        struct vcpu_svm *svm = to_svm(vcpu);
        struct kvm_run *kvm_run = vcpu->run;
 
-       /* SEV-ES guests must use the CR write traps to track CR registers. */
-       if (!is_sev_es_guest(vcpu)) {
-               if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE))
-                       vcpu->arch.cr0 = svm->vmcb->save.cr0;
-               if (npt_enabled)
-                       vcpu->arch.cr3 = svm->vmcb->save.cr3;
-       }
-
        if (unlikely(exit_fastpath == EXIT_FASTPATH_EXIT_USERSPACE))
                return 0;
 
@@ -4527,11 +4519,17 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
        if (!static_cpu_has(X86_FEATURE_V_SPEC_CTRL))
                x86_spec_ctrl_restore_host(svm->virt_spec_ctrl);
 
+       /* SEV-ES guests must use the CR write traps to track CR registers. */
        if (!is_sev_es_guest(vcpu)) {
                vcpu->arch.cr2 = svm->vmcb->save.cr2;
                vcpu->arch.regs[VCPU_REGS_RAX] = svm->vmcb->save.rax;
                vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
                vcpu->arch.rip = svm->vmcb->save.rip;
+
+               if (!svm_is_intercept(svm, INTERCEPT_CR0_WRITE))
+                       vcpu->arch.cr0 = svm->vmcb->save.cr0;
+               if (npt_enabled)
+                       vcpu->arch.cr3 = svm->vmcb->save.cr3;
        }
        kvm_reset_dirty_registers(vcpu);