From: Greg Kroah-Hartman Date: Mon, 19 May 2025 06:52:41 +0000 (+0200) Subject: 5.15-stable patches X-Git-Tag: v5.15.184~55 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=16551088758ff0e4c0529ec2e9a6c1db22d83ac1;p=thirdparty%2Fkernel%2Fstable-queue.git 5.15-stable patches added patches: documentation-x86-bugs-its-add-its-documentation.patch x86-alternative-optimize-returns-patching.patch x86-alternatives-remove-faulty-optimization.patch x86-its-add-support-for-its-safe-indirect-thunk.patch x86-its-add-support-for-its-safe-return-thunk.patch x86-its-add-vmexit-option-to-skip-mitigation-on-some-cpus.patch x86-its-align-rets-in-bhb-clear-sequence-to-avoid-thunking.patch x86-its-enable-indirect-target-selection-mitigation.patch x86-its-enumerate-indirect-target-selection-its-bug.patch x86-its-fineibt-paranoid-vs-its.patch x86-its-fix-build-errors-when-config_modules-n.patch x86-its-use-dynamic-thunks-for-indirect-branches.patch x86-nospec-simplify-jmp-call-_nospec.patch x86-speculation-add-a-conditional-cs-prefix-to-call_nospec.patch x86-speculation-remove-the-extra-ifdef-around-call_nospec.patch x86-speculation-simplify-and-make-call_nospec-consistent.patch --- diff --git a/queue-5.15/documentation-x86-bugs-its-add-its-documentation.patch b/queue-5.15/documentation-x86-bugs-its-add-its-documentation.patch new file mode 100644 index 0000000000..640d05fbf1 --- /dev/null +++ b/queue-5.15/documentation-x86-bugs-its-add-its-documentation.patch @@ -0,0 +1,190 @@ +From stable+bounces-144648-greg=kroah.com@vger.kernel.org Sat May 17 02:00:58 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:00:46 -0700 +Subject: Documentation: x86/bugs/its: Add ITS documentation +To: stable@vger.kernel.org +Message-ID: <20250516-its-5-15-v3-5-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit 1ac116ce6468670eeda39345a5585df308243dca upstream. + +Add the admin-guide for Indirect Target Selection (ITS). + +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Josh Poimboeuf +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + Documentation/admin-guide/hw-vuln/index.rst | 1 + Documentation/admin-guide/hw-vuln/indirect-target-selection.rst | 156 ++++++++++ + 2 files changed, 157 insertions(+) + +--- a/Documentation/admin-guide/hw-vuln/index.rst ++++ b/Documentation/admin-guide/hw-vuln/index.rst +@@ -22,3 +22,4 @@ are configurable at compile, boot or run + gather_data_sampling.rst + srso + reg-file-data-sampling ++ indirect-target-selection +--- /dev/null ++++ b/Documentation/admin-guide/hw-vuln/indirect-target-selection.rst +@@ -0,0 +1,156 @@ ++.. SPDX-License-Identifier: GPL-2.0 ++ ++Indirect Target Selection (ITS) ++=============================== ++ ++ITS is a vulnerability in some Intel CPUs that support Enhanced IBRS and were ++released before Alder Lake. ITS may allow an attacker to control the prediction ++of indirect branches and RETs located in the lower half of a cacheline. ++ ++ITS is assigned CVE-2024-28956 with a CVSS score of 4.7 (Medium). ++ ++Scope of Impact ++--------------- ++- **eIBRS Guest/Host Isolation**: Indirect branches in KVM/kernel may still be ++ predicted with unintended target corresponding to a branch in the guest. ++ ++- **Intra-Mode BTI**: In-kernel training such as through cBPF or other native ++ gadgets. ++ ++- **Indirect Branch Prediction Barrier (IBPB)**: After an IBPB, indirect ++ branches may still be predicted with targets corresponding to direct branches ++ executed prior to the IBPB. This is fixed by the IPU 2025.1 microcode, which ++ should be available via distro updates. Alternatively microcode can be ++ obtained from Intel's github repository [#f1]_. ++ ++Affected CPUs ++------------- ++Below is the list of ITS affected CPUs [#f2]_ [#f3]_: ++ ++ ======================== ============ ==================== =============== ++ Common name Family_Model eIBRS Intra-mode BTI ++ Guest/Host Isolation ++ ======================== ============ ==================== =============== ++ SKYLAKE_X (step >= 6) 06_55H Affected Affected ++ ICELAKE_X 06_6AH Not affected Affected ++ ICELAKE_D 06_6CH Not affected Affected ++ ICELAKE_L 06_7EH Not affected Affected ++ TIGERLAKE_L 06_8CH Not affected Affected ++ TIGERLAKE 06_8DH Not affected Affected ++ KABYLAKE_L (step >= 12) 06_8EH Affected Affected ++ KABYLAKE (step >= 13) 06_9EH Affected Affected ++ COMETLAKE 06_A5H Affected Affected ++ COMETLAKE_L 06_A6H Affected Affected ++ ROCKETLAKE 06_A7H Not affected Affected ++ ======================== ============ ==================== =============== ++ ++- All affected CPUs enumerate Enhanced IBRS feature. ++- IBPB isolation is affected on all ITS affected CPUs, and need a microcode ++ update for mitigation. ++- None of the affected CPUs enumerate BHI_CTRL which was introduced in Golden ++ Cove (Alder Lake and Sapphire Rapids). This can help guests to determine the ++ host's affected status. ++- Intel Atom CPUs are not affected by ITS. ++ ++Mitigation ++---------- ++As only the indirect branches and RETs that have their last byte of instruction ++in the lower half of the cacheline are vulnerable to ITS, the basic idea behind ++the mitigation is to not allow indirect branches in the lower half. ++ ++This is achieved by relying on existing retpoline support in the kernel, and in ++compilers. ITS-vulnerable retpoline sites are runtime patched to point to newly ++added ITS-safe thunks. These safe thunks consists of indirect branch in the ++second half of the cacheline. Not all retpoline sites are patched to thunks, if ++a retpoline site is evaluated to be ITS-safe, it is replaced with an inline ++indirect branch. ++ ++Dynamic thunks ++~~~~~~~~~~~~~~ ++From a dynamically allocated pool of safe-thunks, each vulnerable site is ++replaced with a new thunk, such that they get a unique address. This could ++improve the branch prediction accuracy. Also, it is a defense-in-depth measure ++against aliasing. ++ ++Note, for simplicity, indirect branches in eBPF programs are always replaced ++with a jump to a static thunk in __x86_indirect_its_thunk_array. If required, ++in future this can be changed to use dynamic thunks. ++ ++All vulnerable RETs are replaced with a static thunk, they do not use dynamic ++thunks. This is because RETs get their prediction from RSB mostly that does not ++depend on source address. RETs that underflow RSB may benefit from dynamic ++thunks. But, RETs significantly outnumber indirect branches, and any benefit ++from a unique source address could be outweighed by the increased icache ++footprint and iTLB pressure. ++ ++Retpoline ++~~~~~~~~~ ++Retpoline sequence also mitigates ITS-unsafe indirect branches. For this ++reason, when retpoline is enabled, ITS mitigation only relocates the RETs to ++safe thunks. Unless user requested the RSB-stuffing mitigation. ++ ++Mitigation in guests ++^^^^^^^^^^^^^^^^^^^^ ++All guests deploy ITS mitigation by default, irrespective of eIBRS enumeration ++and Family/Model of the guest. This is because eIBRS feature could be hidden ++from a guest. One exception to this is when a guest enumerates BHI_DIS_S, which ++indicates that the guest is running on an unaffected host. ++ ++To prevent guests from unnecessarily deploying the mitigation on unaffected ++platforms, Intel has defined ITS_NO bit(62) in MSR IA32_ARCH_CAPABILITIES. When ++a guest sees this bit set, it should not enumerate the ITS bug. Note, this bit ++is not set by any hardware, but is **intended for VMMs to synthesize** it for ++guests as per the host's affected status. ++ ++Mitigation options ++^^^^^^^^^^^^^^^^^^ ++The ITS mitigation can be controlled using the "indirect_target_selection" ++kernel parameter. The available options are: ++ ++ ======== =================================================================== ++ on (default) Deploy the "Aligned branch/return thunks" mitigation. ++ If spectre_v2 mitigation enables retpoline, aligned-thunks are only ++ deployed for the affected RET instructions. Retpoline mitigates ++ indirect branches. ++ ++ off Disable ITS mitigation. ++ ++ vmexit Equivalent to "=on" if the CPU is affected by guest/host isolation ++ part of ITS. Otherwise, mitigation is not deployed. This option is ++ useful when host userspace is not in the threat model, and only ++ attacks from guest to host are considered. ++ ++ force Force the ITS bug and deploy the default mitigation. ++ ======== =================================================================== ++ ++Sysfs reporting ++--------------- ++ ++The sysfs file showing ITS mitigation status is: ++ ++ /sys/devices/system/cpu/vulnerabilities/indirect_target_selection ++ ++Note, microcode mitigation status is not reported in this file. ++ ++The possible values in this file are: ++ ++.. list-table:: ++ ++ * - Not affected ++ - The processor is not vulnerable. ++ * - Vulnerable ++ - System is vulnerable and no mitigation has been applied. ++ * - Vulnerable, KVM: Not affected ++ - System is vulnerable to intra-mode BTI, but not affected by eIBRS ++ guest/host isolation. ++ * - Mitigation: Aligned branch/return thunks ++ - The mitigation is enabled, affected indirect branches and RETs are ++ relocated to safe thunks. ++ ++References ++---------- ++.. [#f1] Microcode repository - https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files ++ ++.. [#f2] Affected Processors list - https://www.intel.com/content/www/us/en/developer/topic-technology/software-security-guidance/processors-affected-consolidated-product-cpu-model.html ++ ++.. [#f3] Affected Processors list (machine readable) - https://github.com/intel/Intel-affected-processor-list diff --git a/queue-5.15/series b/queue-5.15/series index b3166b3d4d..f6c3a9e652 100644 --- a/queue-5.15/series +++ b/queue-5.15/series @@ -11,3 +11,19 @@ net-dsa-sja1105-discard-incoming-frames-in-br_state_.patch alsa-sh-snd_aica-should-depend-on-sh_dma_api.patch qlcnic-fix-memory-leak-in-qlcnic_sriov_channel_cfg_c.patch nfsv4-pnfs-reset-the-layout-state-after-a-layoutretu.patch +x86-nospec-simplify-jmp-call-_nospec.patch +x86-speculation-simplify-and-make-call_nospec-consistent.patch +x86-speculation-add-a-conditional-cs-prefix-to-call_nospec.patch +x86-speculation-remove-the-extra-ifdef-around-call_nospec.patch +documentation-x86-bugs-its-add-its-documentation.patch +x86-its-enumerate-indirect-target-selection-its-bug.patch +x86-its-add-support-for-its-safe-indirect-thunk.patch +x86-alternative-optimize-returns-patching.patch +x86-alternatives-remove-faulty-optimization.patch +x86-its-add-support-for-its-safe-return-thunk.patch +x86-its-enable-indirect-target-selection-mitigation.patch +x86-its-add-vmexit-option-to-skip-mitigation-on-some-cpus.patch +x86-its-align-rets-in-bhb-clear-sequence-to-avoid-thunking.patch +x86-its-use-dynamic-thunks-for-indirect-branches.patch +x86-its-fix-build-errors-when-config_modules-n.patch +x86-its-fineibt-paranoid-vs-its.patch diff --git a/queue-5.15/x86-alternative-optimize-returns-patching.patch b/queue-5.15/x86-alternative-optimize-returns-patching.patch new file mode 100644 index 0000000000..ba2d4cd77c --- /dev/null +++ b/queue-5.15/x86-alternative-optimize-returns-patching.patch @@ -0,0 +1,62 @@ +From stable+bounces-144651-greg=kroah.com@vger.kernel.org Sat May 17 02:01:45 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:01:32 -0700 +Subject: x86/alternative: Optimize returns patching +To: stable@vger.kernel.org +Cc: "Borislav Petkov (AMD)" +Message-ID: <20250516-its-5-15-v3-8-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: "Borislav Petkov (AMD)" + +commit d2408e043e7296017420aa5929b3bba4d5e61013 upstream. + +Instead of decoding each instruction in the return sites range only to +realize that that return site is a jump to the default return thunk +which is needed - X86_FEATURE_RETHUNK is enabled - lift that check +before the loop and get rid of that loop overhead. + +Add comments about what gets patched, while at it. + +Signed-off-by: Borislav Petkov (AMD) +Acked-by: Peter Zijlstra (Intel) +Link: https://lore.kernel.org/r/20230512120952.7924-1-bp@alien8.de +Signed-off-by: Pawan Gupta +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/kernel/alternative.c | 13 ++++++++++--- + 1 file changed, 10 insertions(+), 3 deletions(-) + +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -620,13 +620,12 @@ static int patch_return(void *addr, stru + { + int i = 0; + ++ /* Patch the custom return thunks... */ + if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { +- if (x86_return_thunk == __x86_return_thunk) +- return -1; +- + i = JMP32_INSN_SIZE; + __text_gen_insn(bytes, JMP32_INSN_OPCODE, addr, x86_return_thunk, i); + } else { ++ /* ... or patch them out if not needed. */ + bytes[i++] = RET_INSN_OPCODE; + } + +@@ -639,6 +638,14 @@ void __init_or_module noinline apply_ret + { + s32 *s; + ++ /* ++ * Do not patch out the default return thunks if those needed are the ++ * ones generated by the compiler. ++ */ ++ if (cpu_feature_enabled(X86_FEATURE_RETHUNK) && ++ (x86_return_thunk == __x86_return_thunk)) ++ return; ++ + for (s = start; s < end; s++) { + void *dest = NULL, *addr = (void *)s + *s; + struct insn insn; diff --git a/queue-5.15/x86-alternatives-remove-faulty-optimization.patch b/queue-5.15/x86-alternatives-remove-faulty-optimization.patch new file mode 100644 index 0000000000..d7f8372d8d --- /dev/null +++ b/queue-5.15/x86-alternatives-remove-faulty-optimization.patch @@ -0,0 +1,54 @@ +From stable+bounces-144652-greg=kroah.com@vger.kernel.org Sat May 17 02:01:53 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:01:47 -0700 +Subject: x86/alternatives: Remove faulty optimization +To: stable@vger.kernel.org +Cc: Josh Poimboeuf +Message-ID: <20250516-its-5-15-v3-9-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Josh Poimboeuf + +commit 4ba89dd6ddeca2a733bdaed7c9a5cbe4e19d9124 upstream. + +The following commit + + 095b8303f383 ("x86/alternative: Make custom return thunk unconditional") + +made '__x86_return_thunk' a placeholder value. All code setting +X86_FEATURE_RETHUNK also changes the value of 'x86_return_thunk'. So +the optimization at the beginning of apply_returns() is dead code. + +Also, before the above-mentioned commit, the optimization actually had a +bug It bypassed __static_call_fixup(), causing some raw returns to +remain unpatched in static call trampolines. Thus the 'Fixes' tag. + +Fixes: d2408e043e72 ("x86/alternative: Optimize returns patching") +Signed-off-by: Josh Poimboeuf +Signed-off-by: Ingo Molnar +Signed-off-by: Borislav Petkov (AMD) +Acked-by: Borislav Petkov (AMD) +Link: https://lore.kernel.org/r/16d19d2249d4485d8380fb215ffaae81e6b8119e.1693889988.git.jpoimboe@kernel.org +Signed-off-by: Pawan Gupta +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/kernel/alternative.c | 8 -------- + 1 file changed, 8 deletions(-) + +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -638,14 +638,6 @@ void __init_or_module noinline apply_ret + { + s32 *s; + +- /* +- * Do not patch out the default return thunks if those needed are the +- * ones generated by the compiler. +- */ +- if (cpu_feature_enabled(X86_FEATURE_RETHUNK) && +- (x86_return_thunk == __x86_return_thunk)) +- return; +- + for (s = start; s < end; s++) { + void *dest = NULL, *addr = (void *)s + *s; + struct insn insn; diff --git a/queue-5.15/x86-its-add-support-for-its-safe-indirect-thunk.patch b/queue-5.15/x86-its-add-support-for-its-safe-indirect-thunk.patch new file mode 100644 index 0000000000..431bd65eaa --- /dev/null +++ b/queue-5.15/x86-its-add-support-for-its-safe-indirect-thunk.patch @@ -0,0 +1,249 @@ +From stable+bounces-144650-greg=kroah.com@vger.kernel.org Sat May 17 02:01:24 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:01:17 -0700 +Subject: x86/its: Add support for ITS-safe indirect thunk +To: stable@vger.kernel.org +Cc: Josh Poimboeuf , Alexandre Chartre +Message-ID: <20250516-its-5-15-v3-7-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit 8754e67ad4ac692c67ff1f99c0d07156f04ae40c upstream. + +Due to ITS, indirect branches in the lower half of a cacheline may be +vulnerable to branch target injection attack. + +Introduce ITS-safe thunks to patch indirect branches in the lower half of +cacheline with the thunk. Also thunk any eBPF generated indirect branches +in emit_indirect_jump(). + +Below category of indirect branches are not mitigated: + +- Indirect branches in the .init section are not mitigated because they are + discarded after boot. +- Indirect branches that are explicitly marked retpoline-safe. + +Note that retpoline also mitigates the indirect branches against ITS. This +is because the retpoline sequence fills an RSB entry before RET, and it +does not suffer from RSB-underflow part of the ITS. + +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Josh Poimboeuf +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/Kconfig | 11 +++++ + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/nospec-branch.h | 5 ++ + arch/x86/kernel/alternative.c | 77 +++++++++++++++++++++++++++++++++++ + arch/x86/kernel/vmlinux.lds.S | 6 ++ + arch/x86/lib/retpoline.S | 28 ++++++++++++ + arch/x86/net/bpf_jit_comp.c | 6 ++ + 7 files changed, 133 insertions(+), 1 deletion(-) + +--- a/arch/x86/Kconfig ++++ b/arch/x86/Kconfig +@@ -2517,6 +2517,17 @@ config MITIGATION_SPECTRE_BHI + indirect branches. + See + ++config MITIGATION_ITS ++ bool "Enable Indirect Target Selection mitigation" ++ depends on CPU_SUP_INTEL && X86_64 ++ depends on RETPOLINE && RETHUNK ++ default y ++ help ++ Enable Indirect Target Selection (ITS) mitigation. ITS is a bug in ++ BPU on some Intel CPUs that may allow Spectre V2 style attacks. If ++ disabled, mitigation cannot be enabled via cmdline. ++ See ++ + endif + + config ARCH_HAS_ADD_PAGES +--- a/arch/x86/include/asm/cpufeatures.h ++++ b/arch/x86/include/asm/cpufeatures.h +@@ -433,6 +433,7 @@ + #define X86_FEATURE_BHI_CTRL (21*32+ 2) /* "" BHI_DIS_S HW control available */ + #define X86_FEATURE_CLEAR_BHB_HW (21*32+ 3) /* "" BHI_DIS_S HW control enabled */ + #define X86_FEATURE_CLEAR_BHB_LOOP_ON_VMEXIT (21*32+ 4) /* "" Clear branch history at vmexit using SW loop */ ++#define X86_FEATURE_INDIRECT_THUNK_ITS (21*32 + 5) /* "" Use thunk for indirect branches in lower half of cacheline */ + + /* + * BUG word(s) +--- a/arch/x86/include/asm/nospec-branch.h ++++ b/arch/x86/include/asm/nospec-branch.h +@@ -271,6 +271,11 @@ extern void (*x86_return_thunk)(void); + + typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; + ++#define ITS_THUNK_SIZE 64 ++typedef u8 its_thunk_t[ITS_THUNK_SIZE]; ++ ++extern its_thunk_t __x86_indirect_its_thunk_array[]; ++ + #define GEN(reg) \ + extern retpoline_thunk_t __x86_indirect_thunk_ ## reg; + #include +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -395,6 +395,74 @@ static int emit_indirect(int op, int reg + return i; + } + ++#ifdef CONFIG_MITIGATION_ITS ++ ++static int __emit_trampoline(void *addr, struct insn *insn, u8 *bytes, ++ void *call_dest, void *jmp_dest) ++{ ++ u8 op = insn->opcode.bytes[0]; ++ int i = 0; ++ ++ /* ++ * Clang does 'weird' Jcc __x86_indirect_thunk_r11 conditional ++ * tail-calls. Deal with them. ++ */ ++ if (is_jcc32(insn)) { ++ bytes[i++] = op; ++ op = insn->opcode.bytes[1]; ++ goto clang_jcc; ++ } ++ ++ if (insn->length == 6) ++ bytes[i++] = 0x2e; /* CS-prefix */ ++ ++ switch (op) { ++ case CALL_INSN_OPCODE: ++ __text_gen_insn(bytes+i, op, addr+i, ++ call_dest, ++ CALL_INSN_SIZE); ++ i += CALL_INSN_SIZE; ++ break; ++ ++ case JMP32_INSN_OPCODE: ++clang_jcc: ++ __text_gen_insn(bytes+i, op, addr+i, ++ jmp_dest, ++ JMP32_INSN_SIZE); ++ i += JMP32_INSN_SIZE; ++ break; ++ ++ default: ++ WARN(1, "%pS %px %*ph\n", addr, addr, 6, addr); ++ return -1; ++ } ++ ++ WARN_ON_ONCE(i != insn->length); ++ ++ return i; ++} ++ ++static int emit_its_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes) ++{ ++ return __emit_trampoline(addr, insn, bytes, ++ __x86_indirect_its_thunk_array[reg], ++ __x86_indirect_its_thunk_array[reg]); ++} ++ ++/* Check if an indirect branch is at ITS-unsafe address */ ++static bool cpu_wants_indirect_its_thunk_at(unsigned long addr, int reg) ++{ ++ if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) ++ return false; ++ ++ /* Indirect branch opcode is 2 or 3 bytes depending on reg */ ++ addr += 1 + reg / 8; ++ ++ /* Lower-half of the cacheline? */ ++ return !(addr & 0x20); ++} ++#endif ++ + /* + * Rewrite the compiler generated retpoline thunk calls. + * +@@ -466,6 +534,15 @@ static int patch_retpoline(void *addr, s + bytes[i++] = 0xe8; /* LFENCE */ + } + ++#ifdef CONFIG_MITIGATION_ITS ++ /* ++ * Check if the address of last byte of emitted-indirect is in ++ * lower-half of the cacheline. Such branches need ITS mitigation. ++ */ ++ if (cpu_wants_indirect_its_thunk_at((unsigned long)addr + i, reg)) ++ return emit_its_trampoline(addr, insn, reg, bytes); ++#endif ++ + ret = emit_indirect(op, reg, bytes + i); + if (ret < 0) + return ret; +--- a/arch/x86/kernel/vmlinux.lds.S ++++ b/arch/x86/kernel/vmlinux.lds.S +@@ -532,6 +532,12 @@ INIT_PER_CPU(irq_stack_backing_store); + "SRSO function pair won't alias"); + #endif + ++#if defined(CONFIG_MITIGATION_ITS) && !defined(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B) ++. = ASSERT(__x86_indirect_its_thunk_rax & 0x20, "__x86_indirect_thunk_rax not in second half of cacheline"); ++. = ASSERT(((__x86_indirect_its_thunk_rcx - __x86_indirect_its_thunk_rax) % 64) == 0, "Indirect thunks are not cacheline apart"); ++. = ASSERT(__x86_indirect_its_thunk_array == __x86_indirect_its_thunk_rax, "Gap in ITS thunk array"); ++#endif ++ + #endif /* CONFIG_X86_64 */ + + #ifdef CONFIG_KEXEC_CORE +--- a/arch/x86/lib/retpoline.S ++++ b/arch/x86/lib/retpoline.S +@@ -254,6 +254,34 @@ SYM_FUNC_START(entry_untrain_ret) + SYM_FUNC_END(entry_untrain_ret) + __EXPORT_THUNK(entry_untrain_ret) + ++#ifdef CONFIG_MITIGATION_ITS ++ ++.macro ITS_THUNK reg ++ ++SYM_INNER_LABEL(__x86_indirect_its_thunk_\reg, SYM_L_GLOBAL) ++ UNWIND_HINT_EMPTY ++ ANNOTATE_NOENDBR ++ ANNOTATE_RETPOLINE_SAFE ++ jmp *%\reg ++ int3 ++ .align 32, 0xcc /* fill to the end of the line */ ++ .skip 32, 0xcc /* skip to the next upper half */ ++.endm ++ ++/* ITS mitigation requires thunks be aligned to upper half of cacheline */ ++.align 64, 0xcc ++.skip 32, 0xcc ++SYM_CODE_START(__x86_indirect_its_thunk_array) ++ ++#define GEN(reg) ITS_THUNK reg ++#include ++#undef GEN ++ ++ .align 64, 0xcc ++SYM_CODE_END(__x86_indirect_its_thunk_array) ++ ++#endif ++ + SYM_CODE_START(__x86_return_thunk) + UNWIND_HINT_FUNC + ANNOTATE_NOENDBR +--- a/arch/x86/net/bpf_jit_comp.c ++++ b/arch/x86/net/bpf_jit_comp.c +@@ -446,7 +446,11 @@ static void emit_indirect_jump(u8 **ppro + u8 *prog = *pprog; + + #ifdef CONFIG_RETPOLINE +- if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { ++ if (IS_ENABLED(CONFIG_MITIGATION_ITS) && ++ cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) { ++ OPTIMIZER_HIDE_VAR(reg); ++ emit_jump(&prog, &__x86_indirect_its_thunk_array[reg], ip); ++ } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { + EMIT_LFENCE(); + EMIT2(0xFF, 0xE0 + reg); + } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE)) { diff --git a/queue-5.15/x86-its-add-support-for-its-safe-return-thunk.patch b/queue-5.15/x86-its-add-support-for-its-safe-return-thunk.patch new file mode 100644 index 0000000000..533872e1e6 --- /dev/null +++ b/queue-5.15/x86-its-add-support-for-its-safe-return-thunk.patch @@ -0,0 +1,179 @@ +From stable+bounces-144653-greg=kroah.com@vger.kernel.org Sat May 17 02:02:12 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:02:03 -0700 +Subject: x86/its: Add support for ITS-safe return thunk +To: stable@vger.kernel.org +Cc: Josh Poimboeuf , Alexandre Chartre +Message-ID: <20250516-its-5-15-v3-10-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit a75bf27fe41abe658c53276a0c486c4bf9adecfc upstream. + +RETs in the lower half of cacheline may be affected by ITS bug, +specifically when the RSB-underflows. Use ITS-safe return thunk for such +RETs. + +RETs that are not patched: + +- RET in retpoline sequence does not need to be patched, because the + sequence itself fills an RSB before RET. +- RETs in .init section are not reachable after init. +- RETs that are explicitly marked safe with ANNOTATE_UNRET_SAFE. + +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Josh Poimboeuf +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/alternative.h | 14 ++++++++++++++ + arch/x86/include/asm/nospec-branch.h | 6 ++++++ + arch/x86/kernel/alternative.c | 17 ++++++++++++++++- + arch/x86/kernel/ftrace.c | 2 +- + arch/x86/kernel/static_call.c | 2 +- + arch/x86/kernel/vmlinux.lds.S | 4 ++++ + arch/x86/lib/retpoline.S | 13 ++++++++++++- + arch/x86/net/bpf_jit_comp.c | 2 +- + 8 files changed, 55 insertions(+), 5 deletions(-) + +--- a/arch/x86/include/asm/alternative.h ++++ b/arch/x86/include/asm/alternative.h +@@ -80,6 +80,20 @@ extern void apply_returns(s32 *start, s3 + + struct module; + ++#ifdef CONFIG_RETHUNK ++extern bool cpu_wants_rethunk(void); ++extern bool cpu_wants_rethunk_at(void *addr); ++#else ++static __always_inline bool cpu_wants_rethunk(void) ++{ ++ return false; ++} ++static __always_inline bool cpu_wants_rethunk_at(void *addr) ++{ ++ return false; ++} ++#endif ++ + #ifdef CONFIG_SMP + extern void alternatives_smp_module_add(struct module *mod, char *name, + void *locks, void *locks_end, +--- a/arch/x86/include/asm/nospec-branch.h ++++ b/arch/x86/include/asm/nospec-branch.h +@@ -250,6 +250,12 @@ extern void __x86_return_thunk(void); + static inline void __x86_return_thunk(void) {} + #endif + ++#ifdef CONFIG_MITIGATION_ITS ++extern void its_return_thunk(void); ++#else ++static inline void its_return_thunk(void) {} ++#endif ++ + extern void retbleed_return_thunk(void); + extern void srso_return_thunk(void); + extern void srso_alias_return_thunk(void); +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -605,6 +605,21 @@ void __init_or_module noinline apply_ret + + #ifdef CONFIG_RETHUNK + ++bool cpu_wants_rethunk(void) ++{ ++ return cpu_feature_enabled(X86_FEATURE_RETHUNK); ++} ++ ++bool cpu_wants_rethunk_at(void *addr) ++{ ++ if (!cpu_feature_enabled(X86_FEATURE_RETHUNK)) ++ return false; ++ if (x86_return_thunk != its_return_thunk) ++ return true; ++ ++ return !((unsigned long)addr & 0x20); ++} ++ + /* + * Rewrite the compiler generated return thunk tail-calls. + * +@@ -621,7 +636,7 @@ static int patch_return(void *addr, stru + int i = 0; + + /* Patch the custom return thunks... */ +- if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { ++ if (cpu_wants_rethunk_at(addr)) { + i = JMP32_INSN_SIZE; + __text_gen_insn(bytes, JMP32_INSN_OPCODE, addr, x86_return_thunk, i); + } else { +--- a/arch/x86/kernel/ftrace.c ++++ b/arch/x86/kernel/ftrace.c +@@ -367,7 +367,7 @@ create_trampoline(struct ftrace_ops *ops + goto fail; + + ip = trampoline + size; +- if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) ++ if (cpu_wants_rethunk_at(ip)) + __text_gen_insn(ip, JMP32_INSN_OPCODE, ip, x86_return_thunk, JMP32_INSN_SIZE); + else + memcpy(ip, retq, sizeof(retq)); +--- a/arch/x86/kernel/static_call.c ++++ b/arch/x86/kernel/static_call.c +@@ -81,7 +81,7 @@ static void __ref __static_call_transfor + break; + + case RET: +- if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) ++ if (cpu_wants_rethunk_at(insn)) + code = text_gen_insn(JMP32_INSN_OPCODE, insn, x86_return_thunk); + else + code = &retinsn; +--- a/arch/x86/kernel/vmlinux.lds.S ++++ b/arch/x86/kernel/vmlinux.lds.S +@@ -538,6 +538,10 @@ INIT_PER_CPU(irq_stack_backing_store); + . = ASSERT(__x86_indirect_its_thunk_array == __x86_indirect_its_thunk_rax, "Gap in ITS thunk array"); + #endif + ++#if defined(CONFIG_MITIGATION_ITS) && !defined(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B) ++. = ASSERT(its_return_thunk & 0x20, "its_return_thunk not in second half of cacheline"); ++#endif ++ + #endif /* CONFIG_X86_64 */ + + #ifdef CONFIG_KEXEC_CORE +--- a/arch/x86/lib/retpoline.S ++++ b/arch/x86/lib/retpoline.S +@@ -280,7 +280,18 @@ SYM_CODE_START(__x86_indirect_its_thunk_ + .align 64, 0xcc + SYM_CODE_END(__x86_indirect_its_thunk_array) + +-#endif ++.align 64, 0xcc ++.skip 32, 0xcc ++SYM_CODE_START(its_return_thunk) ++ UNWIND_HINT_FUNC ++ ANNOTATE_NOENDBR ++ ANNOTATE_UNRET_SAFE ++ ret ++ int3 ++SYM_CODE_END(its_return_thunk) ++EXPORT_SYMBOL(its_return_thunk) ++ ++#endif /* CONFIG_MITIGATION_ITS */ + + SYM_CODE_START(__x86_return_thunk) + UNWIND_HINT_FUNC +--- a/arch/x86/net/bpf_jit_comp.c ++++ b/arch/x86/net/bpf_jit_comp.c +@@ -466,7 +466,7 @@ static void emit_return(u8 **pprog, u8 * + { + u8 *prog = *pprog; + +- if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) { ++ if (cpu_wants_rethunk()) { + emit_jump(&prog, x86_return_thunk, ip); + } else { + EMIT1(0xC3); /* ret */ diff --git a/queue-5.15/x86-its-add-vmexit-option-to-skip-mitigation-on-some-cpus.patch b/queue-5.15/x86-its-add-vmexit-option-to-skip-mitigation-on-some-cpus.patch new file mode 100644 index 0000000000..caebeb53f5 --- /dev/null +++ b/queue-5.15/x86-its-add-vmexit-option-to-skip-mitigation-on-some-cpus.patch @@ -0,0 +1,143 @@ +From stable+bounces-144655-greg=kroah.com@vger.kernel.org Sat May 17 02:02:42 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:02:35 -0700 +Subject: x86/its: Add "vmexit" option to skip mitigation on some CPUs +To: stable@vger.kernel.org +Cc: Josh Poimboeuf , Alexandre Chartre +Message-ID: <20250516-its-5-15-v3-12-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit 2665281a07e19550944e8354a2024635a7b2714a upstream. + +Ice Lake generation CPUs are not affected by guest/host isolation part of +ITS. If a user is only concerned about KVM guests, they can now choose a +new cmdline option "vmexit" that will not deploy the ITS mitigation when +CPU is not affected by guest/host isolation. This saves the performance +overhead of ITS mitigation on Ice Lake gen CPUs. + +When "vmexit" option selected, if the CPU is affected by ITS guest/host +isolation, the default ITS mitigation is deployed. + +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Josh Poimboeuf +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + Documentation/admin-guide/kernel-parameters.txt | 2 ++ + arch/x86/include/asm/cpufeatures.h | 1 + + arch/x86/kernel/cpu/bugs.c | 11 +++++++++++ + arch/x86/kernel/cpu/common.c | 19 ++++++++++++------- + 4 files changed, 26 insertions(+), 7 deletions(-) + +--- a/Documentation/admin-guide/kernel-parameters.txt ++++ b/Documentation/admin-guide/kernel-parameters.txt +@@ -1934,6 +1934,8 @@ + off: Disable mitigation. + force: Force the ITS bug and deploy default + mitigation. ++ vmexit: Only deploy mitigation if CPU is affected by ++ guest/host isolation part of ITS. + + For details see: + Documentation/admin-guide/hw-vuln/indirect-target-selection.rst +--- a/arch/x86/include/asm/cpufeatures.h ++++ b/arch/x86/include/asm/cpufeatures.h +@@ -485,4 +485,5 @@ + #define X86_BUG_BHI X86_BUG(1*32 + 3) /* CPU is affected by Branch History Injection */ + #define X86_BUG_IBPB_NO_RET X86_BUG(1*32 + 4) /* "ibpb_no_ret" IBPB omits return target predictions */ + #define X86_BUG_ITS X86_BUG(1*32 + 5) /* CPU is affected by Indirect Target Selection */ ++#define X86_BUG_ITS_NATIVE_ONLY X86_BUG(1*32 + 6) /* CPU is affected by ITS, VMX is not affected */ + #endif /* _ASM_X86_CPUFEATURES_H */ +--- a/arch/x86/kernel/cpu/bugs.c ++++ b/arch/x86/kernel/cpu/bugs.c +@@ -1158,15 +1158,18 @@ do_cmd_auto: + enum its_mitigation_cmd { + ITS_CMD_OFF, + ITS_CMD_ON, ++ ITS_CMD_VMEXIT, + }; + + enum its_mitigation { + ITS_MITIGATION_OFF, ++ ITS_MITIGATION_VMEXIT_ONLY, + ITS_MITIGATION_ALIGNED_THUNKS, + }; + + static const char * const its_strings[] = { + [ITS_MITIGATION_OFF] = "Vulnerable", ++ [ITS_MITIGATION_VMEXIT_ONLY] = "Mitigation: Vulnerable, KVM: Not affected", + [ITS_MITIGATION_ALIGNED_THUNKS] = "Mitigation: Aligned branch/return thunks", + }; + +@@ -1192,6 +1195,8 @@ static int __init its_parse_cmdline(char + } else if (!strcmp(str, "force")) { + its_cmd = ITS_CMD_ON; + setup_force_cpu_bug(X86_BUG_ITS); ++ } else if (!strcmp(str, "vmexit")) { ++ its_cmd = ITS_CMD_VMEXIT; + } else { + pr_err("Ignoring unknown indirect_target_selection option (%s).", str); + } +@@ -1239,6 +1244,12 @@ static void __init its_select_mitigation + case ITS_CMD_OFF: + its_mitigation = ITS_MITIGATION_OFF; + break; ++ case ITS_CMD_VMEXIT: ++ if (boot_cpu_has_bug(X86_BUG_ITS_NATIVE_ONLY)) { ++ its_mitigation = ITS_MITIGATION_VMEXIT_ONLY; ++ goto out; ++ } ++ fallthrough; + case ITS_CMD_ON: + its_mitigation = ITS_MITIGATION_ALIGNED_THUNKS; + if (!boot_cpu_has(X86_FEATURE_RETPOLINE)) +--- a/arch/x86/kernel/cpu/common.c ++++ b/arch/x86/kernel/cpu/common.c +@@ -1143,6 +1143,8 @@ static const __initconst struct x86_cpu_ + #define RFDS BIT(7) + /* CPU is affected by Indirect Target Selection */ + #define ITS BIT(8) ++/* CPU is affected by Indirect Target Selection, but guest-host isolation is not affected */ ++#define ITS_NATIVE_ONLY BIT(9) + + static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst = { + VULNBL_INTEL_STEPPINGS(IVYBRIDGE, X86_STEPPING_ANY, SRBDS), +@@ -1163,16 +1165,16 @@ static const struct x86_cpu_id cpu_vuln_ + VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x0, 0xc), MMIO | RETBLEED | GDS | SRBDS), + VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS | ITS), + VULNBL_INTEL_STEPPINGS(CANNONLAKE_L, X86_STEPPING_ANY, RETBLEED), +- VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS), +- VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPING_ANY, MMIO | GDS | ITS), +- VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPING_ANY, MMIO | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS | ITS_NATIVE_ONLY), ++ VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPING_ANY, MMIO | GDS | ITS | ITS_NATIVE_ONLY), ++ VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPING_ANY, MMIO | GDS | ITS | ITS_NATIVE_ONLY), + VULNBL_INTEL_STEPPINGS(COMETLAKE, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS), + VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x0, 0x0), MMIO | RETBLEED | ITS), + VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS), +- VULNBL_INTEL_STEPPINGS(TIGERLAKE_L, X86_STEPPING_ANY, GDS | ITS), +- VULNBL_INTEL_STEPPINGS(TIGERLAKE, X86_STEPPING_ANY, GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(TIGERLAKE_L, X86_STEPPING_ANY, GDS | ITS | ITS_NATIVE_ONLY), ++ VULNBL_INTEL_STEPPINGS(TIGERLAKE, X86_STEPPING_ANY, GDS | ITS | ITS_NATIVE_ONLY), + VULNBL_INTEL_STEPPINGS(LAKEFIELD, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED), +- VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | ITS | ITS_NATIVE_ONLY), + VULNBL_INTEL_STEPPINGS(ALDERLAKE, X86_STEPPING_ANY, RFDS), + VULNBL_INTEL_STEPPINGS(ALDERLAKE_L, X86_STEPPING_ANY, RFDS), + VULNBL_INTEL_STEPPINGS(RAPTORLAKE, X86_STEPPING_ANY, RFDS), +@@ -1389,8 +1391,11 @@ static void __init cpu_set_bug_bits(stru + if (cpu_has(c, X86_FEATURE_AMD_IBPB) && !cpu_has(c, X86_FEATURE_AMD_IBPB_RET)) + setup_force_cpu_bug(X86_BUG_IBPB_NO_RET); + +- if (vulnerable_to_its(x86_arch_cap_msr)) ++ if (vulnerable_to_its(x86_arch_cap_msr)) { + setup_force_cpu_bug(X86_BUG_ITS); ++ if (cpu_matches(cpu_vuln_blacklist, ITS_NATIVE_ONLY)) ++ setup_force_cpu_bug(X86_BUG_ITS_NATIVE_ONLY); ++ } + + if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN)) + return; diff --git a/queue-5.15/x86-its-align-rets-in-bhb-clear-sequence-to-avoid-thunking.patch b/queue-5.15/x86-its-align-rets-in-bhb-clear-sequence-to-avoid-thunking.patch new file mode 100644 index 0000000000..c0f205dd1a --- /dev/null +++ b/queue-5.15/x86-its-align-rets-in-bhb-clear-sequence-to-avoid-thunking.patch @@ -0,0 +1,104 @@ +From stable+bounces-144656-greg=kroah.com@vger.kernel.org Sat May 17 02:02:59 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:02:50 -0700 +Subject: x86/its: Align RETs in BHB clear sequence to avoid thunking +To: stable@vger.kernel.org +Cc: Andrew Cooper , Alexandre Chartre +Message-ID: <20250516-its-5-15-v3-13-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit f0cd7091cc5a032c8870b4285305d9172569d126 upstream. + +The software mitigation for BHI is to execute BHB clear sequence at syscall +entry, and possibly after a cBPF program. ITS mitigation thunks RETs in the +lower half of the cacheline. This causes the RETs in the BHB clear sequence +to be thunked as well, adding unnecessary branches to the BHB clear +sequence. + +Since the sequence is in hot path, align the RET instructions in the +sequence to avoid thunking. + +This is how disassembly clear_bhb_loop() looks like after this change: + + 0x44 <+4>: mov $0x5,%ecx + 0x49 <+9>: call 0xffffffff81001d9b + 0x4e <+14>: jmp 0xffffffff81001de5 + 0x53 <+19>: int3 + ... + 0x9b <+91>: call 0xffffffff81001dce + 0xa0 <+96>: ret + 0xa1 <+97>: int3 + ... + 0xce <+142>: mov $0x5,%eax + 0xd3 <+147>: jmp 0xffffffff81001dd6 + 0xd5 <+149>: nop + 0xd6 <+150>: sub $0x1,%eax + 0xd9 <+153>: jne 0xffffffff81001dd3 + 0xdb <+155>: sub $0x1,%ecx + 0xde <+158>: jne 0xffffffff81001d9b + 0xe0 <+160>: ret + 0xe1 <+161>: int3 + 0xe2 <+162>: int3 + 0xe3 <+163>: int3 + 0xe4 <+164>: int3 + 0xe5 <+165>: lfence + 0xe8 <+168>: pop %rbp + 0xe9 <+169>: ret + +Suggested-by: Andrew Cooper +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/entry/entry_64.S | 20 +++++++++++++++++--- + 1 file changed, 17 insertions(+), 3 deletions(-) + +--- a/arch/x86/entry/entry_64.S ++++ b/arch/x86/entry/entry_64.S +@@ -1530,7 +1530,9 @@ SYM_CODE_END(rewind_stack_and_make_dead) + * ORC to unwind properly. + * + * The alignment is for performance and not for safety, and may be safely +- * refactored in the future if needed. ++ * refactored in the future if needed. The .skips are for safety, to ensure ++ * that all RETs are in the second half of a cacheline to mitigate Indirect ++ * Target Selection, rather than taking the slowpath via its_return_thunk. + */ + SYM_FUNC_START(clear_bhb_loop) + push %rbp +@@ -1540,10 +1542,22 @@ SYM_FUNC_START(clear_bhb_loop) + call 1f + jmp 5f + .align 64, 0xcc ++ /* ++ * Shift instructions so that the RET is in the upper half of the ++ * cacheline and don't take the slowpath to its_return_thunk. ++ */ ++ .skip 32 - (.Lret1 - 1f), 0xcc + ANNOTATE_INTRA_FUNCTION_CALL + 1: call 2f +- RET ++.Lret1: RET + .align 64, 0xcc ++ /* ++ * As above shift instructions for RET at .Lret2 as well. ++ * ++ * This should be ideally be: .skip 32 - (.Lret2 - 2f), 0xcc ++ * but some Clang versions (e.g. 18) don't like this. ++ */ ++ .skip 32 - 18, 0xcc + 2: movl $5, %eax + 3: jmp 4f + nop +@@ -1551,7 +1565,7 @@ SYM_FUNC_START(clear_bhb_loop) + jnz 3b + sub $1, %ecx + jnz 1b +- RET ++.Lret2: RET + 5: lfence + pop %rbp + RET diff --git a/queue-5.15/x86-its-enable-indirect-target-selection-mitigation.patch b/queue-5.15/x86-its-enable-indirect-target-selection-mitigation.patch new file mode 100644 index 0000000000..ae765a4101 --- /dev/null +++ b/queue-5.15/x86-its-enable-indirect-target-selection-mitigation.patch @@ -0,0 +1,343 @@ +From stable+bounces-144654-greg=kroah.com@vger.kernel.org Sat May 17 02:02:26 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:02:19 -0700 +Subject: x86/its: Enable Indirect Target Selection mitigation +To: stable@vger.kernel.org +Cc: Josh Poimboeuf , Alexandre Chartre +Message-ID: <20250516-its-5-15-v3-11-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit f4818881c47fd91fcb6d62373c57c7844e3de1c0 upstream. + +Indirect Target Selection (ITS) is a bug in some pre-ADL Intel CPUs with +eIBRS. It affects prediction of indirect branch and RETs in the +lower half of cacheline. Due to ITS such branches may get wrongly predicted +to a target of (direct or indirect) branch that is located in the upper +half of the cacheline. + +Scope of impact +=============== + +Guest/host isolation +-------------------- +When eIBRS is used for guest/host isolation, the indirect branches in the +VMM may still be predicted with targets corresponding to branches in the +guest. + +Intra-mode +---------- +cBPF or other native gadgets can be used for intra-mode training and +disclosure using ITS. + +User/kernel isolation +--------------------- +When eIBRS is enabled user/kernel isolation is not impacted. + +Indirect Branch Prediction Barrier (IBPB) +----------------------------------------- +After an IBPB, indirect branches may be predicted with targets +corresponding to direct branches which were executed prior to IBPB. This is +mitigated by a microcode update. + +Add cmdline parameter indirect_target_selection=off|on|force to control the +mitigation to relocate the affected branches to an ITS-safe thunk i.e. +located in the upper half of cacheline. Also add the sysfs reporting. + +When retpoline mitigation is deployed, ITS safe-thunks are not needed, +because retpoline sequence is already ITS-safe. Similarly, when call depth +tracking (CDT) mitigation is deployed (retbleed=stuff), ITS safe return +thunk is not used, as CDT prevents RSB-underflow. + +To not overcomplicate things, ITS mitigation is not supported with +spectre-v2 lfence;jmp mitigation. Moreover, it is less practical to deploy +lfence;jmp mitigation on ITS affected parts anyways. + +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Josh Poimboeuf +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + Documentation/ABI/testing/sysfs-devices-system-cpu | 1 + Documentation/admin-guide/kernel-parameters.txt | 13 ++ + arch/x86/kernel/cpu/bugs.c | 128 ++++++++++++++++++++- + drivers/base/cpu.c | 8 + + include/linux/cpu.h | 2 + 5 files changed, 149 insertions(+), 3 deletions(-) + +--- a/Documentation/ABI/testing/sysfs-devices-system-cpu ++++ b/Documentation/ABI/testing/sysfs-devices-system-cpu +@@ -512,6 +512,7 @@ Description: information about CPUs hete + + What: /sys/devices/system/cpu/vulnerabilities + /sys/devices/system/cpu/vulnerabilities/gather_data_sampling ++ /sys/devices/system/cpu/vulnerabilities/indirect_target_selection + /sys/devices/system/cpu/vulnerabilities/itlb_multihit + /sys/devices/system/cpu/vulnerabilities/l1tf + /sys/devices/system/cpu/vulnerabilities/mds +--- a/Documentation/admin-guide/kernel-parameters.txt ++++ b/Documentation/admin-guide/kernel-parameters.txt +@@ -1926,6 +1926,18 @@ + different crypto accelerators. This option can be used + to achieve best performance for particular HW. + ++ indirect_target_selection= [X86,Intel] Mitigation control for Indirect ++ Target Selection(ITS) bug in Intel CPUs. Updated ++ microcode is also required for a fix in IBPB. ++ ++ on: Enable mitigation (default). ++ off: Disable mitigation. ++ force: Force the ITS bug and deploy default ++ mitigation. ++ ++ For details see: ++ Documentation/admin-guide/hw-vuln/indirect-target-selection.rst ++ + init= [KNL] + Format: + Run specified binary instead of /sbin/init as init +@@ -3073,6 +3085,7 @@ + improves system performance, but it may also + expose users to several CPU vulnerabilities. + Equivalent to: gather_data_sampling=off [X86] ++ indirect_target_selection=off [X86] + kpti=0 [ARM64] + kvm.nx_huge_pages=off [X86] + l1tf=off [X86] +--- a/arch/x86/kernel/cpu/bugs.c ++++ b/arch/x86/kernel/cpu/bugs.c +@@ -48,6 +48,7 @@ static void __init srbds_select_mitigati + static void __init l1d_flush_select_mitigation(void); + static void __init gds_select_mitigation(void); + static void __init srso_select_mitigation(void); ++static void __init its_select_mitigation(void); + + /* The base value of the SPEC_CTRL MSR without task-specific bits set */ + u64 x86_spec_ctrl_base; +@@ -66,6 +67,14 @@ static DEFINE_MUTEX(spec_ctrl_mutex); + + void (*x86_return_thunk)(void) __ro_after_init = &__x86_return_thunk; + ++static void __init set_return_thunk(void *thunk) ++{ ++ if (x86_return_thunk != __x86_return_thunk) ++ pr_warn("x86/bugs: return thunk changed\n"); ++ ++ x86_return_thunk = thunk; ++} ++ + /* Update SPEC_CTRL MSR and its cached copy unconditionally */ + static void update_spec_ctrl(u64 val) + { +@@ -174,6 +183,7 @@ void __init cpu_select_mitigations(void) + */ + srso_select_mitigation(); + gds_select_mitigation(); ++ its_select_mitigation(); + } + + /* +@@ -1081,7 +1091,7 @@ do_cmd_auto: + setup_force_cpu_cap(X86_FEATURE_UNRET); + + if (IS_ENABLED(CONFIG_RETHUNK)) +- x86_return_thunk = retbleed_return_thunk; ++ set_return_thunk(retbleed_return_thunk); + + if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD && + boot_cpu_data.x86_vendor != X86_VENDOR_HYGON) +@@ -1143,6 +1153,105 @@ do_cmd_auto: + } + + #undef pr_fmt ++#define pr_fmt(fmt) "ITS: " fmt ++ ++enum its_mitigation_cmd { ++ ITS_CMD_OFF, ++ ITS_CMD_ON, ++}; ++ ++enum its_mitigation { ++ ITS_MITIGATION_OFF, ++ ITS_MITIGATION_ALIGNED_THUNKS, ++}; ++ ++static const char * const its_strings[] = { ++ [ITS_MITIGATION_OFF] = "Vulnerable", ++ [ITS_MITIGATION_ALIGNED_THUNKS] = "Mitigation: Aligned branch/return thunks", ++}; ++ ++static enum its_mitigation its_mitigation __ro_after_init = ITS_MITIGATION_ALIGNED_THUNKS; ++ ++static enum its_mitigation_cmd its_cmd __ro_after_init = ++ IS_ENABLED(CONFIG_MITIGATION_ITS) ? ITS_CMD_ON : ITS_CMD_OFF; ++ ++static int __init its_parse_cmdline(char *str) ++{ ++ if (!str) ++ return -EINVAL; ++ ++ if (!IS_ENABLED(CONFIG_MITIGATION_ITS)) { ++ pr_err("Mitigation disabled at compile time, ignoring option (%s)", str); ++ return 0; ++ } ++ ++ if (!strcmp(str, "off")) { ++ its_cmd = ITS_CMD_OFF; ++ } else if (!strcmp(str, "on")) { ++ its_cmd = ITS_CMD_ON; ++ } else if (!strcmp(str, "force")) { ++ its_cmd = ITS_CMD_ON; ++ setup_force_cpu_bug(X86_BUG_ITS); ++ } else { ++ pr_err("Ignoring unknown indirect_target_selection option (%s).", str); ++ } ++ ++ return 0; ++} ++early_param("indirect_target_selection", its_parse_cmdline); ++ ++static void __init its_select_mitigation(void) ++{ ++ enum its_mitigation_cmd cmd = its_cmd; ++ ++ if (!boot_cpu_has_bug(X86_BUG_ITS) || cpu_mitigations_off()) { ++ its_mitigation = ITS_MITIGATION_OFF; ++ return; ++ } ++ ++ /* Exit early to avoid irrelevant warnings */ ++ if (cmd == ITS_CMD_OFF) { ++ its_mitigation = ITS_MITIGATION_OFF; ++ goto out; ++ } ++ if (spectre_v2_enabled == SPECTRE_V2_NONE) { ++ pr_err("WARNING: Spectre-v2 mitigation is off, disabling ITS\n"); ++ its_mitigation = ITS_MITIGATION_OFF; ++ goto out; ++ } ++ if (!IS_ENABLED(CONFIG_RETPOLINE) || !IS_ENABLED(CONFIG_RETHUNK)) { ++ pr_err("WARNING: ITS mitigation depends on retpoline and rethunk support\n"); ++ its_mitigation = ITS_MITIGATION_OFF; ++ goto out; ++ } ++ if (IS_ENABLED(CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B)) { ++ pr_err("WARNING: ITS mitigation is not compatible with CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_64B\n"); ++ its_mitigation = ITS_MITIGATION_OFF; ++ goto out; ++ } ++ if (boot_cpu_has(X86_FEATURE_RETPOLINE_LFENCE)) { ++ pr_err("WARNING: ITS mitigation is not compatible with lfence mitigation\n"); ++ its_mitigation = ITS_MITIGATION_OFF; ++ goto out; ++ } ++ ++ switch (cmd) { ++ case ITS_CMD_OFF: ++ its_mitigation = ITS_MITIGATION_OFF; ++ break; ++ case ITS_CMD_ON: ++ its_mitigation = ITS_MITIGATION_ALIGNED_THUNKS; ++ if (!boot_cpu_has(X86_FEATURE_RETPOLINE)) ++ setup_force_cpu_cap(X86_FEATURE_INDIRECT_THUNK_ITS); ++ setup_force_cpu_cap(X86_FEATURE_RETHUNK); ++ set_return_thunk(its_return_thunk); ++ break; ++ } ++out: ++ pr_info("%s\n", its_strings[its_mitigation]); ++} ++ ++#undef pr_fmt + #define pr_fmt(fmt) "Spectre V2 : " fmt + + static enum spectre_v2_user_mitigation spectre_v2_user_stibp __ro_after_init = +@@ -2592,10 +2701,10 @@ static void __init srso_select_mitigatio + + if (boot_cpu_data.x86 == 0x19) { + setup_force_cpu_cap(X86_FEATURE_SRSO_ALIAS); +- x86_return_thunk = srso_alias_return_thunk; ++ set_return_thunk(srso_alias_return_thunk); + } else { + setup_force_cpu_cap(X86_FEATURE_SRSO); +- x86_return_thunk = srso_return_thunk; ++ set_return_thunk(srso_return_thunk); + } + srso_mitigation = SRSO_MITIGATION_SAFE_RET; + } else { +@@ -2775,6 +2884,11 @@ static ssize_t rfds_show_state(char *buf + return sysfs_emit(buf, "%s\n", rfds_strings[rfds_mitigation]); + } + ++static ssize_t its_show_state(char *buf) ++{ ++ return sysfs_emit(buf, "%s\n", its_strings[its_mitigation]); ++} ++ + static char *stibp_state(void) + { + if (spectre_v2_in_eibrs_mode(spectre_v2_enabled) && +@@ -2959,6 +3073,9 @@ static ssize_t cpu_show_common(struct de + case X86_BUG_RFDS: + return rfds_show_state(buf); + ++ case X86_BUG_ITS: ++ return its_show_state(buf); ++ + default: + break; + } +@@ -3038,4 +3155,9 @@ ssize_t cpu_show_reg_file_data_sampling( + { + return cpu_show_common(dev, attr, buf, X86_BUG_RFDS); + } ++ ++ssize_t cpu_show_indirect_target_selection(struct device *dev, struct device_attribute *attr, char *buf) ++{ ++ return cpu_show_common(dev, attr, buf, X86_BUG_ITS); ++} + #endif +--- a/drivers/base/cpu.c ++++ b/drivers/base/cpu.c +@@ -595,6 +595,12 @@ ssize_t __weak cpu_show_reg_file_data_sa + return sysfs_emit(buf, "Not affected\n"); + } + ++ssize_t __weak cpu_show_indirect_target_selection(struct device *dev, ++ struct device_attribute *attr, char *buf) ++{ ++ return sysfs_emit(buf, "Not affected\n"); ++} ++ + static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL); + static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL); + static DEVICE_ATTR(spectre_v2, 0444, cpu_show_spectre_v2, NULL); +@@ -609,6 +615,7 @@ static DEVICE_ATTR(retbleed, 0444, cpu_s + static DEVICE_ATTR(gather_data_sampling, 0444, cpu_show_gds, NULL); + static DEVICE_ATTR(spec_rstack_overflow, 0444, cpu_show_spec_rstack_overflow, NULL); + static DEVICE_ATTR(reg_file_data_sampling, 0444, cpu_show_reg_file_data_sampling, NULL); ++static DEVICE_ATTR(indirect_target_selection, 0444, cpu_show_indirect_target_selection, NULL); + + static struct attribute *cpu_root_vulnerabilities_attrs[] = { + &dev_attr_meltdown.attr, +@@ -625,6 +632,7 @@ static struct attribute *cpu_root_vulner + &dev_attr_gather_data_sampling.attr, + &dev_attr_spec_rstack_overflow.attr, + &dev_attr_reg_file_data_sampling.attr, ++ &dev_attr_indirect_target_selection.attr, + NULL + }; + +--- a/include/linux/cpu.h ++++ b/include/linux/cpu.h +@@ -76,6 +76,8 @@ extern ssize_t cpu_show_gds(struct devic + struct device_attribute *attr, char *buf); + extern ssize_t cpu_show_reg_file_data_sampling(struct device *dev, + struct device_attribute *attr, char *buf); ++extern ssize_t cpu_show_indirect_target_selection(struct device *dev, ++ struct device_attribute *attr, char *buf); + + extern __printf(4, 5) + struct device *cpu_device_create(struct device *parent, void *drvdata, diff --git a/queue-5.15/x86-its-enumerate-indirect-target-selection-its-bug.patch b/queue-5.15/x86-its-enumerate-indirect-target-selection-its-bug.patch new file mode 100644 index 0000000000..c6f3d39a68 --- /dev/null +++ b/queue-5.15/x86-its-enumerate-indirect-target-selection-its-bug.patch @@ -0,0 +1,168 @@ +From stable+bounces-144649-greg=kroah.com@vger.kernel.org Sat May 17 02:01:10 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:01:01 -0700 +Subject: x86/its: Enumerate Indirect Target Selection (ITS) bug +To: stable@vger.kernel.org +Cc: Josh Poimboeuf , Alexandre Chartre +Message-ID: <20250516-its-5-15-v3-6-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit 159013a7ca18c271ff64192deb62a689b622d860 upstream. + +ITS bug in some pre-Alderlake Intel CPUs may allow indirect branches in the +first half of a cache line get predicted to a target of a branch located in +the second half of the cache line. + +Set X86_BUG_ITS on affected CPUs. Mitigation to follow in later commits. + +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Josh Poimboeuf +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 8 +++++ + arch/x86/kernel/cpu/common.c | 58 +++++++++++++++++++++++++++++-------- + arch/x86/kvm/x86.c | 4 +- + 4 files changed, 58 insertions(+), 13 deletions(-) + +--- a/arch/x86/include/asm/cpufeatures.h ++++ b/arch/x86/include/asm/cpufeatures.h +@@ -483,4 +483,5 @@ + #define X86_BUG_RFDS X86_BUG(1*32 + 2) /* CPU is vulnerable to Register File Data Sampling */ + #define X86_BUG_BHI X86_BUG(1*32 + 3) /* CPU is affected by Branch History Injection */ + #define X86_BUG_IBPB_NO_RET X86_BUG(1*32 + 4) /* "ibpb_no_ret" IBPB omits return target predictions */ ++#define X86_BUG_ITS X86_BUG(1*32 + 5) /* CPU is affected by Indirect Target Selection */ + #endif /* _ASM_X86_CPUFEATURES_H */ +--- a/arch/x86/include/asm/msr-index.h ++++ b/arch/x86/include/asm/msr-index.h +@@ -183,6 +183,14 @@ + * VERW clears CPU Register + * File. + */ ++#define ARCH_CAP_ITS_NO BIT_ULL(62) /* ++ * Not susceptible to ++ * Indirect Target Selection. ++ * This bit is not set by ++ * HW, but is synthesized by ++ * VMMs for guests to know ++ * their affected status. ++ */ + + #define MSR_IA32_FLUSH_CMD 0x0000010b + #define L1D_FLUSH BIT(0) /* +--- a/arch/x86/kernel/cpu/common.c ++++ b/arch/x86/kernel/cpu/common.c +@@ -1141,6 +1141,8 @@ static const __initconst struct x86_cpu_ + #define GDS BIT(6) + /* CPU is affected by Register File Data Sampling */ + #define RFDS BIT(7) ++/* CPU is affected by Indirect Target Selection */ ++#define ITS BIT(8) + + static const struct x86_cpu_id cpu_vuln_blacklist[] __initconst = { + VULNBL_INTEL_STEPPINGS(IVYBRIDGE, X86_STEPPING_ANY, SRBDS), +@@ -1152,22 +1154,25 @@ static const struct x86_cpu_id cpu_vuln_ + VULNBL_INTEL_STEPPINGS(BROADWELL_G, X86_STEPPING_ANY, SRBDS), + VULNBL_INTEL_STEPPINGS(BROADWELL_X, X86_STEPPING_ANY, MMIO), + VULNBL_INTEL_STEPPINGS(BROADWELL, X86_STEPPING_ANY, SRBDS), +- VULNBL_INTEL_STEPPINGS(SKYLAKE_X, X86_STEPPING_ANY, MMIO | RETBLEED | GDS), ++ VULNBL_INTEL_STEPPINGS(SKYLAKE_X, X86_STEPPINGS(0x0, 0x5), MMIO | RETBLEED | GDS), ++ VULNBL_INTEL_STEPPINGS(SKYLAKE_X, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | ITS), + VULNBL_INTEL_STEPPINGS(SKYLAKE_L, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS), + VULNBL_INTEL_STEPPINGS(SKYLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS), +- VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS), +- VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS), ++ VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPINGS(0x0, 0xb), MMIO | RETBLEED | GDS | SRBDS), ++ VULNBL_INTEL_STEPPINGS(KABYLAKE_L, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS | ITS), ++ VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPINGS(0x0, 0xc), MMIO | RETBLEED | GDS | SRBDS), ++ VULNBL_INTEL_STEPPINGS(KABYLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | SRBDS | ITS), + VULNBL_INTEL_STEPPINGS(CANNONLAKE_L, X86_STEPPING_ANY, RETBLEED), +- VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS), +- VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPING_ANY, MMIO | GDS), +- VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPING_ANY, MMIO | GDS), +- VULNBL_INTEL_STEPPINGS(COMETLAKE, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS), +- VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x0, 0x0), MMIO | RETBLEED), +- VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS), +- VULNBL_INTEL_STEPPINGS(TIGERLAKE_L, X86_STEPPING_ANY, GDS), +- VULNBL_INTEL_STEPPINGS(TIGERLAKE, X86_STEPPING_ANY, GDS), ++ VULNBL_INTEL_STEPPINGS(ICELAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(ICELAKE_D, X86_STEPPING_ANY, MMIO | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(ICELAKE_X, X86_STEPPING_ANY, MMIO | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(COMETLAKE, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPINGS(0x0, 0x0), MMIO | RETBLEED | ITS), ++ VULNBL_INTEL_STEPPINGS(COMETLAKE_L, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED | GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(TIGERLAKE_L, X86_STEPPING_ANY, GDS | ITS), ++ VULNBL_INTEL_STEPPINGS(TIGERLAKE, X86_STEPPING_ANY, GDS | ITS), + VULNBL_INTEL_STEPPINGS(LAKEFIELD, X86_STEPPING_ANY, MMIO | MMIO_SBDS | RETBLEED), +- VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS), ++ VULNBL_INTEL_STEPPINGS(ROCKETLAKE, X86_STEPPING_ANY, MMIO | RETBLEED | GDS | ITS), + VULNBL_INTEL_STEPPINGS(ALDERLAKE, X86_STEPPING_ANY, RFDS), + VULNBL_INTEL_STEPPINGS(ALDERLAKE_L, X86_STEPPING_ANY, RFDS), + VULNBL_INTEL_STEPPINGS(RAPTORLAKE, X86_STEPPING_ANY, RFDS), +@@ -1231,6 +1236,32 @@ static bool __init vulnerable_to_rfds(u6 + return cpu_matches(cpu_vuln_blacklist, RFDS); + } + ++static bool __init vulnerable_to_its(u64 x86_arch_cap_msr) ++{ ++ /* The "immunity" bit trumps everything else: */ ++ if (x86_arch_cap_msr & ARCH_CAP_ITS_NO) ++ return false; ++ if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL) ++ return false; ++ ++ /* None of the affected CPUs have BHI_CTRL */ ++ if (boot_cpu_has(X86_FEATURE_BHI_CTRL)) ++ return false; ++ ++ /* ++ * If a VMM did not expose ITS_NO, assume that a guest could ++ * be running on a vulnerable hardware or may migrate to such ++ * hardware. ++ */ ++ if (boot_cpu_has(X86_FEATURE_HYPERVISOR)) ++ return true; ++ ++ if (cpu_matches(cpu_vuln_blacklist, ITS)) ++ return true; ++ ++ return false; ++} ++ + static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c) + { + u64 x86_arch_cap_msr = x86_read_arch_cap_msr(); +@@ -1358,6 +1389,9 @@ static void __init cpu_set_bug_bits(stru + if (cpu_has(c, X86_FEATURE_AMD_IBPB) && !cpu_has(c, X86_FEATURE_AMD_IBPB_RET)) + setup_force_cpu_bug(X86_BUG_IBPB_NO_RET); + ++ if (vulnerable_to_its(x86_arch_cap_msr)) ++ setup_force_cpu_bug(X86_BUG_ITS); ++ + if (cpu_matches(cpu_vuln_whitelist, NO_MELTDOWN)) + return; + +--- a/arch/x86/kvm/x86.c ++++ b/arch/x86/kvm/x86.c +@@ -1499,7 +1499,7 @@ static unsigned int num_msr_based_featur + ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \ + ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \ + ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO | \ +- ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO) ++ ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO | ARCH_CAP_ITS_NO) + + static u64 kvm_get_arch_capabilities(void) + { +@@ -1538,6 +1538,8 @@ static u64 kvm_get_arch_capabilities(voi + data |= ARCH_CAP_MDS_NO; + if (!boot_cpu_has_bug(X86_BUG_RFDS)) + data |= ARCH_CAP_RFDS_NO; ++ if (!boot_cpu_has_bug(X86_BUG_ITS)) ++ data |= ARCH_CAP_ITS_NO; + + if (!boot_cpu_has(X86_FEATURE_RTM)) { + /* diff --git a/queue-5.15/x86-its-fineibt-paranoid-vs-its.patch b/queue-5.15/x86-its-fineibt-paranoid-vs-its.patch new file mode 100644 index 0000000000..766d7dc39c --- /dev/null +++ b/queue-5.15/x86-its-fineibt-paranoid-vs-its.patch @@ -0,0 +1,121 @@ +From stable+bounces-144659-greg=kroah.com@vger.kernel.org Sat May 17 02:03:44 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:03:37 -0700 +Subject: x86/its: FineIBT-paranoid vs ITS +To: stable@vger.kernel.org +Cc: Peter Zijlstra +Message-ID: <20250516-its-5-15-v3-16-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Peter Zijlstra + +commit e52c1dc7455d32c8a55f9949d300e5e87d011fa6 upstream. + +FineIBT-paranoid was using the retpoline bytes for the paranoid check, +disabling retpolines, because all parts that have IBT also have eIBRS +and thus don't need no stinking retpolines. + +Except... ITS needs the retpolines for indirect calls must not be in +the first half of a cacheline :-/ + +So what was the paranoid call sequence: + + : + 0: 41 ba 78 56 34 12 mov $0x12345678, %r10d + 6: 45 3b 53 f7 cmp -0x9(%r11), %r10d + a: 4d 8d 5b lea -0x10(%r11), %r11 + e: 75 fd jne d + 10: 41 ff d3 call *%r11 + 13: 90 nop + +Now becomes: + + : + 0: 41 ba 78 56 34 12 mov $0x12345678, %r10d + 6: 45 3b 53 f7 cmp -0x9(%r11), %r10d + a: 4d 8d 5b f0 lea -0x10(%r11), %r11 + e: 2e e8 XX XX XX XX cs call __x86_indirect_paranoid_thunk_r11 + + Where the paranoid_thunk looks like: + + 1d: (bad) + __x86_indirect_paranoid_thunk_r11: + 1e: 75 fd jne 1d + __x86_indirect_its_thunk_r11: + 20: 41 ff eb jmp *%r11 + 23: cc int3 + +[ dhansen: remove initialization to false ] + +Signed-off-by: Peter Zijlstra (Intel) +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Alexandre Chartre +[ Just a portion of the original commit, in order to fix a build issue + in stable kernels due to backports ] +Tested-by: Holger Hoffstätte +Link: https://lore.kernel.org/r/20250514113952.GB16434@noisy.programming.kicks-ass.net +Signed-off-by: Greg Kroah-Hartman +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/alternative.h | 8 ++++++++ + arch/x86/kernel/alternative.c | 8 ++++++++ + arch/x86/net/bpf_jit_comp.c | 2 +- + 3 files changed, 17 insertions(+), 1 deletion(-) + +--- a/arch/x86/include/asm/alternative.h ++++ b/arch/x86/include/asm/alternative.h +@@ -5,6 +5,7 @@ + #include + #include + #include ++#include + + #define ALTINSTR_FLAG_INV (1 << 15) + #define ALT_NOT(feat) ((feat) | ALTINSTR_FLAG_INV) +@@ -84,10 +85,17 @@ struct module; + extern void its_init_mod(struct module *mod); + extern void its_fini_mod(struct module *mod); + extern void its_free_mod(struct module *mod); ++extern u8 *its_static_thunk(int reg); + #else /* CONFIG_MITIGATION_ITS */ + static inline void its_init_mod(struct module *mod) { } + static inline void its_fini_mod(struct module *mod) { } + static inline void its_free_mod(struct module *mod) { } ++static inline u8 *its_static_thunk(int reg) ++{ ++ WARN_ONCE(1, "ITS not compiled in"); ++ ++ return NULL; ++} + #endif + + #ifdef CONFIG_RETHUNK +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -597,6 +597,14 @@ static bool cpu_wants_indirect_its_thunk + /* Lower-half of the cacheline? */ + return !(addr & 0x20); + } ++ ++u8 *its_static_thunk(int reg) ++{ ++ u8 *thunk = __x86_indirect_its_thunk_array[reg]; ++ ++ return thunk; ++} ++ + #endif + + /* +--- a/arch/x86/net/bpf_jit_comp.c ++++ b/arch/x86/net/bpf_jit_comp.c +@@ -449,7 +449,7 @@ static void emit_indirect_jump(u8 **ppro + if (IS_ENABLED(CONFIG_MITIGATION_ITS) && + cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) { + OPTIMIZER_HIDE_VAR(reg); +- emit_jump(&prog, &__x86_indirect_its_thunk_array[reg], ip); ++ emit_jump(&prog, its_static_thunk(reg), ip); + } else if (cpu_feature_enabled(X86_FEATURE_RETPOLINE_LFENCE)) { + EMIT_LFENCE(); + EMIT2(0xFF, 0xE0 + reg); diff --git a/queue-5.15/x86-its-fix-build-errors-when-config_modules-n.patch b/queue-5.15/x86-its-fix-build-errors-when-config_modules-n.patch new file mode 100644 index 0000000000..4378ef322e --- /dev/null +++ b/queue-5.15/x86-its-fix-build-errors-when-config_modules-n.patch @@ -0,0 +1,59 @@ +From stable+bounces-144658-greg=kroah.com@vger.kernel.org Sat May 17 02:03:28 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:03:22 -0700 +Subject: x86/its: Fix build errors when CONFIG_MODULES=n +To: stable@vger.kernel.org +Cc: Eric Biggers , Greg Kroah-Hartman +Message-ID: <20250516-its-5-15-v3-15-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Eric Biggers + +commit 9f35e33144ae5377d6a8de86dd3bd4d995c6ac65 upstream. + +Fix several build errors when CONFIG_MODULES=n, including the following: + +../arch/x86/kernel/alternative.c:195:25: error: incomplete definition of type 'struct module' + 195 | for (int i = 0; i < mod->its_num_pages; i++) { + + [ pawan: backport: Bring ITS dynamic thunk code under CONFIG_MODULES ] + +Fixes: 872df34d7c51 ("x86/its: Use dynamic thunks for indirect branches") +Cc: stable@vger.kernel.org +Signed-off-by: Eric Biggers +Acked-by: Dave Hansen +Tested-by: Steven Rostedt (Google) +Reviewed-by: Alexandre Chartre +Signed-off-by: Linus Torvalds +Signed-off-by: Greg Kroah-Hartman +Signed-off-by: Pawan Gupta +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/kernel/alternative.c | 9 +++++++++ + 1 file changed, 9 insertions(+) + +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -399,6 +399,7 @@ static int emit_indirect(int op, int reg + + #ifdef CONFIG_MITIGATION_ITS + ++#ifdef CONFIG_MODULES + static struct module *its_mod; + static void *its_page; + static unsigned int its_offset; +@@ -519,6 +520,14 @@ static void *its_allocate_thunk(int reg) + + return thunk; + } ++#else /* CONFIG_MODULES */ ++ ++static void *its_allocate_thunk(int reg) ++{ ++ return NULL; ++} ++ ++#endif /* CONFIG_MODULES */ + + static int __emit_trampoline(void *addr, struct insn *insn, u8 *bytes, + void *call_dest, void *jmp_dest) diff --git a/queue-5.15/x86-its-use-dynamic-thunks-for-indirect-branches.patch b/queue-5.15/x86-its-use-dynamic-thunks-for-indirect-branches.patch new file mode 100644 index 0000000000..c64db94b76 --- /dev/null +++ b/queue-5.15/x86-its-use-dynamic-thunks-for-indirect-branches.patch @@ -0,0 +1,267 @@ +From stable+bounces-144657-greg=kroah.com@vger.kernel.org Sat May 17 02:03:13 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:03:06 -0700 +Subject: x86/its: Use dynamic thunks for indirect branches +To: stable@vger.kernel.org +Cc: Peter Zijlstra +Message-ID: <20250516-its-5-15-v3-14-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Peter Zijlstra + +commit 872df34d7c51a79523820ea6a14860398c639b87 upstream. + +ITS mitigation moves the unsafe indirect branches to a safe thunk. This +could degrade the prediction accuracy as the source address of indirect +branches becomes same for different execution paths. + +To improve the predictions, and hence the performance, assign a separate +thunk for each indirect callsite. This is also a defense-in-depth measure +to avoid indirect branches aliasing with each other. + +As an example, 5000 dynamic thunks would utilize around 16 bits of the +address space, thereby gaining entropy. For a BTB that uses +32 bits for indexing, dynamic thunks could provide better prediction +accuracy over fixed thunks. + +Have ITS thunks be variable sized and use EXECMEM_MODULE_TEXT such that +they are both more flexible (got to extend them later) and live in 2M TLBs, +just like kernel code, avoiding undue TLB pressure. + + [ pawan: CONFIG_EXECMEM and CONFIG_EXECMEM_ROX are not supported on + backport kernel, made changes to use module_alloc() and + set_memory_*() for dynamic thunks. ] + +Signed-off-by: Peter Zijlstra (Intel) +Signed-off-by: Pawan Gupta +Signed-off-by: Dave Hansen +Reviewed-by: Alexandre Chartre +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/alternative.h | 10 ++ + arch/x86/kernel/alternative.c | 133 ++++++++++++++++++++++++++++++++++++- + arch/x86/kernel/module.c | 7 + + include/linux/module.h | 5 + + 4 files changed, 152 insertions(+), 3 deletions(-) + +--- a/arch/x86/include/asm/alternative.h ++++ b/arch/x86/include/asm/alternative.h +@@ -80,6 +80,16 @@ extern void apply_returns(s32 *start, s3 + + struct module; + ++#ifdef CONFIG_MITIGATION_ITS ++extern void its_init_mod(struct module *mod); ++extern void its_fini_mod(struct module *mod); ++extern void its_free_mod(struct module *mod); ++#else /* CONFIG_MITIGATION_ITS */ ++static inline void its_init_mod(struct module *mod) { } ++static inline void its_fini_mod(struct module *mod) { } ++static inline void its_free_mod(struct module *mod) { } ++#endif ++ + #ifdef CONFIG_RETHUNK + extern bool cpu_wants_rethunk(void); + extern bool cpu_wants_rethunk_at(void *addr); +--- a/arch/x86/kernel/alternative.c ++++ b/arch/x86/kernel/alternative.c +@@ -18,6 +18,7 @@ + #include + #include + #include ++#include + #include + #include + #include +@@ -30,6 +31,7 @@ + #include + #include + #include ++#include + + int __read_mostly alternatives_patched; + +@@ -397,6 +399,127 @@ static int emit_indirect(int op, int reg + + #ifdef CONFIG_MITIGATION_ITS + ++static struct module *its_mod; ++static void *its_page; ++static unsigned int its_offset; ++ ++/* Initialize a thunk with the "jmp *reg; int3" instructions. */ ++static void *its_init_thunk(void *thunk, int reg) ++{ ++ u8 *bytes = thunk; ++ int i = 0; ++ ++ if (reg >= 8) { ++ bytes[i++] = 0x41; /* REX.B prefix */ ++ reg -= 8; ++ } ++ bytes[i++] = 0xff; ++ bytes[i++] = 0xe0 + reg; /* jmp *reg */ ++ bytes[i++] = 0xcc; ++ ++ return thunk; ++} ++ ++void its_init_mod(struct module *mod) ++{ ++ if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) ++ return; ++ ++ mutex_lock(&text_mutex); ++ its_mod = mod; ++ its_page = NULL; ++} ++ ++void its_fini_mod(struct module *mod) ++{ ++ int i; ++ ++ if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) ++ return; ++ ++ WARN_ON_ONCE(its_mod != mod); ++ ++ its_mod = NULL; ++ its_page = NULL; ++ mutex_unlock(&text_mutex); ++ ++ for (i = 0; i < mod->its_num_pages; i++) { ++ void *page = mod->its_page_array[i]; ++ set_memory_ro((unsigned long)page, 1); ++ set_memory_x((unsigned long)page, 1); ++ } ++} ++ ++void its_free_mod(struct module *mod) ++{ ++ int i; ++ ++ if (!cpu_feature_enabled(X86_FEATURE_INDIRECT_THUNK_ITS)) ++ return; ++ ++ for (i = 0; i < mod->its_num_pages; i++) { ++ void *page = mod->its_page_array[i]; ++ module_memfree(page); ++ } ++ kfree(mod->its_page_array); ++} ++ ++static void *its_alloc(void) ++{ ++ void *page = module_alloc(PAGE_SIZE); ++ ++ if (!page) ++ return NULL; ++ ++ if (its_mod) { ++ void *tmp = krealloc(its_mod->its_page_array, ++ (its_mod->its_num_pages+1) * sizeof(void *), ++ GFP_KERNEL); ++ if (!tmp) { ++ module_memfree(page); ++ return NULL; ++ } ++ ++ its_mod->its_page_array = tmp; ++ its_mod->its_page_array[its_mod->its_num_pages++] = page; ++ } ++ ++ return page; ++} ++ ++static void *its_allocate_thunk(int reg) ++{ ++ int size = 3 + (reg / 8); ++ void *thunk; ++ ++ if (!its_page || (its_offset + size - 1) >= PAGE_SIZE) { ++ its_page = its_alloc(); ++ if (!its_page) { ++ pr_err("ITS page allocation failed\n"); ++ return NULL; ++ } ++ memset(its_page, INT3_INSN_OPCODE, PAGE_SIZE); ++ its_offset = 32; ++ } ++ ++ /* ++ * If the indirect branch instruction will be in the lower half ++ * of a cacheline, then update the offset to reach the upper half. ++ */ ++ if ((its_offset + size - 1) % 64 < 32) ++ its_offset = ((its_offset - 1) | 0x3F) + 33; ++ ++ thunk = its_page + its_offset; ++ its_offset += size; ++ ++ set_memory_rw((unsigned long)its_page, 1); ++ thunk = its_init_thunk(thunk, reg); ++ set_memory_ro((unsigned long)its_page, 1); ++ set_memory_x((unsigned long)its_page, 1); ++ ++ return thunk; ++} ++ + static int __emit_trampoline(void *addr, struct insn *insn, u8 *bytes, + void *call_dest, void *jmp_dest) + { +@@ -444,9 +567,13 @@ clang_jcc: + + static int emit_its_trampoline(void *addr, struct insn *insn, int reg, u8 *bytes) + { +- return __emit_trampoline(addr, insn, bytes, +- __x86_indirect_its_thunk_array[reg], +- __x86_indirect_its_thunk_array[reg]); ++ u8 *thunk = __x86_indirect_its_thunk_array[reg]; ++ u8 *tmp = its_allocate_thunk(reg); ++ ++ if (tmp) ++ thunk = tmp; ++ ++ return __emit_trampoline(addr, insn, bytes, thunk, thunk); + } + + /* Check if an indirect branch is at ITS-unsafe address */ +--- a/arch/x86/kernel/module.c ++++ b/arch/x86/kernel/module.c +@@ -283,10 +283,16 @@ int module_finalize(const Elf_Ehdr *hdr, + void *pseg = (void *)para->sh_addr; + apply_paravirt(pseg, pseg + para->sh_size); + } ++ ++ its_init_mod(me); ++ + if (retpolines) { + void *rseg = (void *)retpolines->sh_addr; + apply_retpolines(rseg, rseg + retpolines->sh_size); + } ++ ++ its_fini_mod(me); ++ + if (returns) { + void *rseg = (void *)returns->sh_addr; + apply_returns(rseg, rseg + returns->sh_size); +@@ -317,4 +323,5 @@ int module_finalize(const Elf_Ehdr *hdr, + void module_arch_cleanup(struct module *mod) + { + alternatives_smp_module_del(mod); ++ its_free_mod(mod); + } +--- a/include/linux/module.h ++++ b/include/linux/module.h +@@ -528,6 +528,11 @@ struct module { + atomic_t refcnt; + #endif + ++#ifdef CONFIG_MITIGATION_ITS ++ int its_num_pages; ++ void **its_page_array; ++#endif ++ + #ifdef CONFIG_CONSTRUCTORS + /* Constructor functions. */ + ctor_fn_t *ctors; diff --git a/queue-5.15/x86-nospec-simplify-jmp-call-_nospec.patch b/queue-5.15/x86-nospec-simplify-jmp-call-_nospec.patch new file mode 100644 index 0000000000..55003b8b00 --- /dev/null +++ b/queue-5.15/x86-nospec-simplify-jmp-call-_nospec.patch @@ -0,0 +1,72 @@ +From stable+bounces-144643-greg=kroah.com@vger.kernel.org Sat May 17 01:59:51 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 16:59:44 -0700 +Subject: x86,nospec: Simplify {JMP,CALL}_NOSPEC +To: stable@vger.kernel.org +Cc: Peter Zijlstra +Message-ID: <20250516-its-5-15-v3-1-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Peter Zijlstra + +commit 09d09531a51a24635bc3331f56d92ee7092f5516 upstream. + +Have {JMP,CALL}_NOSPEC generate the same code GCC does for indirect +calls and rely on the objtool retpoline patching infrastructure. + +There's no reason these should be alternatives while the vast bulk of +compiler generated retpolines are not. + +Signed-off-by: Peter Zijlstra (Intel) +Signed-off-by: Pawan Gupta +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/nospec-branch.h | 24 ++++++++++++++++++------ + 1 file changed, 18 insertions(+), 6 deletions(-) + +--- a/arch/x86/include/asm/nospec-branch.h ++++ b/arch/x86/include/asm/nospec-branch.h +@@ -119,25 +119,37 @@ + .endm + + /* ++ * Equivalent to -mindirect-branch-cs-prefix; emit the 5 byte jmp/call ++ * to the retpoline thunk with a CS prefix when the register requires ++ * a RAX prefix byte to encode. Also see apply_retpolines(). ++ */ ++.macro __CS_PREFIX reg:req ++ .irp rs,r8,r9,r10,r11,r12,r13,r14,r15 ++ .ifc \reg,\rs ++ .byte 0x2e ++ .endif ++ .endr ++.endm ++ ++/* + * JMP_NOSPEC and CALL_NOSPEC macros can be used instead of a simple + * indirect jmp/call which may be susceptible to the Spectre variant 2 + * attack. + */ + .macro JMP_NOSPEC reg:req + #ifdef CONFIG_RETPOLINE +- ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), \ +- __stringify(jmp __x86_indirect_thunk_\reg), X86_FEATURE_RETPOLINE, \ +- __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; jmp *%\reg), X86_FEATURE_RETPOLINE_LFENCE ++ __CS_PREFIX \reg ++ jmp __x86_indirect_thunk_\reg + #else + jmp *%\reg ++ int3 + #endif + .endm + + .macro CALL_NOSPEC reg:req + #ifdef CONFIG_RETPOLINE +- ALTERNATIVE_2 __stringify(ANNOTATE_RETPOLINE_SAFE; call *%\reg), \ +- __stringify(call __x86_indirect_thunk_\reg), X86_FEATURE_RETPOLINE, \ +- __stringify(lfence; ANNOTATE_RETPOLINE_SAFE; call *%\reg), X86_FEATURE_RETPOLINE_LFENCE ++ __CS_PREFIX \reg ++ call __x86_indirect_thunk_\reg + #else + call *%\reg + #endif diff --git a/queue-5.15/x86-speculation-add-a-conditional-cs-prefix-to-call_nospec.patch b/queue-5.15/x86-speculation-add-a-conditional-cs-prefix-to-call_nospec.patch new file mode 100644 index 0000000000..356daf2ce9 --- /dev/null +++ b/queue-5.15/x86-speculation-add-a-conditional-cs-prefix-to-call_nospec.patch @@ -0,0 +1,73 @@ +From stable+bounces-144645-greg=kroah.com@vger.kernel.org Sat May 17 02:00:22 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:00:15 -0700 +Subject: x86/speculation: Add a conditional CS prefix to CALL_NOSPEC +To: stable@vger.kernel.org +Cc: Josh Poimboeuf +Message-ID: <20250516-its-5-15-v3-3-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit 052040e34c08428a5a388b85787e8531970c0c67 upstream. + +Retpoline mitigation for spectre-v2 uses thunks for indirect branches. To +support this mitigation compilers add a CS prefix with +-mindirect-branch-cs-prefix. For an indirect branch in asm, this needs to +be added manually. + +CS prefix is already being added to indirect branches in asm files, but not +in inline asm. Add CS prefix to CALL_NOSPEC for inline asm as well. There +is no JMP_NOSPEC for inline asm. + +Reported-by: Josh Poimboeuf +Signed-off-by: Pawan Gupta +Signed-off-by: Ingo Molnar +Cc: Andrew Cooper +Cc: Peter Zijlstra +Link: https://lore.kernel.org/r/20250228-call-nospec-v3-2-96599fed0f33@linux.intel.com +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/nospec-branch.h | 19 +++++++++++++++---- + 1 file changed, 15 insertions(+), 4 deletions(-) + +--- a/arch/x86/include/asm/nospec-branch.h ++++ b/arch/x86/include/asm/nospec-branch.h +@@ -119,9 +119,8 @@ + .endm + + /* +- * Equivalent to -mindirect-branch-cs-prefix; emit the 5 byte jmp/call +- * to the retpoline thunk with a CS prefix when the register requires +- * a RAX prefix byte to encode. Also see apply_retpolines(). ++ * Emits a conditional CS prefix that is compatible with ++ * -mindirect-branch-cs-prefix. + */ + .macro __CS_PREFIX reg:req + .irp rs,r8,r9,r10,r11,r12,r13,r14,r15 +@@ -282,11 +281,23 @@ extern retpoline_thunk_t __x86_indirect_ + #ifdef CONFIG_X86_64 + + /* ++ * Emits a conditional CS prefix that is compatible with ++ * -mindirect-branch-cs-prefix. ++ */ ++#define __CS_PREFIX(reg) \ ++ ".irp rs,r8,r9,r10,r11,r12,r13,r14,r15\n" \ ++ ".ifc \\rs," reg "\n" \ ++ ".byte 0x2e\n" \ ++ ".endif\n" \ ++ ".endr\n" ++ ++/* + * Inline asm uses the %V modifier which is only in newer GCC + * which is ensured when CONFIG_RETPOLINE is defined. + */ + #ifdef CONFIG_RETPOLINE +-#define CALL_NOSPEC "call __x86_indirect_thunk_%V[thunk_target]\n" ++#define CALL_NOSPEC __CS_PREFIX("%V[thunk_target]") \ ++ "call __x86_indirect_thunk_%V[thunk_target]\n" + #else + #define CALL_NOSPEC "call *%[thunk_target]\n" + #endif diff --git a/queue-5.15/x86-speculation-remove-the-extra-ifdef-around-call_nospec.patch b/queue-5.15/x86-speculation-remove-the-extra-ifdef-around-call_nospec.patch new file mode 100644 index 0000000000..03e8bc2732 --- /dev/null +++ b/queue-5.15/x86-speculation-remove-the-extra-ifdef-around-call_nospec.patch @@ -0,0 +1,59 @@ +From stable+bounces-144646-greg=kroah.com@vger.kernel.org Sat May 17 02:00:39 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 17:00:30 -0700 +Subject: x86/speculation: Remove the extra #ifdef around CALL_NOSPEC +To: stable@vger.kernel.org +Message-ID: <20250516-its-5-15-v3-4-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit c8c81458863ab686cda4fe1e603fccaae0f12460 upstream. + +Commit: + + 010c4a461c1d ("x86/speculation: Simplify and make CALL_NOSPEC consistent") + +added an #ifdef CONFIG_RETPOLINE around the CALL_NOSPEC definition. This is +not required as this code is already under a larger #ifdef. + +Remove the extra #ifdef, no functional change. + +vmlinux size remains same before and after this change: + + CONFIG_RETPOLINE=y: + text data bss dec hex filename + 25434752 7342290 2301212 35078254 217406e vmlinux.before + 25434752 7342290 2301212 35078254 217406e vmlinux.after + + # CONFIG_RETPOLINE is not set: + text data bss dec hex filename + 22943094 6214994 1550152 30708240 1d49210 vmlinux.before + 22943094 6214994 1550152 30708240 1d49210 vmlinux.after + + [ pawan: s/CONFIG_MITIGATION_RETPOLINE/CONFIG_RETPOLINE/ ] + +Signed-off-by: Pawan Gupta +Signed-off-by: Ingo Molnar +Reviewed-by: Josh Poimboeuf +Link: https://lore.kernel.org/r/20250320-call-nospec-extra-ifdef-v1-1-d9b084d24820@linux.intel.com +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/nospec-branch.h | 4 ---- + 1 file changed, 4 deletions(-) + +--- a/arch/x86/include/asm/nospec-branch.h ++++ b/arch/x86/include/asm/nospec-branch.h +@@ -295,12 +295,8 @@ extern retpoline_thunk_t __x86_indirect_ + * Inline asm uses the %V modifier which is only in newer GCC + * which is ensured when CONFIG_RETPOLINE is defined. + */ +-#ifdef CONFIG_RETPOLINE + #define CALL_NOSPEC __CS_PREFIX("%V[thunk_target]") \ + "call __x86_indirect_thunk_%V[thunk_target]\n" +-#else +-#define CALL_NOSPEC "call *%[thunk_target]\n" +-#endif + + # define THUNK_TARGET(addr) [thunk_target] "r" (addr) + diff --git a/queue-5.15/x86-speculation-simplify-and-make-call_nospec-consistent.patch b/queue-5.15/x86-speculation-simplify-and-make-call_nospec-consistent.patch new file mode 100644 index 0000000000..c06a84cc53 --- /dev/null +++ b/queue-5.15/x86-speculation-simplify-and-make-call_nospec-consistent.patch @@ -0,0 +1,62 @@ +From stable+bounces-144644-greg=kroah.com@vger.kernel.org Sat May 17 02:00:07 2025 +From: Pawan Gupta +Date: Fri, 16 May 2025 16:59:59 -0700 +Subject: x86/speculation: Simplify and make CALL_NOSPEC consistent +To: stable@vger.kernel.org +Cc: Peter Zijlstra +Message-ID: <20250516-its-5-15-v3-2-16fcdaaea544@linux.intel.com> +Content-Disposition: inline + +From: Pawan Gupta + +commit cfceff8526a426948b53445c02bcb98453c7330d upstream. + +CALL_NOSPEC macro is used to generate Spectre-v2 mitigation friendly +indirect branches. At compile time the macro defaults to indirect branch, +and at runtime those can be patched to thunk based mitigations. + +This approach is opposite of what is done for the rest of the kernel, where +the compile time default is to replace indirect calls with retpoline thunk +calls. + +Make CALL_NOSPEC consistent with the rest of the kernel, default to +retpoline thunk at compile time when CONFIG_RETPOLINE is +enabled. + + [ pawan: s/CONFIG_MITIGATION_RETPOLINE/CONFIG_RETPOLINE/ ] + +Signed-off-by: Pawan Gupta +Signed-off-by: Ingo Molnar +Cc: Andrew Cooper +Cc: Peter Zijlstra +Link: https://lore.kernel.org/r/20250228-call-nospec-v3-1-96599fed0f33@linux.intel.com +Signed-off-by: Greg Kroah-Hartman +--- + arch/x86/include/asm/nospec-branch.h | 15 +++++---------- + 1 file changed, 5 insertions(+), 10 deletions(-) + +--- a/arch/x86/include/asm/nospec-branch.h ++++ b/arch/x86/include/asm/nospec-branch.h +@@ -285,16 +285,11 @@ extern retpoline_thunk_t __x86_indirect_ + * Inline asm uses the %V modifier which is only in newer GCC + * which is ensured when CONFIG_RETPOLINE is defined. + */ +-# define CALL_NOSPEC \ +- ALTERNATIVE_2( \ +- ANNOTATE_RETPOLINE_SAFE \ +- "call *%[thunk_target]\n", \ +- "call __x86_indirect_thunk_%V[thunk_target]\n", \ +- X86_FEATURE_RETPOLINE, \ +- "lfence;\n" \ +- ANNOTATE_RETPOLINE_SAFE \ +- "call *%[thunk_target]\n", \ +- X86_FEATURE_RETPOLINE_LFENCE) ++#ifdef CONFIG_RETPOLINE ++#define CALL_NOSPEC "call __x86_indirect_thunk_%V[thunk_target]\n" ++#else ++#define CALL_NOSPEC "call *%[thunk_target]\n" ++#endif + + # define THUNK_TARGET(addr) [thunk_target] "r" (addr) +