bpf: Fix atomic probe zero-extension

[ Upstream commit df34ec9db6f521118895f22795da49f2ec01f8cf ]

Zero-extending results of atomic probe operations fails with:

    verifier bug. zext_dst is set, but no reg is defined

The problem is that insn_def_regno() handles BPF_ATOMICs, but not
BPF_PROBE_ATOMICs. Fix by adding the missing condition.

Fixes: d503a04f8bc0 ("bpf: Add support for certain atomics in bpf_arena to x86 JIT")
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20240701234304.14336-2-iii@linux.ibm.com
Signed-off-by: Sasha Levin <sashal@kernel.org>

diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 214a9fa8c6fb74f7310ee9152e60a330e3ca9708..e1e08e62a2f2fa404918db565da73d60b20ae82a 100644 (file)
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -3215,7 +3215,8 @@ static int insn_def_regno(const struct bpf_insn *insn)
        case BPF_ST:
                return -1;
        case BPF_STX:
-               if (BPF_MODE(insn->code) == BPF_ATOMIC &&
+               if ((BPF_MODE(insn->code) == BPF_ATOMIC ||
+                    BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
                    (insn->imm & BPF_FETCH)) {
                        if (insn->imm == BPF_CMPXCHG)
                                return BPF_REG_0;