bpf: Prefer packs that won't trigger an IBPB flush on allocation

Currently BPF pack allocator picks the chunks from the first available
pack. While this is okay, it naturally leads to more frequent flushes
when there are multiple packs in the system that weren't used since the
last flush.

As an optimization prefer allocating the new programs from packs that
are unused since last flush. When all packs are dirty, allocation forces
a flush and marks all packs clean.

Below are some future optimizations ideas:

  1. Currently, the "dirty" tracking is only done at the pack-level.
     Flush frequency can further be reduced with chunk-level tracking.
     This requires a new bitmap per-pack to track the dirty state.
  2. IBPB flush is done on all CPUs, even if only a single CPU ran the
     BPF program. On a system with hundreds of CPUs this could be a
     major bottleneck forcing hundreds of IPIs to deliver the flush.
     The solution is to track the CPUs where a BPF program ran, and
     issue IBPB only on those CPUs.
  3. Avoid IBPB when flush is already done at other sources (e.g.
     context switch).

Signed-off-by: Pawan Gupta <pawan.kumar.gupta@linux.intel.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>

diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 50aba113ef9dcc1e8b437ed0ba7bcacd00358146..1b32b9f2491f40cb18f0a1b2c877c0178958fb0b 100644 (file)
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -948,8 +948,8 @@ out:
 void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool was_classic)
 {
        unsigned int nbits = BPF_PROG_SIZE_TO_NBITS(size);
-       struct bpf_prog_pack *pack;
-       unsigned long pos;
+       struct bpf_prog_pack *pack, *fallback_pack = NULL;
+       unsigned long pos, fallback_pos = 0;
        void *ptr = NULL;
 
        mutex_lock(&pack_mutex);
@@ -981,8 +981,29 @@ void *bpf_prog_pack_alloc(u32 size, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool
        list_for_each_entry(pack, &pack_list, list) {
                pos = bitmap_find_next_zero_area(pack->bitmap, BPF_PROG_CHUNK_COUNT, 0,
                                                 nbits, 0);
-               if (pos < BPF_PROG_CHUNK_COUNT)
+               if (pos >= BPF_PROG_CHUNK_COUNT)
+                       continue;
+               /* Flush not enabled, use any pack */
+               if (!static_branch_unlikely(&bpf_pred_flush_enabled))
                        goto found_free_area;
+               /*
+                * cBPF reuse of a dirty pack triggers a flush, so prefer a
+                * clean pack for cBPF. eBPF never flushes, so pick the first
+                * free pack, dirty or clean.
+                */
+               if (!was_classic || !pack->arch_flush_needed)
+                       goto found_free_area;
+               if (!fallback_pack) {
+                       fallback_pack = pack;
+                       fallback_pos = pos;
+               }
+       }
+
+       /* No preferred pack found */
+       if (fallback_pack) {
+               pack = fallback_pack;
+               pos = fallback_pos;
+               goto found_free_area;
        }
 
        pack = alloc_new_pack(bpf_fill_ill_insns);