io_uring/rsrc: skip atomic refcount for uncloned buffers

io_buffer_unmap() performs an atomic decrement of the io_mapped_ubuf's
reference count in case it has been cloned into another io_ring_ctx's
registered buffer table. This is an expensive operation and unnecessary
in the common case that the io_mapped_ubuf is only registered once.
Load the reference count first and check whether it's 1. In that case,
skip the atomic decrement and immediately free the io_mapped_ubuf.

Signed-off-by: Caleb Sander Mateos <csander@purestorage.com>
Link: https://lore.kernel.org/r/20250619143435.3474028-1-csander@purestorage.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index d724602697e7cfdf0229718d7be8874bf135c15d..fc51ca7de73302f1b58650bc9ec9e05dbdc45281 100644 (file)
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -135,8 +135,10 @@ static void io_free_imu(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)
 
 static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu)
 {
-       if (!refcount_dec_and_test(&imu->refs))
-               return;
+       if (unlikely(refcount_read(&imu->refs) > 1)) {
+               if (!refcount_dec_and_test(&imu->refs))
+                       return;
+       }
 
        if (imu->acct_pages)
                io_unaccount_mem(ctx, imu->acct_pages);