io_uring: defer iowq cqe overflow via task_work

Don't handle CQE overflows in io_req_complete_post() and defer it to
flush_completions. It cuts some duplication, and I also want to limit
the number of places directly overflowing completions.

Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/9046410ac27e18f2baa6f7cdb363ec921cbc3b79.1742829388.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index e6c46294827355d071253f2941295067ee78c2b6..1fcfe62cecd98bf4c6c00a5f0877d354e0c45abe 100644 (file)
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -892,6 +892,7 @@ bool io_req_post_cqe(struct io_kiocb *req, s32 res, u32 cflags)
 static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)
 {
        struct io_ring_ctx *ctx = req->ctx;
+       bool completed = true;
 
        /*
         * All execution paths but io-wq use the deferred completions by
@@ -905,18 +906,20 @@ static void io_req_complete_post(struct io_kiocb *req, unsigned issue_flags)
         * the submitter task context, IOPOLL protects with uring_lock.
         */
        if (ctx->lockless_cq || (req->flags & REQ_F_REISSUE)) {
+defer_complete:
                req->io_task_work.func = io_req_task_complete;
                io_req_task_work_add(req);
                return;
        }
 
        io_cq_lock(ctx);
-       if (!(req->flags & REQ_F_CQE_SKIP)) {
-               if (!io_fill_cqe_req(ctx, req))
-                       io_req_cqe_overflow(req);
-       }
+       if (!(req->flags & REQ_F_CQE_SKIP))
+               completed = io_fill_cqe_req(ctx, req);
        io_cq_unlock_post(ctx);
 
+       if (!completed)
+               goto defer_complete;
+
        /*
         * We don't free the request here because we know it's called from
         * io-wq only, which holds a reference, so it cannot be the last put.