io_uring: fix mixed cqe overflow handling

I started to see zcrx data corruptions. That turned out to be due
to CQ tail pointing to a stale entry which happened to be from
a zcrx request. I.e. the tail is incremented without the CQE
memory being changed.

The culprit is __io_cqring_overflow_flush() passing "cqe32=true"
to io_get_cqe_overflow() for non-mixed CQE32 setups, which only
expects it to be set for mixed 32B CQEs and not for SETUP_CQE32.

The fix is slightly hacky, long term it's better to unify mixed and
CQE32 handling.

Fixes: e26dca67fde19 ("io_uring: add support for IORING_SETUP_CQE_MIXED")
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>

diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 296667ba712cefb59bd540565a773f6eff092e1a..02339b74ba8d4e86b43638123444ca2bb5e58331 100644 (file)
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -634,6 +634,8 @@ static void __io_cqring_overflow_flush(struct io_ring_ctx *ctx, bool dying)
                        is_cqe32 = true;
                        cqe_size <<= 1;
                }
+               if (ctx->flags & IORING_SETUP_CQE32)
+                       is_cqe32 = false;
 
                if (!dying) {
                        if (!io_get_cqe_overflow(ctx, &cqe, true, is_cqe32))