net/mlx5e: Avoid copying payload to the skb's linear part

mlx5e_skb_from_cqe_mpwrq_nonlinear() copies MLX5E_RX_MAX_HEAD (256)
bytes from the page-pool to the skb's linear part. Those 256 bytes
include part of the payload.

When attempting to do GRO in skb_gro_receive, if headlen > data_offset
(and skb->head_frag is not set), we end up aggregating packets in the
frag_list.

This is of course not good when we are CPU-limited. Also causes a worse
skb->len/truesize ratio,...

So, let's avoid copying parts of the payload to the linear part. We use
eth_get_headlen() to parse the headers and compute the length of the
protocol headers, which will be used to copy the relevant bits of the
skb's linear part.

We still allocate MLX5E_RX_MAX_HEAD for the skb so that if the networking
stack needs to call pskb_may_pull() later on, we don't need to reallocate
memory.

This gives a nice throughput increase (ARM Neoverse-V2 with CX-7 NIC and
LRO enabled):

BEFORE:
=======
(netserver pinned to core receiving interrupts)
$ netperf -H 10.221.81.118 -T 80,9 -P 0 -l 60 -- -m 256K -M 256K
 87380  16384 262144    60.01    32547.82

(netserver pinned to adjacent core receiving interrupts)
$ netperf -H 10.221.81.118 -T 80,10 -P 0 -l 60 -- -m 256K -M 256K
 87380  16384 262144    60.00    52531.67

AFTER:
======
(netserver pinned to core receiving interrupts)
$ netperf -H 10.221.81.118 -T 80,9 -P 0 -l 60 -- -m 256K -M 256K
 87380  16384 262144    60.00    52896.06

(netserver pinned to adjacent core receiving interrupts)
 $ netperf -H 10.221.81.118 -T 80,10 -P 0 -l 60 -- -m 256K -M 256K
 87380  16384 262144    60.00    85094.90

Additional tests across a larger range of parameters w/ and w/o LRO, w/
and w/o IPv6-encapsulation, different MTUs (1500, 4096, 9000), different
TCP read/write-sizes as well as UDP benchmarks, all have shown equal or
better performance with this patch.

For XDP pull at most ETH_HLEN bytes in the linear area so that XDP_PASS
can also benefit from this improvement and keep things simple when
dealing with skb geometry changes from the XDP program.

Reviewed-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Saeed Mahameed <saeedm@nvidia.com>
Signed-off-by: Christoph Paasch <cpaasch@openai.com>
Signed-off-by: Dragos Tatulea <dtatulea@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Link: https://patch.msgid.link/20260601061522.398044-3-tariqt@nvidia.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 75ccf40a7f173341b615745326f94246c9a2d282..6fbc0441c4b845b5f4f6276b55d3d2e9ac45fa9e 100644 (file)
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -1912,7 +1912,6 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
                                   u32 page_idx)
 {
        struct mlx5e_frag_page *frag_page = &wi->alloc_units.frag_pages[page_idx];
-       u16 headlen = min_t(u16, MLX5E_RX_MAX_HEAD, cqe_bcnt);
        struct mlx5e_frag_page *head_page = frag_page;
        struct mlx5e_frag_page *linear_page = NULL;
        struct mlx5e_xdp_buff *mxbuf = &rq->mxbuf;
@@ -1928,6 +1927,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
        u32 linear_frame_sz;
        u16 linear_data_len;
        u16 linear_hr;
+       u16 headlen;
 
        if (unlikely(cqe_bcnt > rq->hw_mtu)) {
                u8 lro_num_seg = get_cqe_lro_num_seg(cqe);
@@ -1971,11 +1971,14 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
                net_prefetchw(va); /* xdp_frame data area */
                net_prefetchw(skb->data);
 
+               headlen = min(MLX5E_RX_MAX_HEAD, cqe_bcnt);
                addr = page_pool_get_dma_addr_netmem(head_page->netmem);
                dma_sync_single_for_cpu(rq->pdev, addr + head_offset,
                                        ALIGN(headlen, sizeof(long)),
                                        rq->buff.map_dir);
 
+               headlen = eth_get_headlen(rq->netdev, head_addr, headlen);
+
                frag_offset += headlen;
                byte_cnt -= headlen;
                linear_hr = skb_headroom(skb);
@@ -2060,9 +2063,9 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w
                                pagep->frags++;
                        while (++pagep < frag_page);
 
-                       headlen = min_t(u16, MLX5E_RX_MAX_HEAD - len,
-                                       skb->data_len);
-                       __pskb_pull_tail(skb, headlen);
+                       if (len < ETH_HLEN)
+                               __pskb_pull_tail(skb, min(ETH_HLEN - len,
+                                                         skb->data_len));
                }
        } else {
                if (xdp_buff_has_frags(&mxbuf->xdp)) {