net/mlx5: Fix Unbinding uplink-netdev in switchdev mode

net/mlx5: Fix Unbinding uplink-netdev in switchdev mode

It is possible to unbind the uplink ETH driver while the E-Switch is
in switchdev mode. This leads to netdevice reference counting issues[1],
as the driver removal path was not designed to clean up from this state.

During uplink ETH driver removal (_mlx5e_remove), the code now waits for
any concurrent E-Switch mode transition to finish. It then removes the
REPs auxiliary device, if exists. This ensures a graceful cleanup.

[1]
unregister_netdevice: waiting for eth2 to become free. Usage count = 2
ref_tracker: netdev@00000000c912e04b has 1/1 users at
     ib_device_set_netdev+0x130/0x270 [ib_core]
     mlx5_ib_vport_rep_load+0xf4/0x3e0 [mlx5_ib]
     mlx5_esw_offloads_rep_load+0xc7/0xe0 [mlx5_core]
     esw_offloads_enable+0x583/0x900 [mlx5_core]
     mlx5_eswitch_enable_locked+0x1b2/0x290 [mlx5_core]
     mlx5_devlink_eswitch_mode_set+0x107/0x3e0 [mlx5_core]
     devlink_nl_eswitch_set_doit+0x60/0xd0
     genl_family_rcv_msg_doit+0xe0/0x130
     genl_rcv_msg+0x183/0x290
     netlink_rcv_skb+0x4b/0xf0
     genl_rcv+0x24/0x40
     netlink_unicast+0x255/0x380
     netlink_sendmsg+0x1f3/0x420
     __sock_sendmsg+0x38/0x60
     __sys_sendto+0x119/0x180
     __x64_sys_sendto+0x20/0x30

Fixes: 7a9fb35e8c3a ("net/mlx5e: Do not reload ethernet ports when changing eswitch mode")
Signed-off-by: Shay Drory <shayd@nvidia.com>
Reviewed-by: Mark Bloch <mbloch@nvidia.com>
Signed-off-by: Tariq Toukan <tariqt@nvidia.com>
Reviewed-by: Simon Horman <horms@kernel.org>
Link: https://patch.msgid.link/1769411695-18820-2-git-send-email-tariqt@nvidia.com
Signed-off-by: Jakub Kicinski <kuba@kernel.org>