From: Greg Kroah-Hartman Date: Fri, 9 Apr 2021 09:33:57 +0000 (+0200) Subject: 5.4-stable patches X-Git-Tag: v4.4.266~13 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=7e72dec47ae15f2ff2ed0db679bb58f6dec9022b;p=thirdparty%2Fkernel%2Fstable-queue.git 5.4-stable patches added patches: nvme-mpath-replace-direct_make_request-with-generic_make_request.patch --- diff --git a/queue-5.4/nvme-mpath-replace-direct_make_request-with-generic_make_request.patch b/queue-5.4/nvme-mpath-replace-direct_make_request-with-generic_make_request.patch new file mode 100644 index 00000000000..1b8d397e1ca --- /dev/null +++ b/queue-5.4/nvme-mpath-replace-direct_make_request-with-generic_make_request.patch @@ -0,0 +1,82 @@ +From sagi@grimberg.me Fri Apr 9 11:33:14 2021 +From: Sagi Grimberg +Date: Fri, 2 Apr 2021 13:08:41 -0700 +Subject: nvme-mpath: replace direct_make_request with generic_make_request +To: +Cc: Christoph Hellwig , Keith Busch , linux-nvme@lists.infradead.org +Message-ID: <20210402200841.347696-1-sagi@grimberg.me> + +From: Sagi Grimberg + +The below patches caused a regression in a multipath setup: +Fixes: 9f98772ba307 ("nvme-rdma: fix controller reset hang during traffic") +Fixes: 2875b0aecabe ("nvme-tcp: fix controller reset hang during traffic") + +These patches on their own are correct because they fixed a controller reset +regression. + +When we reset/teardown a controller, we must freeze and quiesce the namespaces +request queues to make sure that we safely stop inflight I/O submissions. +Freeze is mandatory because if our hctx map changed between reconnects, +blk_mq_update_nr_hw_queues will immediately attempt to freeze the queue, and +if it still has pending submissions (that are still quiesced) it will hang. +This is what the above patches fixed. + +However, by freezing the namespaces request queues, and only unfreezing them +when we successfully reconnect, inflight submissions that are running +concurrently can now block grabbing the nshead srcu until either we successfully +reconnect or ctrl_loss_tmo expired (or the user explicitly disconnected). + +This caused a deadlock [1] when a different controller (different path on the +same subsystem) became live (i.e. optimized/non-optimized). This is because +nvme_mpath_set_live needs to synchronize the nshead srcu before requeueing I/O +in order to make sure that current_path is visible to future (re)submisions. +However the srcu lock is taken by a blocked submission on a frozen request +queue, and we have a deadlock. + +In recent kernels (v5.9+) direct_make_request was replaced by submit_bio_noacct +which does not have this issue because it bio_list will be active when +nvme-mpath calls submit_bio_noacct on the bottom device (because it was +populated when submit_bio was triggered on it. + +Hence, we need to fix all the kernels that were before submit_bio_noacct was +introduced. + +[1]: +Workqueue: nvme-wq nvme_tcp_reconnect_ctrl_work [nvme_tcp] +Call Trace: + __schedule+0x293/0x730 + schedule+0x33/0xa0 + schedule_timeout+0x1d3/0x2f0 + wait_for_completion+0xba/0x140 + __synchronize_srcu.part.21+0x91/0xc0 + synchronize_srcu_expedited+0x27/0x30 + synchronize_srcu+0xce/0xe0 + nvme_mpath_set_live+0x64/0x130 [nvme_core] + nvme_update_ns_ana_state+0x2c/0x30 [nvme_core] + nvme_update_ana_state+0xcd/0xe0 [nvme_core] + nvme_parse_ana_log+0xa1/0x180 [nvme_core] + nvme_read_ana_log+0x76/0x100 [nvme_core] + nvme_mpath_init+0x122/0x180 [nvme_core] + nvme_init_identify+0x80e/0xe20 [nvme_core] + nvme_tcp_setup_ctrl+0x359/0x660 [nvme_tcp] + nvme_tcp_reconnect_ctrl_work+0x24/0x70 [nvme_tcp] + +Signed-off-by: Sagi Grimberg +Signed-off-by: Greg Kroah-Hartman + +--- + drivers/nvme/host/multipath.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +--- a/drivers/nvme/host/multipath.c ++++ b/drivers/nvme/host/multipath.c +@@ -330,7 +330,7 @@ static blk_qc_t nvme_ns_head_make_reques + trace_block_bio_remap(bio->bi_disk->queue, bio, + disk_devt(ns->head->disk), + bio->bi_iter.bi_sector); +- ret = direct_make_request(bio); ++ ret = generic_make_request(bio); + } else if (nvme_available_path(head)) { + dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n"); + diff --git a/queue-5.4/series b/queue-5.4/series index ba389a891d0..05c5945c12b 100644 --- a/queue-5.4/series +++ b/queue-5.4/series @@ -18,3 +18,4 @@ cifs-revalidate-mapping-when-we-open-files-for-smb1-.patch cifs-silently-ignore-unknown-oplock-break-handle.patch bpf-x86-validate-computation-of-branch-displacements-for-x86-64.patch bpf-x86-validate-computation-of-branch-displacements-for-x86-32.patch +nvme-mpath-replace-direct_make_request-with-generic_make_request.patch