src/patches/suse-2.6.27.39/patches.fixes/blk-timeout-splice-timeout-list

   1 From: Hannes Reinecke <hare@suse.de>
   2 Subject: Deadlock during multipath failover
   3 References: bnc#475107
   4
   5 During multipath failover tests with SCSI on System z, the kernel
   6 deadlocks in this situation:
   7
   8 >  STACK:
   9 >  0 blk_add_timer+206 [0x2981ea]
  10 >  1 blk_rq_timed_out+132 [0x2982a8]
  11 >  2 blk_abort_request+114 [0x29833e]
  12 >  3 blk_abort_queue+92 [0x2983a8]
  13 >  4 deactivate_path+74 [0x3e00009625a]
  14 >  5 run_workqueue+236 [0x149e04]
  15 >  6 worker_thread+294 [0x149fce]
  16 >  7 kthread+110 [0x14f436]
  17 >  8 kernel_thread_starter+6 [0x10941a]
  18
  19 blk_abort_queue takes the queue_lock with spinlock_irqsave and
  20 walks the timer_list with list_for_each_entry_safe. Since a
  21 path to a SCSI device just failed, the rport state is
  22 FC_PORTSTATE_BLOCKED. This rport state triggers blk_add_timer()
  23 that calls list_add_tail() to move the request to the end of
  24 timer_list. Thus, the list_for_each_entry_safe never reaches
  25 the end of the timer_list, it continously moves the requests
  26 to the end of the list.
  27
  28 The rport state FC_PORTSTATE_BLOCKED would end when the function
  29 fc_timeout_deleted_rport() would run to remove the rport. But
  30 this function was schedules from queue_delayed_work.
  31 The timer already expired, but the timer function does not run,
  32 because the timer interrupt is disabled from the
  33 spinlock_irqsave call.
  34
  35 But just using a list_splice_init() here we will be traversing
  36 our private list and break the deadlock.
  37 And the timer would be triggered correctly as blk_add_timer()
  38 will always add a one second delay here, during which we should
  39 be able to process the list.
  40
  41 Signed-off-by: Hannes Reinecke <hare@suse.de>
  42
  43 ---
  44  block/blk-timeout.c |    5 ++++-
  45  1 file changed, 4 insertions(+), 1 deletion(-)
  46
  47 --- a/block/blk-timeout.c
  48 +++ b/block/blk-timeout.c
  49 @@ -150,12 +150,15 @@ void blk_abort_queue(struct request_queu
  50  {
  51         unsigned long flags;
  52         struct request *rq, *tmp;
  53 +       LIST_HEAD(list);
  54
  55         spin_lock_irqsave(q->queue_lock, flags);
  56
  57         elv_abort_queue(q);
  58
  59 -       list_for_each_entry_safe(rq, tmp, &q->timeout_list, timeout_list)
  60 +       list_splice_init(&q->timeout_list, &list);
  61 +
  62 +       list_for_each_entry_safe(rq, tmp, &list, timeout_list)
  63                 blk_abort_request(rq);
  64
  65         spin_unlock_irqrestore(q->queue_lock, flags);