From: Greg Kroah-Hartman Date: Sun, 7 Mar 2021 15:00:38 +0000 (+0100) Subject: 4.4-stable patches X-Git-Tag: v5.4.104~23 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=12cca9831f6e955b2c987d65028d47382e9a4c8f;p=thirdparty%2Fkernel%2Fstable-queue.git 4.4-stable patches added patches: futex-fix-irq-self-deadlock-and-satisfy-assertion.patch futex-fix-spin_lock-spin_unlock_irq-imbalance.patch --- diff --git a/queue-4.4/futex-fix-irq-self-deadlock-and-satisfy-assertion.patch b/queue-4.4/futex-fix-irq-self-deadlock-and-satisfy-assertion.patch new file mode 100644 index 00000000000..fd9672417dc --- /dev/null +++ b/queue-4.4/futex-fix-irq-self-deadlock-and-satisfy-assertion.patch @@ -0,0 +1,184 @@ +From foo@baz Sun Mar 7 03:58:59 PM CET 2021 +From: Thomas Schoebel-Theuer +Date: Sun, 7 Mar 2021 08:26:29 +0100 +Subject: futex: fix irq self-deadlock and satisfy assertion +To: tst@schoebel-theuer.de, stable@vger.kernel.org, Greg Kroah-Hartman , Lee Jones , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Thomas Schoebel-Theuer , Arnd Bergmann , linux-kernel@vger.kernel.org (open list) +Message-ID: <20210307072629.5955-1-tst@schoebel-theuer.de> + +From: Thomas Schoebel-Theuer + +This patch and problem analysis is specific for 4.4 LTS, due to incomplete +backporting of other fixes. Later LTS series have different backports. + +Since v4.4.257 when CONFIG_PROVE_LOCKING=y +the following triggers right after reboot of our pre-life systems +which equal our production setup: + +Mar 03 11:27:33 icpu-test-bap10 kernel: ================================= +Mar 03 11:27:33 icpu-test-bap10 kernel: [ INFO: inconsistent lock state ] +Mar 03 11:27:33 icpu-test-bap10 kernel: 4.4.259-rc1-grsec+ #730 Not tainted +Mar 03 11:27:33 icpu-test-bap10 kernel: --------------------------------- +Mar 03 11:27:33 icpu-test-bap10 kernel: inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage. +Mar 03 11:27:33 icpu-test-bap10 kernel: apache2-ssl/9310 [HC0[0]:SC0[0]:HE1:SE1] takes: +Mar 03 11:27:33 icpu-test-bap10 kernel: (&p->pi_lock){?.-.-.}, at: [] pi_state_update_owner+0x51/0xd7 +Mar 03 11:27:33 icpu-test-bap10 kernel: {IN-HARDIRQ-W} state was registered at: +Mar 03 11:27:33 icpu-test-bap10 kernel: [] __lock_acquire+0x3a7/0xe4a +Mar 03 11:27:33 icpu-test-bap10 kernel: [] lock_acquire+0x18d/0x1bc +Mar 03 11:27:33 icpu-test-bap10 kernel: [] _raw_spin_lock_irqsave+0x3e/0x50 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] try_to_wake_up+0x2c/0x210 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] default_wake_function+0xd/0xf +Mar 03 11:27:33 icpu-test-bap10 kernel: [] autoremove_wake_function+0x11/0x35 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] __wake_up_common+0x48/0x7c +Mar 03 11:27:33 icpu-test-bap10 kernel: [] __wake_up+0x34/0x46 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] megasas_complete_int_cmd+0x31/0x33 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] megasas_complete_cmd+0x570/0x57b +Mar 03 11:27:33 icpu-test-bap10 kernel: [] complete_cmd_fusion+0x23e/0x33d +Mar 03 11:27:33 icpu-test-bap10 kernel: [] megasas_isr_fusion+0x67/0x74 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_irq_event_percpu+0x134/0x311 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_irq_event+0x33/0x51 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_edge_irq+0xa3/0xc2 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] handle_irq+0xf9/0x101 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] do_IRQ+0x80/0xf5 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ret_from_intr+0x0/0x20 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] arch_cpu_idle+0xa/0xc +Mar 03 11:27:33 icpu-test-bap10 kernel: [] default_idle_call+0x1e/0x20 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] cpu_startup_entry+0x141/0x22f +Mar 03 11:27:33 icpu-test-bap10 kernel: [] rest_init+0x135/0x13b +Mar 03 11:27:33 icpu-test-bap10 kernel: [] start_kernel+0x3fa/0x40a +Mar 03 11:27:33 icpu-test-bap10 kernel: [] x86_64_start_reservations+0x2a/0x2c +Mar 03 11:27:33 icpu-test-bap10 kernel: [] x86_64_start_kernel+0x11f/0x12c +Mar 03 11:27:33 icpu-test-bap10 kernel: irq event stamp: 1457 +Mar 03 11:27:33 icpu-test-bap10 kernel: hardirqs last enabled at (1457): [] get_user_pages_fast+0xeb/0x14f +Mar 03 11:27:33 icpu-test-bap10 kernel: hardirqs last disabled at (1456): [] get_user_pages_fast+0x5f/0x14f +Mar 03 11:27:33 icpu-test-bap10 kernel: softirqs last enabled at (1446): [] release_sock+0x142/0x14d +Mar 03 11:27:33 icpu-test-bap10 kernel: softirqs last disabled at (1444): [] release_sock+0x34/0x14d +Mar 03 11:27:33 icpu-test-bap10 kernel: + other info that might help us debug this: +Mar 03 11:27:33 icpu-test-bap10 kernel: Possible unsafe locking scenario: +Mar 03 11:27:33 icpu-test-bap10 kernel: CPU0 +Mar 03 11:27:33 icpu-test-bap10 kernel: ---- +Mar 03 11:27:33 icpu-test-bap10 kernel: lock(&p->pi_lock); +Mar 03 11:27:33 icpu-test-bap10 kernel: +Mar 03 11:27:33 icpu-test-bap10 kernel: lock(&p->pi_lock); +Mar 03 11:27:33 icpu-test-bap10 kernel: + *** DEADLOCK *** +Mar 03 11:27:33 icpu-test-bap10 kernel: 2 locks held by apache2-ssl/9310: +Mar 03 11:27:33 icpu-test-bap10 kernel: #0: (&(&(__futex_data.queues)[i].lock)->rlock){+.+...}, at: [] do +Mar 03 11:27:33 icpu-test-bap10 kernel: #1: (&lock->wait_lock){+.+...}, at: [] do_futex+0x639/0x809 +Mar 03 11:27:33 icpu-test-bap10 kernel: + stack backtrace: +Mar 03 11:27:33 icpu-test-bap10 kernel: CPU: 13 PID: 9310 UID: 99 Comm: apache2-ssl Not tainted 4.4.259-rc1-grsec+ #730 +Mar 03 11:27:33 icpu-test-bap10 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.11.0 11/02/2019 +Mar 03 11:27:33 icpu-test-bap10 kernel: 0000000000000000 ffff883fb79bfc00 ffffffff816f8fc2 ffff883ffa66d300 +Mar 03 11:27:33 icpu-test-bap10 kernel: ffffffff8eaa71f0 ffff883fb79bfc50 ffffffff81088484 0000000000000000 +Mar 03 11:27:33 icpu-test-bap10 kernel: 0000000000000001 0000000000000001 0000000000000002 ffff883ffa66db58 +Mar 03 11:27:33 icpu-test-bap10 kernel: Call Trace: +Mar 03 11:27:33 icpu-test-bap10 kernel: [] dump_stack+0x94/0xca +Mar 03 11:27:33 icpu-test-bap10 kernel: [] print_usage_bug+0x1bc/0x1d1 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? check_usage_forwards+0x98/0x98 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] mark_lock+0x10c/0x203 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] __lock_acquire+0x416/0xe4a +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x51/0xd7 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] lock_acquire+0x18d/0x1bc +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? lock_acquire+0x18d/0x1bc +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x51/0xd7 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] _raw_spin_lock+0x2a/0x39 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x51/0xd7 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] pi_state_update_owner+0x51/0xd7 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] do_futex+0x6ae/0x809 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] SyS_futex+0x133/0x143 +Mar 03 11:27:33 icpu-test-bap10 kernel: [] ? syscall_trace_enter_phase2+0x1a2/0x1bb +Mar 03 11:27:33 icpu-test-bap10 kernel: [] tracesys_phase2+0x90/0x95 + +Bisecting detects 47e452fcf2f +in the above specific scenario using apache-ssl, +but apparently the missing *_irq() was introduced in +34c8e1c2c02. + +However, just reverting the old _irq() variants to a similar status +than before 34c8e1c2c02, +or using _irqsave() / _irqrestore() as some other backports are doing +in various places, would not really help. + +The fundamental problem is the following violation of the assertion +lockdep_assert_held(&pi_state->pi_mutex.wait_lock) in pi_state_update_owner(): + +Mar 03 12:50:03 icpu-test-bap10 kernel: ------------[ cut here ]------------ +Mar 03 12:50:03 icpu-test-bap10 kernel: WARNING: CPU: 37 PID: 8488 at kernel/futex.c:844 pi_state_update_owner+0x3d/0xd7() +Mar 03 12:50:03 icpu-test-bap10 kernel: Modules linked in: xt_time xt_connlimit xt_connmark xt_NFLOG xt_limit xt_hashlimit veth ip_set_bitmap_port xt_DSCP xt_multiport ip_set_hash_ip xt_owner xt_set ip_set_hash_net xt_state xt_conntrack nf_conntrack_ftp mars lz4_decompress lz4_compress ipmi_devintf x86_pkg_temp_thermal coretemp crct10dif_pclmul crc32_pclmul hed ipmi_si ipmi_msghandler processor crc32c_intel ehci_pci ehci_hcd usbcore i40e usb_common +Mar 03 12:50:03 icpu-test-bap10 kernel: CPU: 37 PID: 8488 UID: 99 Comm: apache2-ssl Not tainted 4.4.259-rc1-grsec+ #737 +Mar 03 12:50:03 icpu-test-bap10 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.11.0 11/02/2019 +Mar 03 12:50:03 icpu-test-bap10 kernel: 0000000000000000 ffff883f863f7c70 ffffffff816f9002 0000000000000000 +Mar 03 12:50:03 icpu-test-bap10 kernel: 0000000000000009 ffff883f863f7ca8 ffffffff8104cda2 ffffffff810abac7 +Mar 03 12:50:03 icpu-test-bap10 kernel: ffff883ffbfe5e80 0000000000000000 ffff883f82ed4bc0 00007fc01c9bf000 +Mar 03 12:50:03 icpu-test-bap10 kernel: Call Trace: +Mar 03 12:50:03 icpu-test-bap10 kernel: [] dump_stack+0x94/0xca +Mar 03 12:50:03 icpu-test-bap10 kernel: [] warn_slowpath_common+0x94/0xad +Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? pi_state_update_owner+0x3d/0xd7 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] warn_slowpath_null+0x15/0x17 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] pi_state_update_owner+0x3d/0xd7 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] free_pi_state+0x2d/0x73 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] unqueue_me_pi+0x1d/0x31 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] futex_lock_pi+0x27a/0x2e8 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? __lock_acquire+0x327/0xe4a +Mar 03 12:50:03 icpu-test-bap10 kernel: [] do_futex+0x784/0x809 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? seccomp_phase1+0xde/0x1e7 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? current_kernel_time64+0xb/0x31 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? current_kernel_time+0xb/0xf +Mar 03 12:50:03 icpu-test-bap10 kernel: [] SyS_futex+0x133/0x143 +Mar 03 12:50:03 icpu-test-bap10 kernel: [] ? syscall_trace_enter_phase2+0x1a2/0x1bb +Mar 03 12:50:03 icpu-test-bap10 kernel: [] tracesys_phase2+0x90/0x95 +Mar 03 12:50:03 icpu-test-bap10 kernel: ---[ end trace 968f95a458dea951 ]--- + +In order to both (1) prevent the self-deadlock, and (2) to satisfy the assertion +at pi_state_update_owner(), some locking with irq disable is needed, +at least in the specific call stack. + +Interestingly, there existed a suchalike locking just before +f08a4af5ccb. + +This is just a quick hotfix, resurrecting some previous +locks at the old places, but now using ->wait_lock in place +of the previous ->pi_lock (which was in place before +f08a4af5ccb). + +The ->pi_lock is now also taken, by the new code +which had been introduced in +34c8e1c2c02. + +When this patch is applied, both the above splats are +no longer triggering at my prelife machines. + +Without this patch, I cannot ensure stable production at +1&1 Ionos. + +Hint for further work: I have not yet tested other call paths, +since I am under time pressure for security reasons. + +Hint for further hardening of 4.4.y and probably some more LTS series: +Probably some more systematic testing with CONFIG_PROVE_LOCKING +(and probably some more options) should be invested +in order to make the 4.4 LTS series really "stable" again. + +Signed-off-by: Thomas Schoebel-Theuer +Cc: Thomas Gleixner +Cc: Lee Jones +Fixes: f08a4af5ccb2 ("futex: Use pi_state_update_owner() in put_pi_state()") +Fixes: 34c8e1c2c025 ("futex: Provide and use pi_state_update_owner()") +Signed-off-by: Greg Kroah-Hartman +--- + kernel/futex.c | 2 ++ + 1 file changed, 2 insertions(+) + +--- a/kernel/futex.c ++++ b/kernel/futex.c +@@ -874,7 +874,9 @@ static void free_pi_state(struct futex_p + * and has cleaned up the pi_state already + */ + if (pi_state->owner) { ++ raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock); + pi_state_update_owner(pi_state, NULL); ++ raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock); + rt_mutex_proxy_unlock(&pi_state->pi_mutex); + } + diff --git a/queue-4.4/futex-fix-spin_lock-spin_unlock_irq-imbalance.patch b/queue-4.4/futex-fix-spin_lock-spin_unlock_irq-imbalance.patch new file mode 100644 index 00000000000..ce440f719e9 --- /dev/null +++ b/queue-4.4/futex-fix-spin_lock-spin_unlock_irq-imbalance.patch @@ -0,0 +1,66 @@ +From foo@baz Sun Mar 7 03:58:59 PM CET 2021 +From: Thomas Schoebel-Theuer +Date: Sun, 7 Mar 2021 08:23:22 +0100 +Subject: futex: fix spin_lock() / spin_unlock_irq() imbalance +To: tst@schoebel-theuer.de, stable@vger.kernel.org, Greg Kroah-Hartman , Lee Jones , Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Thomas Schoebel-Theuer , Arnd Bergmann , linux-kernel@vger.kernel.org (open list) +Message-ID: <20210307072322.5884-1-tst@schoebel-theuer.de> + +From: Thomas Schoebel-Theuer + +This patch and problem analysis is specific for 4.4 LTS, due to incomplete +backporting of other fixes. Later LTS series have different backports. + +The following is obviously incorrect: + +static int wake_futex_pi(u32 __user *uaddr, u32 uval, struct futex_q *this, + struct futex_hash_bucket *hb) +{ +[...] + raw_spin_lock(&pi_state->pi_mutex.wait_lock); +[...] + raw_spin_unlock_irq(&pi_state->pi_mutex.wait_lock); +[...] +} + +The 4.4-specific fix should probably go in the direction of +b4abf91047c, +making everything irq-safe. + +Probably, backporting of b4abf91047c +to 4.4 LTS could thus be another good idea. + +However, this might involve some more 4.4-specific work and +require thorough testing: + +> git log --oneline v4.4..b4abf91047c -- kernel/futex.c kernel/locking/rtmutex.c | wc -l +10 + +So this patch is just an obvious quickfix for now. + +Hint: the lock order is documented in 4.9.y and later. A similar +documenting is missing in 4.4.y. Please somebody either backport also, +or write a new description, if there would be some differences I cannot +easily see at the moment. Without reliable docs, +inspection of the locking correctness may become a pain. + +Signed-off-by: Thomas Schoebel-Theuer +Cc: Thomas Gleixner +Cc: Lee Jones +Fixes: 394fc4981426 ("futex: Rework inconsistent rt_mutex/futex_q state") +Fixes: 6510e4a2d04f ("futex,rt_mutex: Provide futex specific rt_mutex API") +Signed-off-by: Greg Kroah-Hartman +--- + kernel/futex.c | 2 +- + 1 file changed, 1 insertion(+), 1 deletion(-) + +--- a/kernel/futex.c ++++ b/kernel/futex.c +@@ -1408,7 +1408,7 @@ static int wake_futex_pi(u32 __user *uad + if (pi_state->owner != current) + return -EINVAL; + +- raw_spin_lock(&pi_state->pi_mutex.wait_lock); ++ raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock); + new_owner = rt_mutex_next_owner(&pi_state->pi_mutex); + + /* diff --git a/queue-4.4/series b/queue-4.4/series index e69de29bb2d..5d1183c7d64 100644 --- a/queue-4.4/series +++ b/queue-4.4/series @@ -0,0 +1,2 @@ +futex-fix-irq-self-deadlock-and-satisfy-assertion.patch +futex-fix-spin_lock-spin_unlock_irq-imbalance.patch