iov_iter: Convert copy_from_user_iter() to masked user access

copy_from_user_iter() lacks a speculation barrier, which will degrade
performance on some architecture like x86, which would be unfortunate as
copy_from_user_iter() is a critical hotpath function.

Convert copy_from_user_iter() to using masked user access on architecture
that support it. This allows to add the speculation barrier without
impacting performance.

This is similar to what was done for copy_from_user() in commit
0fc810ae3ae1 ("x86/uaccess: Avoid barrier_nospec() in 64-bit
copy_from_user()")

[ tglx: Massage change log ]

Signed-off-by: Christophe Leroy <christophe.leroy@csgroup.eu>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://patch.msgid.link/58e4b07d469ca68a2b9477fe2c1ccc8a44cef131.1763396724.git.christophe.leroy@csgroup.eu

diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 2fe66a6b8789e7c6c2f262768732810472cb1477..a589935bf302587a3721eed03291298ba0a62cab 100644 (file)
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -49,12 +49,16 @@ size_t copy_from_user_iter(void __user *iter_from, size_t progress,
 
        if (should_fail_usercopy())
                return len;
-       if (access_ok(iter_from, len)) {
-               to += progress;
-               instrument_copy_from_user_before(to, iter_from, len);
-               res = raw_copy_from_user(to, iter_from, len);
-               instrument_copy_from_user_after(to, iter_from, len, res);
-       }
+       if (can_do_masked_user_access())
+               iter_from = mask_user_address(iter_from);
+       else if (!access_ok(iter_from, len))
+               return res;
+
+       to += progress;
+       instrument_copy_from_user_before(to, iter_from, len);
+       res = raw_copy_from_user(to, iter_from, len);
+       instrument_copy_from_user_after(to, iter_from, len, res);
+
        return res;
 }