]>
Commit | Line | Data |
---|---|---|
809ae055 SL |
1 | From c4d4bea2c452e16e0ff6efe8b3d148ac188f3e8a Mon Sep 17 00:00:00 2001 |
2 | From: Davidlohr Bueso <dave@stgolabs.net> | |
3 | Date: Thu, 3 Jan 2019 15:27:09 -0800 | |
4 | Subject: fs/epoll: drop ovflist branch prediction | |
5 | ||
6 | [ Upstream commit 76699a67f3041ff4c7af6d6ee9be2bfbf1ffb671 ] | |
7 | ||
8 | The ep->ovflist is a secondary ready-list to temporarily store events | |
9 | that might occur when doing sproc without holding the ep->wq.lock. This | |
10 | accounts for every time we check for ready events and also send events | |
11 | back to userspace; both callbacks, particularly the latter because of | |
12 | copy_to_user, can account for a non-trivial time. | |
13 | ||
14 | As such, the unlikely() check to see if the pointer is being used, seems | |
15 | both misleading and sub-optimal. In fact, we go to an awful lot of | |
16 | trouble to sync both lists, and populating the ovflist is far from an | |
17 | uncommon scenario. | |
18 | ||
19 | For example, profiling a concurrent epoll_wait(2) benchmark, with | |
20 | CONFIG_PROFILE_ANNOTATED_BRANCHES shows that for a two threads a 33% | |
21 | incorrect rate was seen; and when incrementally increasing the number of | |
22 | epoll instances (which is used, for example for multiple queuing load | |
23 | balancing models), up to a 90% incorrect rate was seen. | |
24 | ||
25 | Similarly, by deleting the prediction, 3% throughput boost was seen | |
26 | across incremental threads. | |
27 | ||
28 | Link: http://lkml.kernel.org/r/20181108051006.18751-4-dave@stgolabs.net | |
29 | Signed-off-by: Davidlohr Bueso <dbueso@suse.de> | |
30 | Reviewed-by: Andrew Morton <akpm@linux-foundation.org> | |
31 | Cc: Al Viro <viro@zeniv.linux.org.uk> | |
32 | Cc: Jason Baron <jbaron@akamai.com> | |
33 | Signed-off-by: Andrew Morton <akpm@linux-foundation.org> | |
34 | Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> | |
35 | Signed-off-by: Sasha Levin <sashal@kernel.org> | |
36 | --- | |
37 | fs/eventpoll.c | 2 +- | |
38 | 1 file changed, 1 insertion(+), 1 deletion(-) | |
39 | ||
40 | diff --git a/fs/eventpoll.c b/fs/eventpoll.c | |
41 | index 1b08556776ce..240d9ceb8d0c 100644 | |
42 | --- a/fs/eventpoll.c | |
43 | +++ b/fs/eventpoll.c | |
44 | @@ -1034,7 +1034,7 @@ static int ep_poll_callback(wait_queue_t *wait, unsigned mode, int sync, void *k | |
45 | * semantics). All the events that happen during that period of time are | |
46 | * chained in ep->ovflist and requeued later on. | |
47 | */ | |
48 | - if (unlikely(ep->ovflist != EP_UNACTIVE_PTR)) { | |
49 | + if (ep->ovflist != EP_UNACTIVE_PTR) { | |
50 | if (epi->next == EP_UNACTIVE_PTR) { | |
51 | epi->next = ep->ovflist; | |
52 | ep->ovflist = epi; | |
53 | -- | |
54 | 2.19.1 | |
55 |