MAJOR: fd: remove pending updates upon real close

Dealing with long-lasting updates that outlive a close() is always
going to be quite a problem, not because of the thread that will
discover such updates late, but mostly due to the shared update_list
that will have an entry on hold making it difficult to reuse it, and
requiring that the fd's tgid is changed and the update_mask reset
from a safe location.

After careful inspection, it turns out that all our pollers that support
automatic event removal upon close() do not need any extra bookkeeping,
and that poll and select that use an internal representation already
provide a poller->clo() callback that is already used to update the
local event. As such, it is already safe to reset the update mask and
to remove the event from the shared list just before the final close,
because nothing remains to be done with this FD by the poller.

Doing so considerably simplifies the handling of updates, which will
only have to be inspected by the pollers, while the writers can continue
to consider that the entries are always valid. Another benefit is that
it will be possible to reduce contention on the update_list by just
having one update_list per group (left to be done later if needed).

diff --git a/src/fd.c b/src/fd.c
index ee6cbd2a7b4999e01d872390a0f7cd5bc8a63bbd..04a1c7b04a3123a87e5ea60f9f44a24c1ff01f37 100644 (file)
--- a/src/fd.c
+++ b/src/fd.c
@@ -309,9 +309,20 @@ void _fd_delete_orphan(int fd)
                DISGUISE(setsockopt(fd, SOL_SOCKET, SO_LINGER,
                           (struct linger *) &nolinger, sizeof(struct linger)));
        }
+
+       /* It's expected that a close() will result in the FD disappearing from
+        * pollers, but some pollers may have some internal bookkeeping to be
+        * done prior to the call (e.g. remove references from internal tables).
+        */
        if (cur_poller.clo)
                cur_poller.clo(fd);
 
+       /* we don't want this FD anymore in the global list */
+       fd_rm_from_fd_list(&update_list, fd);
+
+       /* no more updates on this FD are relevant anymore */
+       HA_ATOMIC_STORE(&fdtab[fd].update_mask, 0);
+
        port_range_release_port(fdinfo[fd].port_range, fdinfo[fd].local_port);
        polled_mask[fd].poll_recv = polled_mask[fd].poll_send = 0;
 
@@ -322,6 +333,7 @@ void _fd_delete_orphan(int fd)
 #endif
        fdinfo[fd].port_range = NULL;
        fdtab[fd].owner = NULL;
+
        /* perform the close() call last as it's what unlocks the instant reuse
         * of this FD by any other thread.
         */