From: Willy Tarreau Date: Wed, 7 May 2014 17:22:24 +0000 (+0200) Subject: MAJOR: listener: only start listeners bound to the same processes X-Git-Tag: v1.5-dev25~19 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=ae30253c2714a67c3be4b142d61dbd4dac3c2240;p=thirdparty%2Fhaproxy.git MAJOR: listener: only start listeners bound to the same processes Now that we know what processes a "bind" statement is attached to, we have the ability to avoid starting some of them when they're not on the proper process. This feature is disabled when running in foreground however, so that debug mode continues to work with everything bound to the first and only process. The main purpose of this change is to finally allow the global stats sockets to be each bound to a different process. It can also be used to force haproxy to use different sockets in different processes for the same IP:port. The purpose is that under Linux 3.9 and above (and possibly other OSes), when multiple processes are bound to the same IP:port via different sockets, the system is capable of performing a perfect round-robin between the socket queues instead of letting any process pick all the connections from a queue. This results in a smoother load balancing and may achieve a higher performance with a large enough maxaccept setting. --- diff --git a/doc/configuration.txt b/doc/configuration.txt index 75c17bee96..806ef8b912 100644 --- a/doc/configuration.txt +++ b/doc/configuration.txt @@ -632,7 +632,8 @@ stats bind-process [ all | odd | even | [-] ] ... the stats socket to a specific set of processes, typically the first one. The warning will automatically be disabled when this setting is used, whatever the number of processes used. The maximum process ID depends on the machine's - word size (32 or 64). + word size (32 or 64). A better option consists in using the "process" setting + of the "stats socket" line to force the process on each line. ssl-default-bind-ciphers This setting is only available when support for OpenSSL was built in. It sets @@ -8284,8 +8285,13 @@ process [ all | odd | even | [-] ] remaining process, a warning is emitted, and the listener will either run on the first process of the listener if a single process was specified, or on all of its processes if multiple processes were specified. For the unlikely - case where several ranges are needed, this directive may be repeated. See - also "bind-process" and "nbproc". + case where several ranges are needed, this directive may be repeated. The + main purpose of this directive is to be used with the stats sockets and have + one different socket per process. The second purpose is to have multiple bind + lines sharing the same IP:port but not the same process in a listener, so + that the system can distribute the incoming connections into multiple queues + and allow a smoother inter-process load balancing. Currently Linux 3.9 and + above is known for supporting this. See also "bind-process" and "nbproc". ssl This setting is only available when support for OpenSSL was built in. It diff --git a/src/listener.c b/src/listener.c index 9032a875e8..ec3a39b8b6 100644 --- a/src/listener.c +++ b/src/listener.c @@ -42,15 +42,27 @@ static struct bind_kw_list bind_keywords = { /* This function adds the specified listener's file descriptor to the polling * lists if it is in the LI_LISTEN state. The listener enters LI_READY or - * LI_FULL state depending on its number of connections. + * LI_FULL state depending on its number of connections. In deamon mode, we + * also support binding only the relevant processes to their respective + * listeners. We don't do that in debug mode however. */ void enable_listener(struct listener *listener) { if (listener->state == LI_LISTEN) { - if (listener->nbconn < listener->maxconn) { + if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) && + listener->bind_conf->bind_proc && + !(listener->bind_conf->bind_proc & (1UL << (relative_pid - 1)))) { + /* we don't want to enable this listener and don't + * want any fd event to reach it. + */ + fd_stop_recv(listener->fd); + listener->state = LI_PAUSED; + } + else if (listener->nbconn < listener->maxconn) { fd_want_recv(listener->fd); listener->state = LI_READY; - } else { + } + else { listener->state = LI_FULL; } } @@ -106,12 +118,19 @@ int pause_listener(struct listener *l) * limited and disabled listeners are handled, which means that this function * may replace enable_listener(). The resulting state will either be LI_READY * or LI_FULL. 0 is returned in case of failure to resume (eg: dead socket). + * Listeners bound to a different process are not woken up unless we're in + * foreground mode. */ int resume_listener(struct listener *l) { if (l->state < LI_PAUSED) return 0; + if ((global.mode & (MODE_DAEMON | MODE_SYSTEMD)) && + l->bind_conf->bind_proc && + !(l->bind_conf->bind_proc & (1UL << (relative_pid - 1)))) + return 0; + if (l->proto->sock_prot == IPPROTO_TCP && l->state == LI_PAUSED && listen(l->fd, l->backlog ? l->backlog : l->maxconn) != 0)