Willy Tarreau [Thu, 4 Oct 2012 22:04:16 +0000 (00:04 +0200)]
MEDIUM: checks: enable the PROXY protocol with health checks
When health checks are configured on a server which has the send-proxy
directive and no "port" nor "addr" settings, the health check connections
will automatically use the PROXY protocol. If "port" or "addr" are set,
the "check-send-proxy" directive may be used to force the protocol.
MAJOR: checks: completely use the connection transport layer
With this change, we now use the connection's transport layer to receive
and send data during health checks. It even becomes possible to send data
in multiple times, which was not possible before.
The transport layer used is the same as the one used for the traffic, unless
a specific address and/or port is specified for the checks using "port" or
"addr", in which case the transport layer defaults to raw_sock. An option
will be provided to force SSL checks on different IP/ports later.
Connection errors and timeouts are still reported.
Some situations where strerror() was able to report a precise error after
a failed connect() in the past might not be reported with as much precision
anymore, but the error message was already meaningless. During the tests,
no situation was found where a message became less precise.
MEDIUM: check: add the ctrl and transport layers in the server check structure
Since it's possible for the checks to use a different protocol or transport layer
than the prod traffic, we need to have them referenced in the server. The
SSL checks are not enabled yet, but the transport layers are completely used.
MEDIUM: checks: use real buffers to store requests and responses
Till now the request was made in the trash and sent to the network at
once, and the response was read into a preallocated char[]. Now we
allocate a full buffer for both the request and the response, and make
use of it.
Some of the operations will probably be replaced later with buffer macros
but the point was to ensure we could migrate to use the data layers soon.
One nice improvement caused by this change is that requests are now formed
at the beginning of the check and may safely be sent in multiple chunks if
needed.
REORG: server: move the check-specific parts into a check subsection
The health checks in the servers are becoming a real mess, move them
into their own subsection. We'll soon need to have a struct buffer to
replace the char * as well as check-specific protocol and transport
layers.
MAJOR: checks: make use of the connection layer to send checks
This is a first step, we now use the connection layer without the data
layers (send/recv are still used by hand). The connection is established
using tcp_connect_server() and raw_sock is assumed and forced for now.
fdtab is not manipulated anymore and polling is managed via the connection
layer.
It becomes quite clear that the server needs a second ->ctrl and ->xprt
dedicated to the checks.
Willy Tarreau [Thu, 4 Oct 2012 21:55:57 +0000 (23:55 +0200)]
MEDIUM: connection: add a new local send-proxy transport callback
This callback sends a PROXY protocol line on the outgoing connection,
with the local and remote endpoint information. This is used for local
connections (eg: health checks) where the other end needs to have a
valid address and no connection is relayed.
Willy Tarreau [Thu, 4 Oct 2012 22:10:55 +0000 (00:10 +0200)]
REORG: connection: move the PROXY protocol management to connection.c
It was previously in frontend.c but there is no reason for this anymore
considering that all the information involved is in the connection itself
only. Theorically this should be in the socket layer but we don't have
this yet.
Willy Tarreau [Thu, 4 Oct 2012 20:21:15 +0000 (22:21 +0200)]
MEDIUM: connection: automatically disable polling on error
We absolutely want to disable FD polling after an error is detected,
otherwise the data layer has to do it and it's far from being obvious
at these layers.
The way we did it was a bit tricky in conn_update_*_polling and
conn_*_polling_changes. However it has almost no impact on performance
and code size both for the fast and slow path.
We'll now be able to remove some flag updates in the stream interface.
Willy Tarreau [Thu, 4 Oct 2012 18:20:46 +0000 (20:20 +0200)]
MEDIUM: connection: it's not the data layer's role to validate the connection
Till now we used to perform the L4_CONN check in the data layer
(eg: stream interface) but that does not make sense, because some transport
layers will imply that the connection is opened (eg: SSL), and also because
the complexity to check for this is higher in the data layer than in the
transport layer. This is so much true that some read0 cases did not validate
the connection.
So as of now, the transport layer is responsible for clearing L4_CONN when
it detects an activity, and the data layer may safely rely on this flag. This
only impacts a minor change in raw_sock and stream_interface for now.
Willy Tarreau [Wed, 3 Oct 2012 19:17:23 +0000 (21:17 +0200)]
MEDIUM: session: register a data->wake callback to process errors
The connection layer will soon call ->wake() only when errors happen, and
not ->init(). So make the session layer use this callback to detect errors
and abort connections.
Willy Tarreau [Wed, 3 Oct 2012 19:12:16 +0000 (21:12 +0200)]
MEDIUM: connection: make it possible for data->wake to return an error
Just like ->init(), ->wake() may now be used to return an error and
abort the connection. Currently this is not used but will be with
embryonic sessions.
Willy Tarreau [Wed, 3 Oct 2012 19:04:48 +0000 (21:04 +0200)]
MEDIUM: connection: only call the data->wake callback on activity
We now check the connection flags for changes in order not to call the
data->wake callback when there is no activity. Activity means a change
on any of the CO_FL_*_SH, CO_FL_ERROR, CO_FL_CONNECTED, CO_FL_WAIT_CONN*
flags, as well as a call to data->recv or data->send.
Willy Tarreau [Wed, 3 Oct 2012 18:00:18 +0000 (20:00 +0200)]
MEDIUM: connection: reorganize connection flags
The connection flags have progressively been added one after the other
and were not very well organized. Some of them are often used together
and a number of operations are performed on the DATA/SOCK ENA/POL flags.
Thus, they have been reorganized so that flags that work together are
close to each other (allows immediate operands on ARM) and that polling
changes can be detected with fewer operations using a simple shift and
xor. The handshakes are now the last ones so that it will be easier to
add new ones after without risking a collision. All activity-related
flags are also grouped together.
Willy Tarreau [Tue, 2 Oct 2012 23:39:48 +0000 (01:39 +0200)]
MEDIUM: connection: use a generic data-layer init() callback
The generic data-layer init callback is now used after the transport
layer is complete and before calling the data layer recv/send callbacks.
This allows the session to switch from the embryonic session data layer
to the complete stream interface data layer, by making conn_session_complete()
the data layer's init callback.
It sill looks awkwards that the init() callback must be used opon error,
but except by adding yet another one, it does not seem to be mergeable
into another function (eg: it should probably not be merged with ->wake
to avoid unneeded calls during the handshake, though semantically that
would make sense).
Willy Tarreau [Tue, 2 Oct 2012 19:21:20 +0000 (21:21 +0200)]
MEDIUM: session: use a specific data_cb for embryonic sessions
We don't want to have the recv or send callbacks in embryonic
sessions, and we want the stream interface to be referenced as
the connection owner only once the session is instanciated. So
let's first have the embryonic session be the owner, then replaced
later by the stream interface once the transport layer is ready.
Willy Tarreau [Tue, 2 Oct 2012 18:07:22 +0000 (20:07 +0200)]
MINOR: connection: provide a generic data layer wakeup callback
Instead of calling conn_notify_si() from the connection handler, we
now call data->wake(), which will allow us to use a different callback
with health checks.
Note that we still rely on a flag in order to decide whether or not
to call this function. The reason is that with embryonic sessions,
the callback is already initialized to si_conn_cb without the flag,
and we can't call the SI notify function in the leave path before
the stream interface is initialized.
This issue should be addressed by involving a different data_cb for
embryonic sessions and for stream interfaces, that would be changed
during session_complete() for the final data_cb.
Willy Tarreau [Tue, 2 Oct 2012 22:41:04 +0000 (00:41 +0200)]
REORG: connection: rename app_cb "data"
Now conn->data will designate the data layer which is the client for
the transport layer. In practice it's the stream interface and will
soon also be the health checks.
Willy Tarreau [Tue, 2 Oct 2012 22:19:48 +0000 (00:19 +0200)]
REORG: connection: rename the data layer the "transport layer"
While working on the changes required to make the health checks use the
new connections, it started to become obvious that some naming was not
logical at all in the connections. Specifically, it is not logical to
call the "data layer" the layer which is in charge for all the handshake
and which does not yet provide a data layer once established until a
session has allocated all the required buffers.
In fact, it's more a transport layer, which makes much more sense. The
transport layer offers a medium on which data can transit, and it offers
the functions to move these data when the upper layer requests this. And
it is the upper layer which iterates over the transport layer's functions
to move data which should be called the data layer.
The use case where it's obvious is with embryonic sessions : an incoming
SSL connection is accepted. Only the connection is allocated, not the
buffers nor stream interface, etc... The connection handles the SSL
handshake by itself. Once this handshake is complete, we can't use the
data functions because the buffers and stream interface are not there
yet. Hence we have to first call a specific function to complete the
session initialization, after which we'll be able to use the data
functions. This clearly proves that SSL here is only a transport layer
and that the stream interface constitutes the data layer.
A similar change will be performed to rename app_cb => data, but the
two could not be in the same commit for obvious reasons.
When a connection setup is pending and we receive an error without a
POLL_IN flag, we're certain there will be nothing to read from it and
we can safely report an error without attempting a recv() call. This
will be significantly better for health checks which will avoid a useless
recv() on all failed checks.
Depending on the pollers used, a connection error may be notified
with POLLOUT|POLLERR|POLLHUP. POLLHUP by itself is enough for the
connection handler to call the read actor, which would only consider
this flag as a good indication of a hangup, without considering the
POLLERR flag.
In order to address this, we directly jump to the read0 label if
POLLERR was not set.
This will be important with health checks as we don't want to believe
a connection was properly established when it's not the case !
Willy Tarreau [Thu, 4 Oct 2012 17:19:36 +0000 (19:19 +0200)]
MINOR: signal: really ignore signals configured with no handler
Until now, signals configured with no handler were still enabled and
ignored upon signal reception. Until now it was not an issue but with
SSL causing many EPIPE all the time, it becomes obvious that signal
processing comes with a cost. So set the handler to SIG_IGN when the
function is NULL.
Willy Tarreau [Thu, 4 Oct 2012 19:54:41 +0000 (21:54 +0200)]
BUG/MINOR: epoll: correctly disable FD polling in fd_rem()
When calling fd_rem(), the polling was not correctly disabled because the
->prev state was set to zero instead of the previous value. fd_rem() is
very rarely used, only just before closing a socket.
The effect is that upon an error reported at the connection level, if the
task assigned to the connection was too slow to be woken up because of too
many other tasks in the run queue, the FD was still not disabled and caused
the connection handler to be called again with the same event until the task
was finally executed to close the fd.
This issue only affects the epoll poller, not the sepoll variant nor any of
the other ones.
It was already present in 1.4 and even 1.3 with the same almost unnoticeable
effects. The bug can in fact only be discovered during development where it
emphasizes other bugs.
Willy Tarreau [Thu, 4 Oct 2012 06:47:34 +0000 (08:47 +0200)]
MEDIUM: proxy: add the global frontend to the list of normal proxies
Since recent changes on the global frontend, it was not possible anymore
to soft-reload a process which had a stats socket because the socket would
not be disabled upon reload. The only solution to this endless madness is
to have the global frontend part of normal proxies.
Since we don't want to get an ID that shifts all other proxies and causes
trouble in deployed environments, we assign it ID #0 which other proxies
can't grab, and we don't report it in the stats pages.
Willy Tarreau [Thu, 4 Oct 2012 06:56:31 +0000 (08:56 +0200)]
BUG/MEDIUM: listener: don't pause protocols that do not support it
Pausing a UNIX_STREAM socket results in a major pain because the socket
does not correctly resume, it wakes poll() but return EAGAIN on accept(),
resulting in a busy loop. So let's only pause protocols that support it.
This issues has existed since UNIX sockets were introduced on bind lines.
Willy Tarreau [Thu, 4 Oct 2012 06:01:43 +0000 (08:01 +0200)]
BUG/MINOR: config: use a copy of the file name in proxy configurations
Each proxy contains a reference to the original config file and line
number where it was declared. The pointer used is just a reference to
the one passed to the function instead of being duplicated. The effect
is that it is not valid anymore at the end of the parsing and that all
proxies will be enumerated as coming from the same file on some late
configuration errors. This may happen for exmaple when reporting SSL
certificate issues.
By copying using strdup(), we avoid this issue.
1.4 has the same issue, though no report of the proxy file name is done
out of the config section. Anyway a backport is recommended to ease
post-mortem analysis.
Willy Tarreau [Wed, 3 Oct 2012 22:14:33 +0000 (00:14 +0200)]
BUG/MEDIUM: proxy: must not try to stop disabled proxies upon reload
Hervé Commowick reported an issue : haproxy dies in a segfault during a
soft restart if it tries to pause a disabled proxy. This is because disabled
proxies have no management task so we must not wake the task up. This could
easily remain unnoticed since the old process was expected to go away, so
having it go away faster was not really troubling. However, with sync peers,
it is obvious that there is no peer sync during this reload.
This issue has been introduced in 1.5-dev7 with the removal of the
maintain_proxies() function. No backport is needed.
BUG/MINOR: ssl: report the L4 connection as established when possible
If we get an SSL error during the handshake, we at least try to see
if a syscall reported an error or not. In case of an error, it generally
means that the connection failed. If there is no error, then the connection
established successfully.
The difference is important for health checks which report the precise cause
to the logs and to the stats.
Disables the stateless session resumption (RFC 5077 TLS Ticket
extension) and force to use stateful session resumption. Stateless
session resumption is more expensive in CPU usage.
Emeric Brun [Tue, 2 Oct 2012 11:45:20 +0000 (13:45 +0200)]
MINOR: ssl: add statement 'no-tls-tickets' on bind to disable stateless session resumption
Disables the stateless session resumption (RFC 5077 TLS Ticket extension)
and force to use stateful session resumption.
Stateless session resumption is more expensive in CPU usage.
Emeric Brun [Mon, 1 Oct 2012 15:10:30 +0000 (17:10 +0200)]
MEDIUM: config: authorize frontend and listen without bind.
This allows to easily add/remove "bind" entries to a frontend without
being forced to remove it when the last entry is temporarily removed.
While "disabled" may sometimes work in a frontend, it becomes trickier
on "listen" sections which can also hold servers and be referenced by
other frontends.
Note that a "listen" section with no "bind" is equivalent to a "backend"
section.
Configs without any listeners are still reported as invalid and refuse
to load.
MINOR: ssl : add statements 'notlsv11' and 'notlsv12' and rename 'notlsv1' to 'notlsv10'.
This is because "notlsv1" used to disable TLSv1.0 only and had no effect
on v1.1/v1.2. so better have an option for each version. This applies both
to "bind" and "server" statements.
MINOR: ssl: disable shared memory and locks on session cache if nbproc == 1
We don't needa to lock the memory when there is a single process. This can
make a difference on small systems where locking is much more expensive than
just a test.
Allow to ignore some verify errors and to let them pass the handshake.
Add option 'crt-ignore-err <list>'
Ignore verify errors at depth == 0 (client certificate)
<list> is string 'all' or a comma separated list of verify error IDs
(see http://www.openssl.org/docs/apps/verify.html)
Add option 'ca-ignore-err <list>'
Same as 'crt-ignore-err' for all depths > 0 (CA chain certs)
Ex ignore all errors on CA and expired or not-yet-valid errors
on client certificate:
MEDIUM: ssl: add client certificate authentication support
Add keyword 'verify' on bind:
'verify none': authentication disabled (default)
'verify optional': accept connection without certificate
and process a verify if the client sent a certificate
'verify required': reject connection without certificate
and process a verify if the client send a certificate
Add keyword 'cafile' on bind:
'cafile <path>' path to a client CA file used to verify.
'crlfile <path>' path to a client CRL file used to verify.
MINOR: connection: add a pointer to the connection owner
This will be needed to find the stream interface from the connection
once they're detached, but in the more immediate term, we'll need this
for health checks since they don't use a stream interface.
MEDIUM: monitor: simplify handling of monitor-net and mode health
We were having several different behaviours with monitor-net and
"mode health" :
- monitor-net on TCP connections was evaluated just after accept(),
did not count a connection on the frontend and were not subject
to tcp-request connection rules, and caused an immediate close().
- monitor-net in HTTP mode was evaluated once the session was
accepted (eg: on top of SSL), returned "HTTP/1.0 200 OK\r\n\r\n"
over the connection's data layer and instanciated a session which
was responsible for closing this connection. A connection AND a
session were counted for the frontend ;
- "mode health" with "option httpchk" would do exactly the same as
monitor-net in HTTP mode ;
- "mode health" without "option httpchk" would do the same as above
except that "OK" was returned instead of "HTTP/1.0 200 OK\r\n\r\n".
None of them took care of cleaning the input buffer, sometimes resulting
in a TCP reset to be emitted after the last packet if a request was received
over the connection.
Given the inconsistencies and the complexity in keeping all these features
handled at the right position, we now slightly changed the way they are
handled :
- all of them are handled just after the "tcp-request connection" rules,
so that all of them may be blocked using such rules, offering more
flexibility and consistency ;
- no connection handshake is performed anymore for non-TCP modes
- all of them send the response as raw data over the socket, there is no
more difference between TCP and HTTP mode for example (these rules were
never meant to be served over SSL connections and were never documented
as able to do that).
- any possible pending data on the incoming socket is drained before the
response is sent, in order to avoid the risk of a reset.
- none of them exactly did what was documented !
This results in more consistent, more flexible and more accurate handling of
monitor rules, with smaller and more robust code.
BUG/MAJOR: http: chunk parser was broken with buffer changes
Since at least commit a458b679, msg->sov could become negative in
http_parse_chunk_size() if a chunk size wrapped around the buffer.
The effect is that at some point channel_forward() was called with
a negative size, causing all data to be transferred without being
analyzed anymore.
Since haproxy does not support keep-alive with the server yet, this
issue is not really noticeable, as the server closes the connection
in response. Still, when tunnel mode is used or when pretent-keepalive
is used, it is possible to see the problem.
This issue was reported and diagnosed by William Lallemand at
Exceliance.
It is sometimes useful to completely disable accepting new connections
on a frontend during maintenance operations. By setting a frontend's
maxconn to zero, connections are not accepted anymore until the limit
is increased again.
BUG: stats: fix regression introduced by commit 4348fad1
Recent commit 4348fad1 (listeners: use dual-linked lists to chain listeners
with frontends) broke frontend lookup in stats sockets by using the wrong
iterator in the listeners.
BUG/MINOR: config: check the proper pointer to report unknown protocol
Check the protocol pointer and not the socket to report an unknown family
in servers or peers. This can never happen anyway, it's just to be completely
clean.
BUG: http: do not print garbage on invalid requests in debug mode
Cyril Bonté reported a mangled debug output when an invalid request
was sent with a faulty request line. The reason was the use of the
msg->sl.rq.l offset which was not yet initialized in this case. So
we change the way to report such an error so that first we initialize
it to zero before parsing a message, then we use that to know whether
we can trust it or not. If it's still zero, then we display the whole
buffer, truncated by debug_hdr() to the first CR or LF character, which
results in the first line only.
The same operation was performed for the response, which was wrong too.
There are now too many bind options to still have them in the middle
of the keyword matrix, so let's move them with the server options in
section 5. No new option was documented yet at this point.
MEDIUM: stats: make use of the standard "bind" parsers to parse global socket
The global stats socket statement now makes use of the standard bind parsers.
This results in all UNIX socket options being set by proto_uxst and in all
TCP and SSL options being inherited and usable. For example it is now possible
to enable a stats socket over SSL/TCP by appending the "ssl" keyword and a
certificate after "crt".
The code is simplified since we don't have a special case to parse this config
keyword anymore.
MINOR: ssl: set the listeners' data layer to ssl during parsing
It's better to set all listeners to ssl_sock when seeing the "ssl"
keyword that to loop on all of them afterwards just for this. This
also removes some #ifdefs.
MEDIUM: stats: remove the stats_sock struct from the global struct
Now the stats socket is allocated when the 'stats socket' line is parsed,
and assigned using the standard str2listener(). This has two effects :
- more than one stats socket can now be declared
- stats socket now support protocols other than UNIX
The next step is to remove the duplicate bind config parsing.
BUG: backend: balance hdr was broken since 1.5-dev11
Alex Markham reported and diagnosed a bug appearing on 1.5-dev11,
causing a crash on x86_64 when header hashing is used. The cause is
a missing (int) cast causing a negative offset to appear positive
and the resulting pointer to go out of bounds.
The crash is not possible anymore since 1.5-dev12 because a second
bug caused the negative sign to disappear so the pointer is always
within range but always wrong, so balance hdr() never works anymore.
This fix restores the correct behaviour and ensures the sign is
correct.
REORG: listener: move unix perms from the listener to the bind_conf
Unix permissions are per-bind configuration line and not per listener,
so let's concretize this in the way the config is stored. This avoids
some unneeded loops to set permissions on all listeners.
The access level is not part of the unix perms so it has been moved
away. Once we can use str2listener() to set all listener addresses,
we'll have a bind keyword parser for this one.
MAJOR: listeners: use dual-linked lists to chain listeners with frontends
Navigating through listeners was very inconvenient and error-prone. Not to
mention that listeners were linked in reverse order and reverted afterwards.
In order to definitely get rid of these issues, we now do the following :
- frontends have a dual-linked list of bind_conf
- frontends have a dual-linked list of listeners
- bind_conf have a dual-linked list of listeners
- listeners have a pointer to their bind_conf
This way we can now navigate from anywhere to anywhere and always find the
proper bind_conf for a given listener, as well as find the list of listeners
for a current bind_conf.
MINOR: config: set the bind_conf entry on listeners created from a "listen" line.
Otherwise we would risk a segfault when checking the config's validity
(eg: when looking for conflicts on ID assignments).
Note that the same issue exists with peers_fe and the global stats_fe. All
listeners should be reviewed and simplified to use a compatible declaration
mode.
MEDIUM: config: enumerate full list of registered "bind" keywords upon error
When an unknown "bind" keyword is detected, dump the list of all
registered keywords. Unsupported default alternatives are also reported
as "not supported".
MEDIUM: config: move all unix-specific bind keywords to proto_uxst.c
The "mode", "uid", "gid", "user" and "group" bind options were moved to
proto_uxst as they are unix-specific.
Note that previous versions had a bug here, only the last listener was
updated with the specified settings. However, it almost never happens
that bind lines contain multiple UNIX socket paths so this is not that
much of a problem anyway.
Registering new SSL bind keywords was not particularly handy as it required
many #ifdef in cfgparse.c. Now the code has moved to ssl_sock.c which calls
a register function for all the keywords.
Error reporting was also improved by this move, because the called functions
build an error message using memprintf(), which can span multiple lines if
needed, and each of these errors will be displayed indented in the context of
the bind line being processed. This is important when dealing with certificate
directories which can report multiple errors.
MEDIUM: config: move the "bind" TCP parameters to proto_tcp
Now proto_tcp.c is responsible for the 4 settings it handles :
- defer-accept
- interface
- mss
- transparent
These ones do not need to be handled in cfgparse anymore. If support for a
setting is disabled by a missing build option, then cfgparse correctly
reports :
[ALERT] 255/232700 (2701) : parsing [echo.cfg:114] : 'bind' : 'transparent' option is not implemented in this version (check build options).
MEDIUM: listener: add a minimal framework to register "bind" keyword options
With the arrival of SSL, the "bind" keyword has received even more options,
all of which are processed in cfgparse in a cumbersome way. So it's time to
let modules register their own bind options. This is done very similarly to
the ACLs with a small difference in that we make the difference between an
unknown option and a known, unimplemented option.
Some settings need to be merged per-bind config line and are not necessarily
SSL-specific. It becomes quite inconvenient to have this ssl_conf SSL-specific,
so let's replace it with something more generic.
MINOR: config: add a function to indent error messages
Bind parsers may return multiple errors, so let's make use of a new function
to re-indent multi-line error messages so that they're all reported in their
context.