Simon Horman [Tue, 12 Feb 2013 01:45:54 +0000 (10:45 +0900)]
MEDIUM: checks: Add agent health check
Support a agent health check performed by opening a TCP socket to a
pre-defined port and reading an ASCII string. The string should have one of
the following forms:
* An ASCII representation of an positive integer percentage.
e.g. "75%"
Values in this format will set the weight proportional to the initial
weight of a server as configured when haproxy starts.
* The string "drain".
This will cause the weight of a server to be set to 0, and thus it will
not accept any new connections other than those that are accepted via
persistence.
* The string "down", optionally followed by a description string.
Mark the server as down and log the description string as the reason.
* The string "stopped", optionally followed by a description string.
This currently has the same behaviour as down (iii).
* The string "fail", optionally followed by a description string.
This currently has the same behaviour as down (iii).
A agent health check may be configured using "option lb-agent-chk".
The use of an alternate check-port, used to obtain agent heath check
information described above as opposed to the port of the service,
may be useful in conjunction with this option.
e.g.
option lb-agent-chk
server http1_1 10.0.0.10:80 check port 10000 weight 100
Simon Horman [Tue, 12 Feb 2013 01:45:53 +0000 (10:45 +0900)]
MEDIUM: server: Tighten up parsing of weight string
Detect:
* Empty weight string, including no digits before '%' in relative
weight string
* Trailing garbage, including between the last integer and '%'
in relative weights
The motivation for this is to allow the weight string to be safely
logged if successfully parsed by this function
Simon Horman [Tue, 12 Feb 2013 01:45:51 +0000 (10:45 +0900)]
MEDIUM: server: Break out set weight processing code
Break out set weight processing code.
This is in preparation for reusing the code.
Also, remove duplicate check in nested if clauses.
{px->lbprm.algo & BE_LB_PROP_DYN) is checked by
the immediate outer if clause, so there is no need
to check it a second time.
Simon Horman [Wed, 13 Feb 2013 08:48:00 +0000 (17:48 +0900)]
BUG/MINOR: Correct logic in cut_crlf()
This corrects what appears to be logic errors in cut_crlf().
I assume that the intention of this function is to truncate a
string at the first cr or lf. However, currently lf are ignored.
Also use '\0' instead of 0 as the null character, a cosmetic change.
Cc: Krzysztof Piotr Oledzki <ole@ans.pl> Signed-off-by: Simon Horman <horms@verge.net.au>
[WT: this fix may be backported to 1.4 too]
Currently, to reload haproxy configuration, you have to use "-sf".
There is a problem with this way of doing things. First of all, in the systemd world,
reload commands should be "oneshot" ones, which means they should not be the new main
process but rather a tool which makes a call to it and then exits. With the current approach,
the reload command is the new main command and moreover, it makes the previous one exit.
Systemd only tracks the main program, seeing it ending, it assumes it either finished or failed,
and kills everything remaining as a grabage collector. We then end up with no haproxy running
at all.
This patch adds wrapper around haproxy, no changes at all have been made into it,
so it's not intrusive and doesn't change anything for other hosts. What this wrapper does
is basically launching haproxy as a child, listen to the SIGUSR2 (not to conflict with
haproxy itself) signal, and spawing a new haproxy with "-sf" as a child to relay the
first one.
MEDIUM: New cli option -Ds for systemd compatibility
This patch adds a new option "-Ds" which is exactly like "-D", but instead of
forking n times to get n jobs running and then exiting, prefers to wait for all the
children it just created. With this done, haproxy becomes more systemd-compliant,
without changing anything for other systems.
Willy Tarreau [Tue, 12 Feb 2013 14:23:12 +0000 (15:23 +0100)]
BUG/MEDIUM: checks: fix a race condition between checks and observe layer7
When observe layer7 is enabled on a server, a response may cause a server
to be marked down while a check is in progress. When the check finally
completes, the connection is not properly released in process_chk() because
the server states makes it think that no check was in progress due to the
lastly reported failure.
When a new check gets scheduled, it reuses the same connection structure
which is reinitialized. When the server finally closes the previous
connection, epoll_wait() notifies conn_fd_handler() which sees that the
old connection is still referenced by fdtab[fd], but it can not do anything
with this fd which does not match conn->t.sock.fd. So epoll_wait() keeps
reporting this fd forever.
The solution is to always make process_chk() always take care of closing
the connection and not make it rely on the connection layer to so.
Special thanks go to James Cole and Finn Arne Gangstad who encountered
the issue almost at the same time and took care of reporting a very
detailed analysis with rich information to help understand the issue.
Willy Tarreau [Tue, 5 Feb 2013 17:52:25 +0000 (18:52 +0100)]
BUG/MEDIUM: log: emit '-' for empty fields again
Commit 2b0108ad accidently got rid of the ability to emit a "-" for
empty log fields. This can happen for captured request and response
cookies, as well as for fetches. Since we don't want to have this done
for headers however, we set the default log method when parsing the
format. It is still possible to force the desired mode using +M/-M.
Thierry Fournier [Thu, 24 Jan 2013 13:15:43 +0000 (14:15 +0100)]
BUG/MEDIUM: ssl: openssl 0.9.8 doesn't open /dev/random before chroot
Openssl needs to access /dev/urandom to initialize its internal random
number generator. It does so when it needs a random for the first time,
which fails if it is a handshake performed after the chroot(), causing
all SSL incoming connections to fail.
This fix consists in calling RAND_bytes() to produce a random before
the chroot, which will in turn open /dev/urandom before it's too late,
and avoid the issue.
If the random generator fails to work while processing the config,
haproxy now fails with an error instead of causing SSL connections to
fail at runtime.
Emmanuel Hocdet [Thu, 24 Jan 2013 16:17:15 +0000 (17:17 +0100)]
MEDIUM: ssl: add bind-option "strict-sni"
This new option ensures that there is no possible fallback to a default
certificate if the client does not provide an SNI which is explicitly
handled by a certificate.
Willy Tarreau [Thu, 24 Jan 2013 14:58:42 +0000 (15:58 +0100)]
CLEANUP: http: don't try to deinitialize http compression if it fails before init
In select_compression_response_header(), some tests are rather confusing
as the "fail" label is used to deinitialize the compression context for
the session while it's branched only before initialization succeeds. The
test is always false here and the dereferencing of the comp_algo pointer
which might be null is also confusing. Remove that code which is not needed
anymore since commit ec3e3890 got rid of the latest issues.
Willy Tarreau [Thu, 24 Jan 2013 14:17:20 +0000 (15:17 +0100)]
BUG/MINOR: unix: remove the 'level' field from the ux struct
Commit 290e63aa moved the unix parameters out of the global stats socket
to the bind_conf struct. As such the stats admin level was also moved
overthere, but it remained in the stats global section where it was not
used, except by a nasty memcpy() used to initialize the ux struct in the
bind_conf with too large data. Fortunately, the extra data copied were
the previous level over the new level so it did not have any impact, but
it could have been worse.
Willy Tarreau [Thu, 24 Jan 2013 01:26:43 +0000 (02:26 +0100)]
BUG/MEDIUM: uri_auth: missing NULL check and memory leak on memory shortage
A test is obviously wrong in uri_auth(). If strdup(pass) returns an error
while strdup(user) passes, the NULL pointer is still stored into the
structure. If the user returns the NULL instead, the allocated memory is
not released before returning the error.
The issue was present in 1.4 so the fix should be backported.
Willy Tarreau [Thu, 24 Jan 2013 01:14:42 +0000 (02:14 +0100)]
BUG/MEDIUM: tools: off-by-one in quote_arg()
This function may write the \0 one char too far in the static array.
There is no effect right now as the function has never been used except
maybe in code that was never released. Out-of-tree code might possibly
be affected though (hence the MEDIUM flag).
Willy Tarreau [Thu, 24 Jan 2013 01:06:05 +0000 (02:06 +0100)]
BUG/MEDIUM: signal: signal handler does not properly check for signal bounds
sig is checked for < 0 or > MAX_SIGNAL, but the signal array is
MAX_SIGNAL in size. At the moment, MAX_SIGNAL is 256. If a system supports
more than MAX_SIGNAL signals, then sending signal MAX_SIGNAL to the process
will corrupt one integer in its memory and might also crash the process.
This bug is also present in 1.4 and 1.3, and the fix must be backported.
Willy Tarreau [Thu, 24 Jan 2013 00:25:25 +0000 (01:25 +0100)]
CLEANUP: http: remove a useless null check
srv cannot be null in http_perform_server_redirect(), as it's taken
from the stream interface's target which is always valid for a
server-based redirect, and it was already dereferenced above, so in
practice, gcc already removes the test anyway.
Willy Tarreau [Thu, 24 Jan 2013 00:18:16 +0000 (01:18 +0100)]
BUG/MINOR: log: improper NULL return check on utoa_pad()
utoa_pad() is directly fed into tmplog, which is checked for NULL.
First, when NULLs are possible, they should be put into a temp variable
in order to preserve tmplog, and second, this return value can never be
NULL because the value passed is tv_usec/1000 (between "0" and "999")
with a 4-char output. However better fix the check in case this code gets
improperly copy-pasted for another usage later.
Willy Tarreau [Wed, 23 Jan 2013 23:48:39 +0000 (00:48 +0100)]
BUG/MINOR: cli: show sess should always validate s->listener
Currently s->listener is set for all sessions, but this may not remain
the case forever so we already check s->listener for validity. On check
was missed.
Willy Tarreau [Wed, 23 Jan 2013 23:37:39 +0000 (00:37 +0100)]
BUG/MEDIUM: checks: ensure the health_status is always within bounds
health_adjust() checks for incorrect bounds for the status argument.
With current code, the argument is always a constant from the valid
enum so there is no impact and the check is basically a NOP. However
users running local patches (eg: new checks) might want to recheck
their code.
This fix should be backported to 1.4 which introduced the issue.
Willy Tarreau [Wed, 23 Jan 2013 23:31:24 +0000 (00:31 +0100)]
BUG/MINOR: config: check the proper variable when parsing log minlvl
logsrv->minlvl gets the numeric log level from the equivalent string.
Upon error, ->level was checked due to a wrong copy-paste. The effect
is that a wrong name will silently be ignored and due to minlvl=-1,
will act as if the option was not set.
Willy Tarreau [Wed, 23 Jan 2013 23:29:37 +0000 (00:29 +0100)]
BUG/MINOR: config: free peer's address when exiting upon parsing error
An error caused by an invalid port does not cause the raddr string to
be freed. This is harmless at the moment since we exit, but may have
an impact later if we ever support hot config changes.
Willy Tarreau [Fri, 18 Jan 2013 14:22:41 +0000 (15:22 +0100)]
BUG/MINOR: epoll: use a fix maxevents argument in epoll_wait()
epoll_wait() takes a number of returned events, not the number of
fds to consider. We must not pass it the number of the smallest fd,
as it leads to value zero being used, which is invalid in epoll_wait().
The effect may sometimes be observed with peers sections trying to
connect and causing 2-seconds CPU loops upon a soft reload because
epoll_wait() immediately returns -1 EINVAL instead of waiting for the
timeout to happen.
This fix should be backported to 1.4 too (into ev_epoll and ev_sepoll).
Willy Tarreau [Fri, 18 Jan 2013 10:12:27 +0000 (11:12 +0100)]
BUG/MEDIUM: config: verbosely reject peers sections with multiple local peers
If a peers section contains several instances of the local peer name, only
the first one was considered and the next ones were silently ignored. This
can cause some trouble to debug such a configuration. Now the extra entries
are rejected with an error message indicating where the first occurrence was
found.
Michael Scherer [Sat, 12 Jan 2013 17:35:19 +0000 (18:35 +0100)]
BUG/MEDIUM: remove supplementary groups when changing gid
Without it, haproxy will retain the group membership of root, which may
give more access than intended to the process. For example, haproxy would
still be in the wheel group on Fedora 18, as seen with :
[WT: The issue has been investigated by independent security research team
and realized by itself not being able to allow security exploitation.
Additionally, dropping groups is not allowed to unprivileged users,
though this mode of deployment is quite common. Thus a warning is
emitted in this case to inform the user. The fix could be backported
into all supported versions as the issue has always been there. ]
Willy Tarreau [Thu, 17 Jan 2013 20:34:52 +0000 (21:34 +0100)]
BUG/MEDIUM: peers: only the last peers section was used by tables
Due to a typo in the peers section lookup code, the last declared peers
section was used instead of the one matching the requested name. This bug
has been there since the very first commit on peers section (1.5-dev2).
Willy Tarreau [Thu, 10 Jan 2013 15:22:27 +0000 (16:22 +0100)]
BUG/MINOR: log: temporary fix for lost SSL info in some situations
When using log-format to log the result of sample fetch functions
which rely on the transport layer (eg: ssl*), we have no way to tell
the proxy not to release the connection before logs have caught the
necessary information. As a result, it happens that logging SSL fetch
functions sometimes doesn't return anything for example if the server
is not available and the connection is immediately aborted.
This issue will be fixed with the upcoming patches to finish handling
of sample fetches.
So for the moment, always mark the LW_XPRT flag on the proxy so that
when any fetch method is used, the proxy does not release the transport
layer too fast.
Willy Tarreau [Mon, 7 Jan 2013 15:57:09 +0000 (16:57 +0100)]
OPTIM: splice: assume by default that splice is working correctly
Versions of splice between 2.6.25 and 2.6.27.12 were bogus and would return EAGAIN
on incoming shutdowns. On these versions, we have to call recv() after such a return
in order to find whether splice is OK or not. Since 2.6.27.13 we don't need to do
this anymore, saving one useless recv() call after each splice() returning EAGAIN,
and we can avoid this logic by defining ASSUME_SPLICE_WORKS.
Building with linux2628 automatically enables splice and the flag above since the
kernel is safe. People enabling splice for custom kernels will be able to disable
this logic by hand too.
Willy Tarreau [Mon, 7 Jan 2013 15:38:26 +0000 (16:38 +0100)]
OPTIM: splice: detect shutdowns and avoid splice() == 0
Since last commit introducing EPOLLRDHUP, the splicing code is able to
detect an incoming shutdown without calling splice() == 0. This avoids
one useless syscall.
Willy Tarreau [Mon, 7 Jan 2013 15:19:18 +0000 (16:19 +0100)]
OPTIM: epoll: make use of EPOLLRDHUP
epoll may report pending shutdowns using EPOLLRDHUP. Since this
flag is missing from a number of libcs despite being available
since kernel 2.6.17, let's define it ourselves.
Doing so saves one syscall by allow us to avoid the read()==0 when
the server closes with the respose.
Willy Tarreau [Fri, 4 Jan 2013 15:20:20 +0000 (16:20 +0100)]
MINOR: signal: don't block SIGPROF by default
SIGPROF is blocked then restored to default settings, which sometimes
makes profiling harder. Let's not block it by default nor restore it,
it doesn't serve any purpose anyway.
Willy Tarreau [Sat, 5 Jan 2013 15:20:35 +0000 (16:20 +0100)]
BUG/MINOR: http-compression: lookup Cache-Control in the response, not the request
As stated in both RFC2616 and the http-bis drafts, Cache-Control:
no-transform must be looked up in the response since we're modifying
the response. However, its presence in the request is irrelevant to
any changes in the response :
7.2.1.6. no-transform
The "no-transform" request directive indicates that an intermediary
(whether or not it implements a cache) MUST NOT change the Content-
Encoding, Content-Range or Content-Type request header fields, nor
the request representation.
7.2.2.9. no-transform
The "no-transform" response directive indicates that an intermediary
(regardless of whether it implements a cache) MUST NOT change the
Content-Encoding, Content-Range or Content-Type response header
fields, nor the response representation.
Note: according to the specs, we're supposed to emit the following
response header :
Warning: 214 transformation applied
However no other product seems to do it, so the effect on user agents
is unclear.
Willy Tarreau [Fri, 4 Jan 2013 13:14:57 +0000 (14:14 +0100)]
DOC: fix bogus recommendation on usage of gpc0 counter
The doc pretends that src_inc_gpc0 may be used alone without an integer
match, but this is false and has always been since its introduction in
1.5-dev1. If the ACL is called, the increment will be used, the value
returned, but it will be matched against no value so the resulting ACL
will never be true and the condition will not be met.
Note that clr_gpc0 is trickier, as it returns the previous value which
might also be zero. Thus it's suggested to compare it against any positive
value including zero :
tcp-request connection accept if { src_clr_gpc0 ge 0 }
Some arguments were missing on the sc1/sc2 forms of most ACLs including
gpc0, so this has been fixed too.
Willy Tarreau [Sun, 30 Dec 2012 00:44:24 +0000 (01:44 +0100)]
BUG/MEDIUM: checks: ignore late resets after valid responses
Reinout Verkerk from Trilex reported an issue with servers recently
flapping after an haproxy upgrade. Haproxy checks a simple agent
returning an HTTP response. The issue is that if the request packet
is lost but the simple agent responds before reading the HTTP request
and closes, the server will emit a TCP RST once the request finally
reaches it.
The way checks have been ported to use connections makes the error
flag show up as a failure after the success, reporting a stupid case
where the server is said to be down with a correct response.
In order to fix this, let's ignore the connection's error flag if a
successful check has already been reported. Reinout could verify that
a patched server did not exhibit the problem anymore.
Willy Tarreau [Sun, 30 Dec 2012 00:39:37 +0000 (01:39 +0100)]
BUG/MEDIUM: stream_interface: don't close outgoing connections on shutw()
Commit 7bb68abb introduced the SI_FL_NOHALF flag in dev10. It is used
to automatically close the write side of a connection whose read side
is closed. But the patch also caused the opposite to happen, which is
that a simple shutw() call would immediately close the connection. This
is not desired because when using option abortonclose, we want to pass
the client's shutdown to the server which will decide what to do with
it. So let's avoid the close when SHUTR is not set.
Willy Tarreau [Sat, 29 Dec 2012 23:50:35 +0000 (00:50 +0100)]
BUG/MINOR: http: don't process abortonclose when request was sent
option abortonclose may cause a valid connection to be aborted just
after the request has been sent. This is because we check for it
during the session establishment sequence before checking for write
activity. So if the abort and the connect complete at the same time,
the abort is still considered. Let's check for an explicity partial
write before aborting.
Willy Tarreau [Sat, 29 Dec 2012 20:50:07 +0000 (21:50 +0100)]
BUG/MINOR: time: frequency counters are not totally accurate
When a frontend is rate-limited to 1000 connections per second, the
effective rate measured from the client is 999/s, and connections
experience an average response time of 99.5 ms with a standard
deviation of 2 ms.
The reason for this inaccuracy is that when computing frequency
counters, we use one part of the previous value proportional to the
number of milliseconds remaining in the current second. But even the
last millisecond still uses a part of the past value, which is wrong :
since we have a 1ms resolution, the last millisecond must be dedicated
only to filling the current second.
So we slightly adjust the algorithm to use 999/1000 of the past value
during the first millisecond, and 0/1000 of the past value during the
last millisecond. We also slightly improve the computation by computing
the remaining time instead of the current time in tv_update_date(), so
that we don't have to negate the value in each frequency counter.
Now with the fix, the connection rate measured by both the client and
haproxy is a steady 1000/s, the average response time measured is 99.2ms
and more importantly, the standard deviation has been divided by 3 to
0.6 millisecond.
This fix should also be backported to 1.4 which has the same issue.
Willy Tarreau [Fri, 28 Dec 2012 14:04:05 +0000 (15:04 +0100)]
[RELEASE] Released version 1.5-dev17
Released version 1.5-dev17 with the following main changes :
- MINOR: ssl: Setting global tune.ssl.cachesize value to 0 disables SSL session cache.
- BUG/MEDIUM: stats: fix stats page regression introduced by commit 20b0de5
- BUG/MINOR: stats: last fix was still wrong
- BUG/MINOR: stats: http-request rules still don't cope with stats
- BUG/MINOR: http: http-request add-header emits a corrupted header
- BUG/MEDIUM: stats: disable request analyser when processing POST or HEAD
- BUG/MINOR: log: make log-format, unique-id-format and add-header more independant
- BUILD: log: unused variable svid
- CLEANUP: http: rename the misleading http_check_access_rule
- MINOR: http: move redirect rule processing to its own function
- REORG: config: move the http redirect rule parser to proto_http.c
- MEDIUM: http: add support for "http-request redirect" rules
- MEDIUM: http: add support for "http-request tarpit" rule
Willy Tarreau [Thu, 27 Dec 2012 11:37:57 +0000 (12:37 +0100)]
MEDIUM: http: add support for "http-request tarpit" rule
The "reqtarpit" rule is not very handy to use. Now that we have more
flexibility with "http-request", let's finally make the tarpit rules
usable there.
There are still semantical differences between apply_filters_to_request()
and http_req_get_intercept_rule() because the former updates the counters
while the latter does not. So we currently have almost similar code leafs
for similar conditions, but this should be cleaned up later.
Willy Tarreau [Thu, 27 Dec 2012 11:19:02 +0000 (12:19 +0100)]
MEDIUM: http: add support for "http-request redirect" rules
These are exactly the same as the classic redirect rules except
that they can be interleaved with other http-request rules for
more flexibility.
The redirect parser should probably be changed to stop at the condition
so that the caller puts its own condition pointer. At the moment, the
redirect rule and condition are parsed at once by build_redirect_rule()
and the condition is assigned to the http_req_rule.
Willy Tarreau [Thu, 27 Dec 2012 10:30:54 +0000 (11:30 +0100)]
MINOR: http: move redirect rule processing to its own function
We now have http_apply_redirect_rule() which does all the redirect-specific
job instead of having this inside http_process_req_common().
Also one of the benefit gained from uniformizing this code is that both
keep-alive and close response do emit the PR-- flags. The fix for the
flags could probably be backported to 1.4 though it's very minor.
The previous function http_perform_redirect() was becoming confusing
so it was renamed http_perform_server_redirect() since it only applies
to server-based redirection.
Willy Tarreau [Thu, 27 Dec 2012 09:46:37 +0000 (10:46 +0100)]
CLEANUP: http: rename the misleading http_check_access_rule
Several bugs were introduced recently due to a misunderstanding of how
this function works and what it was supposed to do. Since it's supposed
to only return the pointer to a rule which aborts further processing of
the request, let's rename it to avoid further issues.
The function was also slightly cleaned up without any functional change.
Willy Tarreau [Fri, 28 Dec 2012 08:40:16 +0000 (09:40 +0100)]
BUG/MINOR: log: make log-format, unique-id-format and add-header more independant
It happens that all of them call parse_logformat_line() which sets
proxy->to_log with a number of flags affecting the line format for
all three users. For example, having a unique-id specified disables
the default log-format since fe->to_log is tested when the session
is established.
Similarly, having "option logasap" will cause "+" to be inserted in
unique-id or headers referencing some of the fields depending on
LW_BYTES.
This patch first removes most of the dependency on fe->to_log whenever
possible. The first possible cleanup is to stop checking fe->to_log
for being null, considering that it always contains at least LW_INIT
when any such usage is made of the log-format!
Also, some checks are wrong. s->logs.logwait cannot be nulled by
"logwait &= ~LW_*" since LW_INIT is always there. This results in
getting the wrong log at the end of a request or session when a
unique-id or add-header is set, because logwait is still not null
but the log-format is not checked.
Further cleanups are required. Most LW_* flags should be removed or at
least replaced with what they really mean (eg: depend on client-side
connection, depend on server-side connection, etc...) and this should
only affect logging, not other mechanisms.
This patch fixes the default log-format and tries to limit interferences
between the log formats, but does not pretend to do more for the moment,
since it's the most visible breakage.
Willy Tarreau [Fri, 28 Dec 2012 07:36:50 +0000 (08:36 +0100)]
BUG/MEDIUM: stats: disable request analyser when processing POST or HEAD
After the response headers are sent and the request processing is done,
the buffers are wiped out and the stream interface is closed. We must
then disable the request analysers, otherwise some processing will
happen on a closed stream interface and empty buffers which do not
match, causing all sort of crashes. This issue was introduced with
recent work on the stats, and was reported by Seri.
Willy Tarreau [Fri, 28 Dec 2012 01:44:01 +0000 (02:44 +0100)]
BUG/MINOR: http: http-request add-header emits a corrupted header
David BERARD reported that http-request add-header passes a \0 along
with the header field, which of course is not appropriate. This is
caused by build_logline() which sometimes returns the size with the
trailing zero and sometimes can return an empty string. Let's fix
this function instead of fixing the places where it's used.
Willy Tarreau [Tue, 25 Dec 2012 20:53:25 +0000 (21:53 +0100)]
BUG/MINOR: stats: last fix was still wrong
Previous commit was still wrong, it broke add-header and set-header
because we don't want to leave on these actions.
The http_check_access_rule() function should be redesigned, it was
initially thought for allow/deny rules but now it is executing other
non-final rules and at the same time returning a pointer to the last
final rule. That becomes a bit confusing and will need to be addressed
before we implement redirect and return.
Willy Tarreau [Tue, 25 Dec 2012 19:52:58 +0000 (20:52 +0100)]
BUG/MEDIUM: stats: fix stats page regression introduced by commit 20b0de5
This commit adding http-request add-header/set-header unfortunately introduced
a regression to the handling of the stats page which is not matched anymore.
Willy Tarreau [Mon, 24 Dec 2012 15:48:14 +0000 (16:48 +0100)]
[RELEASE] Released version 1.5-dev16
Released version 1.5-dev16 with the following main changes :
- BUG/MEDIUM: ssl: Prevent ssl error from affecting other connections.
- BUG/MINOR: ssl: error is not reported if it occurs simultaneously with peer close detection.
- MINOR: ssl: add fetch and acl "ssl_c_used" to check if current SSL session uses a client certificate.
- MINOR: contrib: make the iprange tool grep for addresses
- CLEANUP: polling: gcc doesn't always optimize constants away
- OPTIM: poll: optimize fd management functions for low register count CPUs
- CLEANUP: poll: remove a useless double-check on fdtab[fd].owner
- OPTIM: epoll: use a temp variable for intermediary flag computations
- OPTIM: epoll: current fd does not count as a new one
- BUG/MINOR: poll: the I/O handler was called twice for polled I/Os
- MINOR: http: make resp_ver and status ACLs check for the presence of a response
- BUG/MEDIUM: stream-interface: fix possible stalls during transfers
- BUG/MINOR: stream_interface: don't return when the fd is already set
- BUG/MEDIUM: connection: always update connection flags prior to computing polling
- CLEANUP: buffer: use buffer_empty() instead of buffer_len()==0
- BUG/MAJOR: stream_interface: fix occasional data transfer freezes
- BUG/MEDIUM: stream_interface: fix another case where the reader might not be woken up
- BUG/MINOR: http: don't abort client connection on premature responses
- BUILD: no need to clean up when making git-tar
- MINOR: log: add a tag for amount of bytes uploaded from client to server
- BUG/MEDIUM: log: fix possible segfault during config parsing
- MEDIUM: log: change a few log tokens to make them easier to remember
- BUG/MINOR: log: add_to_logformat_list() used the wrong constants
- MEDIUM: log-format: make the format parser more robust and more extensible
- MINOR: sample: support cast from bool to string
- MINOR: samples: add a function to fetch and convert any sample to a string
- MINOR: log: add lf_text_len
- MEDIUM: log: add the ability to include samples in logs
- REORG: stats: massive code reorg and cleanup
- REORG: stats: move the HTTP header injection to proto_http
- REORG: stats: functions are now HTTP/CLI agnostic
- BUG/MINOR: log: fix regression introduced by commit 8a3f52
- MINOR: chunks: centralize the trash chunk allocation
- MEDIUM: stats: use hover boxes instead of title to report details
- MEDIUM: stats: use multi-line tips to display detailed counters
- MINOR: tools: simplify the use of the int to ascii macros
- MINOR: stats: replace STAT_FMT_CSV with STAT_FMT_HTML
- MINOR: http: prepare to support more http-request actions
- MINOR: log: make parse_logformat_string() take a const char *
- MEDIUM: http: add http-request 'add-header' and 'set-header' to build headers
Willy Tarreau [Mon, 24 Dec 2012 14:45:22 +0000 (15:45 +0100)]
MEDIUM: http: add http-request 'add-header' and 'set-header' to build headers
These two new statements allow to pass information extracted from the request
to the server. It's particularly useful for passing SSL information to the
server, but may be used for various other purposes such as combining headers
together to emulate internal variables.
Willy Tarreau [Sun, 23 Dec 2012 17:00:29 +0000 (18:00 +0100)]
MINOR: tools: simplify the use of the int to ascii macros
These macros (U2H, U2A, LIM2A, ...) have been used with an explicit
index for the local storage variable, making it difficult to change
log formats and causing a few issues from time to time. Let's have
a single macro with a rotating index so that up to 10 conversions
may be used in a single call.
Willy Tarreau [Sun, 23 Dec 2012 01:25:03 +0000 (02:25 +0100)]
MEDIUM: stats: use multi-line tips to display detailed counters
Frontend, listener, server and backend detailed counters are now spread
over several lines in a table when the pointer hovers over the <u> area.
The values are much more readble, and the extra space gained this way
allowed to report some percentages.
Note: some incoherencies still exist between some counters. For example,
the backend's cum_conn is increased when a session is assigned instead
of increasing cum_sess.
Willy Tarreau [Sun, 23 Dec 2012 00:59:23 +0000 (01:59 +0100)]
MEDIUM: stats: use hover boxes instead of title to report details
Using titles to report detailed information is not convenient. The
browser decides to wrap the line where it wants, generally the box
quickly fades away, and it's not possible to copy-paste the text
from the box.
By using two levels, we can make a block appear/disappear depending
on whether its parent it being hovered or not, for example :
Toggling visibility ensures that the place required to display the block
is always reserved. This is important to display boxes that are close to
the edges.
Then using <span class="tip">this is a box</span> will make the text
appear only when the upper <u> is hovered.
But this still adds much text. So instead we use a generic <div> tag
which we don't use anywhere else. That way we don't have to specify a
class :
div { display: block; visibility: hidden; }
u:hover div { visibility: hidden; }
This works pretty well, even in old browsers from 2005.
This commit does not change the display format, it only replaces the
title attribute with the div tag. Later commits will adjust the layout.
Willy Tarreau [Sun, 23 Dec 2012 19:22:19 +0000 (20:22 +0100)]
MINOR: chunks: centralize the trash chunk allocation
At the moment, we need trash chunks almost everywhere and the only
correctly implemented one is in the sample code. Let's move this to
the chunks so that all other places can use this allocator.
Additionally, the get_trash_chunk() function now really returns two
different chunks. Previously it used to always overwrite the same
chunk and point it to a different buffer, which was a bit tricky
because it's not obvious that two consecutive results do alias each
other.
Willy Tarreau [Sun, 23 Dec 2012 16:29:07 +0000 (17:29 +0100)]
BUG/MINOR: log: fix regression introduced by commit 8a3f52
The commit above improved error reporting during log parsing, but as
a result, some shared strings such as httplog_format are truncated
during parsing. This is observable upon startup because the second
proxy to use httplog emits a warning.
Let's have the logformat parser duplicate the string while parsing it.
Willy Tarreau [Sat, 22 Dec 2012 22:20:30 +0000 (23:20 +0100)]
REORG: stats: functions are now HTTP/CLI agnostic
All the functions that were called "http" or "raw" have been cleaned up
to support either only HTML and use that word in their name, or support
both HTML and CSV and be usable both from the HTTP and the CLI entry
points.
stats_http_dump() now explicitly checks for HTML mode before adding
HTML-specific headers/trailers, and has been renamed since it's now
used by both entry points.
Some more duplicated code could be removed. The patch looks big but it's
mostly due to re-indents resulting from the moves or comments updates.
http_stats_io_handler()
-> stats_dump_stat_to_buffer() // same as above, but used for CSV or HTML
-> stats_dump_csv_header() // emits the CSV headers (same as above)
-> stats_dump_html_head() // emits the HTML headers
-> stats_dump_html_info() // emits the equivalent of "show info" at the top
-> stats_dump_proxy_to_buffer() // same as above, valid for CSV and HTML
-> stats_dump_html_px_hdr()
-> stats_dump_fe_stats()
-> stats_dump_li_stats()
-> stats_dump_sv_stats()
-> stats_dump_be_stats()
-> stats_dump_html_px_end()
-> stats_dump_html_end() // emits HTML trailer
Willy Tarreau [Sat, 22 Dec 2012 21:03:39 +0000 (22:03 +0100)]
REORG: stats: move the HTTP header injection to proto_http
The HTTP header injection that are performed in dumpstats when responding
or when redirecting a POST request have nothing to do in dumpstats. They
do not use any state from the stats, and are 100% HTTP. Let's make the
headers there in the HTTP core, and have dumpstats only produce stats.
Willy Tarreau [Sat, 22 Dec 2012 19:31:10 +0000 (20:31 +0100)]
REORG: stats: massive code reorg and cleanup
The dumpstats code looks like a spaghetti plate. Several functions are
supposed to be able to do several things but rely on complex states to
dispatch the work to independant functions. Most of the HTML output is
performed within the switch/case statements of the whole state machine.
Let's clean this up by adding new functions to emit the data and have
a few more iterators to avoid relying on so complex states.
The new stats dump sequence looks like this for CLI and for HTTP :
Willy Tarreau [Thu, 20 Dec 2012 23:09:23 +0000 (00:09 +0100)]
MEDIUM: log: add the ability to include samples in logs
Using %[expression] it becomes possible to make the log engine fetch
some samples from the request or the response and provide them in the
logs. Note that this feature is still limited, it does not yet allow
to apply converters, to limit the output length, nor to specify the
direction which should be fetched when a fetch function works in both
directions.
However it's quite convenient to log SSL information or to include some
information that are used in stick tables.
It is worth noting that this has been done in the generic log format
handler, which means that the same information may be used to build the
unique-id header and to pass the information to a backend server.
Willy Tarreau [Thu, 20 Dec 2012 20:23:42 +0000 (21:23 +0100)]
MEDIUM: log-format: make the format parser more robust and more extensible
The log-format parser reached a limit making it hard to add new features.
It also suffers from a weak handling of certain incorrect corner cases,
for example "%{foo}" is emitted as a litteral while syntactically it's an
argument to no variable. Also the argument parser had to redo some of the
job with some cases causing minor memory leaks (eg: ignored args).
This work aims at improving the situation so that slightly better reporting
is possible and that it becomes possible to extend the log format. The code
has a few more states but looks significantly simpler. The parser is now
capable of reporting ignored arguments and truncated lines.
Willy Tarreau [Thu, 20 Dec 2012 20:59:12 +0000 (21:59 +0100)]
BUG/MINOR: log: add_to_logformat_list() used the wrong constants
The <type> argument was checked against LOG_FMT_* but it was passed as
LF_* which are two independant enums. It happens that the 3 first entries
in these enums do match, but this broke some experimental changes which
required another state, so let's fix this now.
Willy Tarreau [Thu, 20 Dec 2012 16:22:52 +0000 (17:22 +0100)]
MEDIUM: log: change a few log tokens to make them easier to remember
Some log tokens have evolved in a way that is not completely logical.
For example, frontend tokens sometimes begin with an 'f' and sometimes
with an 'F'. Same for backend and server.
So let's change a few cases without disrupting compatibility with existing
setups :
Bi => bi
Bp => bp
Ci => ci
Cp => cp
Fi => fi
Fp => fp
Si => si
Sp => sp
cc => CC
cs => CS
st => ST
The old ones are still supported but deprecated and will be unsupported by
the 1.5 release. However, a warning message is emitted when they're encounterd
and it indicates what token should be used to replace them.
Willy Tarreau [Thu, 20 Dec 2012 17:19:26 +0000 (18:19 +0100)]
BUG/MEDIUM: log: fix possible segfault during config parsing
When log format arguments are specified within braces with no '+' nor '-'
prefix, the NULL string is compared with known keywords causing a crash.
This only happens during parsing so it does not affect runtime processing.
Willy Tarreau [Thu, 20 Dec 2012 11:10:09 +0000 (12:10 +0100)]
BUG/MINOR: http: don't abort client connection on premature responses
When a server responds prematurely to a POST request, haproxy used to
cause the transfer to be aborted before the end. This is problematic
because this causes the client to receive a TCP reset when it tries to
push more data, generally preventing it from receiving the response
which contain the reason for the premature reponse (eg: "entity too
large" or an authentication request).
From now on we take care of allowing the upload traffic to flow to the
server even when the response has been received, since the server is
supposed to drain it. That way the client receives the server response.
This bug has been present since 1.4 and the fix should probably be
backported there.
Willy Tarreau [Wed, 19 Dec 2012 17:01:02 +0000 (18:01 +0100)]
BUG/MEDIUM: stream_interface: fix another case where the reader might not be woken up
The code review during the chase for the POST freeze uncovered another possible
issue which might appear when we perform an incomplete read and want to stop because
of READ_DONTWAIT or because we reached the maximum read_poll limit. Reading is
disabled but SI_FL_WAIT_ROOM was not set, possibly causing some cases where a
send() on the other side would not wake the reader up until another activity
on the same side calls the update function which fixes its status.
Willy Tarreau [Wed, 19 Dec 2012 16:34:17 +0000 (17:34 +0100)]
BUG/MAJOR: stream_interface: fix occasional data transfer freezes
Since the changes in connection management, it became necessary to re-enable
polling after a fast-forward transfer would complete.
One such issue was addressed after dev12 by commit 9f7c6a18 (BUG/MAJOR:
stream_interface: certain workloads could cause get stuck) but unfortunately,
it was incomplete as very subtle cases would occasionally remain unaddressed
when a buffer was marked with the NOEXP flag, which is used during POST
uploads. The wake up must be performed even when the flag is there, the
flag is used only to refresh the timeout.
Too many conditions need to be hit together for the situation to be
reproducible, but it systematically appears for some users.
It is particularly important to credit Sander Klein and John Rood from
Picturae ICT ( http://picturae.com/ ) for reporting this bug on the mailing
list, providing configs and countless traces showing the bug in action, and
for their patience testing litteraly tens of snapshots and versions of
supposed fixes during a full week to narrow the commit range until the bug
was really knocked down! As a side effect of their numerous tests, several
other bugs were fixed.
Willy Tarreau [Sun, 16 Dec 2012 18:19:13 +0000 (19:19 +0100)]
BUG/MEDIUM: connection: always update connection flags prior to computing polling
stream_int_chk_rcv_conn() did not clear connection flags before updating them. It
is unsure whether this could have caused the stalled transfers that have been
reported since dev15.
In order to avoid such further issues, we now use a simple inline function to do
all the job.
Willy Tarreau [Sat, 15 Dec 2012 09:12:39 +0000 (10:12 +0100)]
BUG/MINOR: stream_interface: don't return when the fd is already set
Back in the days where polling was made with select() where all FDs
were checked at once, stream_int_chk_snd_conn() used to check whether
the file descriptor it was passed was ready or not, so that it did
not perform the work for nothing.
Right now FDs are checked just before calling the I/O handler so this
test never matches at best, or may return false information at worst.
Since conn_fd_handler() always clears the flags upon exit, it looks
like a missed event cannot happen right now. Still, better remove
this outdated check than wait for it to cause issues.
Willy Tarreau [Sat, 15 Dec 2012 08:18:05 +0000 (09:18 +0100)]
BUG/MEDIUM: stream-interface: fix possible stalls during transfers
Sander Klein reported a rare case of POST transfers being stalled
after a few megabytes since dev15. One possible culprit is the fix
for the CPU spinning issues which is not totally correct, because
stream_int_chk_snd_conn() would inconditionally enable the
CO_FL_CURR_WR_ENA flag.
What could theorically happen is the following sequence :
1) send buffer is empty, server-side polling is disabled
2) client sends some data
3) such data are forwarded to the server using
stream_int_chk_snd_conn()
4) conn->flags |= CO_FL_CURR_WR_ENA
5) si_conn_send_loop() is called
6) raw_sock_from_buf() does a partial write due to full kernel buffers
7) stream_int_chk_snd_conn() detects this and requests to be called
to send the remaining data using __conn_data_want_send(), and clears
the SI_FL_WAIT_DATA flag on the stream interface, indicating that it
is already congestionned.
8) conn_cond_update_polling() calls conn_data_update_polling() which
sees that both CO_FL_DATA_WR_ENA and CO_FL_CURR_WR_ENA are set, so
it does not enable polling on the output fd.
9) the next chunk from the client fills the buffer
10) stream_int_chk_snd_conn() is called again
11) SI_FL_WAIT_DATA is already cleared, so the function immediately
returns without doing anything.
12) the buffer is now full with the FD write polling disabled and
everything deadlocks.
Not that there is no reason for such an issue not to happen the other
way around, from server to client, except maybe that due to the speed
difference between the client and the server, client-side polling is
always enabled and the buffer is never empty.
All this shows that the new polling still looks fragile, in part due
to the double information on the FD status, being both in fdtab[] and
in the connection, which looks unavoidable. We should probably have
some functions to tighten the relation between such flags and avoid
manipulating them by hand.
Also, the effects of chk_snd() on the polling are still under-estimated,
while the relation between the stream_int and the FD is still too much
present. Maybe the function should be rethought to only call the connection's
fd handler. The connection model probably needs two calling conventions
for bottom half and upper half.
Emeric Brun [Fri, 14 Dec 2012 10:21:13 +0000 (11:21 +0100)]
BUG/MEDIUM: ssl: Prevent ssl error from affecting other connections.
J. Maurice reported that ssllabs.com test affects unrelated
legitimate traffic and cause SSL errors and broken connections.
Sometimes openssl store read/write/handshake errors in a global stack. This
stack is not specific to the current session. Openssl API does not clean the
stack at the beginning of a new read/write. And the function used to retrieve
error ID after read/write, returns the generic error SSL_ERROR_SSL if the
global stack is not empty.
The fix consists in cleaning the errors stack after read/write/handshake
errors.
Willy Tarreau [Fri, 14 Dec 2012 07:33:14 +0000 (08:33 +0100)]
MINOR: http: make resp_ver and status ACLs check for the presence of a response
The two ACL fetches "resp_ver" and "status", if used in a request despite
the warning, would return a match of zero length. This is inappropriate,
better return a non-match to be more consistent with other ACL processing.
Willy Tarreau [Thu, 13 Dec 2012 23:17:03 +0000 (00:17 +0100)]
BUG/MINOR: poll: the I/O handler was called twice for polled I/Os
When a polled I/O event is detected, the event is added to the updates
list and the I/O handler is called. Upon return, if the event handler
did not experience an EAGAIN, the event remains in the updates list so
that it will be processed later. But if the event was already in the
spec list, its state is updated and it will be called again immediately
upon exit, by fd_process_spec_events(), so this creates unfairness
between speculative events and polled events.
So don't call the I/O handler upon I/O detection when the FD already is
in the spec list. The fd events are still updated so that the spec list
is up to date with the possible I/O change.
Willy Tarreau [Thu, 13 Dec 2012 23:02:33 +0000 (00:02 +0100)]
OPTIM: epoll: current fd does not count as a new one
The epoll loop checks for newly appeared FDs in order to process them early
if they're accepted sockets. Since the introduction of the fd_ev_set()
calls before the iocb(), the current FD is always in the update list,
and we don't want to check it again, so we must assign the old_updt
index just before calling the I/O handler.
Willy Tarreau [Thu, 13 Dec 2012 22:52:58 +0000 (23:52 +0100)]
OPTIM: epoll: use a temp variable for intermediary flag computations
Playing with fdtab[fd].ev makes gcc constantly reload the pointers
because it does not know they don't alias. Use a temporary variable
instead. This saves a few operations in the fast path.
Looking at the assembly code that updt_fd() and alloc/release_spec_entry
produce in the polling loops, it's clear that gcc has to recompute pointers
several times in a row because of limited spare registers. By better
grouping adjacent structure updates, we improve the code size by around
60 bytes in the fast path on x86.
projects / thirdparty / haproxy.git / log
