Willy Tarreau [Sat, 5 Nov 2016 16:52:06 +0000 (17:52 +0100)]
OPTIM: http: improve parsing performance of long URIs
Searching the trailing space in long URIs takes some time. This can
happen especially on static files and some blogs. By skipping valid
character ranges by 32-bit blocks, it's possible to increase the
HTTP performance from 212k to 216k req/s on requests features a
100-character URI, which is an increase of 2%. This is done for
architectures supporting unaligned accesses (x86_64, x86, armv7a).
There's only a 32-bit version because URIs are rarely long and very
often short, so it's more efficient to limit the systematic overhead
than to try to optimize for the rarest requests.
Willy Tarreau [Sat, 5 Nov 2016 16:35:40 +0000 (17:35 +0100)]
OPTIM: http: improve parsing performance of long header lines
A performance test with 1kB cookies was capping at 194k req/s. After
implementing multi-byte skipping, the performance increased to 212k req/s,
or 9.2% faster. This patch implements this for architectures supporting
unaligned accesses (x86_64, x86, armv7a). Maybe other architectures can
benefit from this but they were not tested yet.
Willy Tarreau [Sat, 5 Nov 2016 14:50:20 +0000 (15:50 +0100)]
OPTIM: http: move all http character classs tables into a single one
We used to have 7 different character classes, each was 256 bytes long,
resulting in almost 2kB being used in the L1 cache. It's as cheap to
test a bit than to check the byte is not null, so let's store a 7-bit
composite value and check for the respective bits there instead.
The executable is now 4 kB smaller and the performance on small
objects increased by about 1% to 222k requests/second with a config
involving 4 http-request rules including 1 header lookup, one header
replacement, and 2 variable assignments.
Willy Tarreau [Fri, 4 Nov 2016 17:47:01 +0000 (18:47 +0100)]
CLEANUP: tools: make ipcpy() preserve the original port
ipcpy() is used to replace an IP address with another one, but it
doesn't preserve the original port so all callers have to do it
manually while it's trivial to do there. Better do it inside the
function.
Willy Tarreau [Wed, 2 Nov 2016 21:37:03 +0000 (22:37 +0100)]
MEDIUM: tools: make str2ip2() preserve existing ports
Often we need to call str2ip2() on an address which already contains a
port without replacing it, so let's ensure we preserve it even if the
family changes.
Gabriele Cerami reported the the exit codes of the systemd-wrapper are
wrong. In short, it directly returns the output of the wait syscall's
status, which is a composite value made of error code an signal numbers.
In general it contains the signal number on the lower bits and the error
code on the higher bits, but exit() truncates it to the lowest 8 bits,
causing config validations to incorrectly report a success. Example :
$ ./haproxy-systemd-wrapper -c -f /dev/null
<7>haproxy-systemd-wrapper: executing /tmp/haproxy -c -f /dev/null -Ds
Configuration file has no error but will not start (no listener) => exit(2).
<5>haproxy-systemd-wrapper: exit, haproxy RC=512
$ echo $?
0
If the process is killed however, the signal number is directly reported
in the exit code.
Let's fix all this to ensure that the exit code matches what the shell does,
which means that codes 0..127 are for exit codes, codes 128..254 for signals,
and code 255 for unknown exit code. Now the return code is correct :
$ ./haproxy-systemd-wrapper -c -f /dev/null
<7>haproxy-systemd-wrapper: executing /tmp/haproxy -c -f /dev/null -Ds
Configuration file has no error but will not start (no listener) => exit(2).
<5>haproxy-systemd-wrapper: exit, haproxy RC=2
$ echo $?
2
Willy Tarreau [Mon, 31 Oct 2016 17:42:52 +0000 (18:42 +0100)]
MINOR: peers: remove the pointer to the stream
There's no reason to use the stream anymore, only the appctx should be
used by a peer. This was a leftover from the migration to appctx and it
caused some confusion, so let's totally drop it now. Note that half of
the patch are just comment updates.
Willy Tarreau [Mon, 31 Oct 2016 16:37:39 +0000 (17:37 +0100)]
MINOR: peers: make peer_session_forceshutdown() use the appctx and not the stream
It was inherited from initial code but we must only manipulate the appctx
and never the stream, otherwise we always risk shooting ourselves in the
foot.
Willy Tarreau [Mon, 31 Oct 2016 16:46:57 +0000 (17:46 +0100)]
BUG/MEDIUM: peers: fix use after free in peer_session_create()
In case of resource allocation error, peer_session_create() frees
everything allocated and returns a pointer to the stream/session that
was put back into the free pool. This stream/session is then assigned
to ps->{stream,session} with no error control. This means that it is
perfectly possible to have a new stream or session being both used for
a regular communication and for a peer at the same time.
In fact it is the only way (for now) to explain a CLOSE_WAIT on peers
connections that was caught in this dump with the stream interface in
SI_ST_CON state while the error field proves the state ought to have
been SI_ST_DIS, very likely indicating two concurrent accesses on the
same area :
Special thanks to Arnaud Gavara who provided lots of valuable input and
ran some validation testing on this patch.
This fix must be backported to 1.6 and 1.5. Note that in 1.5 the
session is not assigned from within the function so some extra checks
may be needed in the callers.
Willy Tarreau [Mon, 31 Oct 2016 16:32:20 +0000 (17:32 +0100)]
BUG/MEDIUM: peers: on shutdown, wake up the appctx, not the stream
This part was missed when peers were ported to the new applet
infrastructure in 1.6, the main stream is woken up instead of the
appctx. This creates a race condition by which it is possible to
wake the stream at the wrong moment and miss an event. This bug
might be at least partially responsible for some of the CLOSE_WAIT
that were reported on peers session upon reload in version 1.6.
Willy Tarreau [Tue, 25 Oct 2016 20:22:00 +0000 (22:22 +0200)]
[RELEASE] Released version 1.7-dev5
Released version 1.7-dev5 with the following main changes :
- MINOR: cfgparse: few memory leaks fixes.
- MEDIUM: log: Decompose %Tq in %Th %Ti %TR
- CLEANUP: logs: remove unused log format field definitions
- BUILD/MAJOR:updated 51d Trie implementation to incorperate latest update to 51Degrees.c
- BUG/MAJOR: stream: properly mark the server address as unset on connect retry
- CLEANUP: proto_http: Removing useless variable assignation
- CLEANUP: dumpstats: Removing useless variables allocation
- CLEANUP: dns: Removing usless variable & assignation
- BUG/MINOR: payload: fix SSLv2 version parser
- MINOR: cli: allow the semi-colon to be escaped on the CLI
- MINOR: cli: change a server health check port through the stats socket
- BUG/MINOR: Fix OSX compilation errors
- MAJOR: check: find out which port to use for health check at run time
- MINOR: server: introduction of 3 new server flags
- MINOR: new update_server_addr_port() function to change both server's ADDR and service PORT
- MINOR: cli: ability to change a server's port
- CLEANUP/MINOR dns: comment do not follow up code update
- MINOR: chunk: new strncat function
- MINOR: dns: wrong DNS_MAX_UDP_MESSAGE value
- MINOR: dns: new MAX values
- MINOR: dns: new macro to compute DNS header size
- MINOR: dns: new DNS structures to store received packets
- MEDIUM: dns: new DNS response parser
- MINOR: dns: query type change when last record is a CNAME
- MINOR: dns: proper domain name validation when receiving DNS response
- MINOR: dns: comments in types/dns.h about structures endianness
- BUG/MINOR: displayed PCRE version is running release
- MINOR: show Built with PCRE version
- MINOR: show Running on zlib version
- MEDIUM: make SO_REUSEPORT configurable
- MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
- BUG/MEDIUM: http/compression: Fix how chunked data are copied during the HTTP body parsing
- BUG/MINOR: stats: report the correct conn_time in backend's html output
- BUG/MEDIUM: dns: don't randomly crash on out-of-memory
- MINOR: Add fe_req_rate sample fetch
- MEDIUM: peers: Fix a peer stick-tables synchronization issue.
- MEDIUM: cli: register CLI keywords with cli_register_kw()
- BUILD: Make use of accept4() on OpenBSD.
- MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
- DOC: fix missed entry for "set-{src,dst}{,-port}"
- BUG/MINOR: vars: use sess and not s->sess in action_store()
- BUG/MINOR: vars: make smp_fetch_var() more robust against misuses
- BUG/MINOR: vars: smp_fetch_var() doesn't depend on HTTP but on the session
- MINOR: stats: output dcon
- CLEANUP: tcp rules: mention everywhere that tcp-conn rules are L4
- MINOR: counters: add new fields for denied_sess
- MEDIUM: tcp: add registration and processing of TCP L5 rules
- MINOR: stats: emit dses
- DOC: document tcp-request session
- MINOR: ssl: add debug traces
- BUILD/CLEANUP: ssl: Check BIO_reset() return code
- BUG/MINOR: ssl: Check malloc return code
- BUG/MINOR: ssl: prevent multiple entries for the same certificate
- BUG/MINOR: systemd: make the wrapper return a non-null status code on error
- BUG/MINOR: systemd: always restore signals before execve()
- BUG/MINOR: systemd: check return value of calloc()
- MINOR: systemd: report it when execve() fails
- BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or failed
- MINOR: proxy: add 'served' field to proxy, equal to total of all servers'
- MINOR: backend: add hash-balance-factor option for hash-type consistent
- MINOR: server: compute a "cumulative weight" to allow chash balancing to hit its target
- MEDIUM: server: Implement bounded-load hash algorithm
- SCRIPTS: make git-show-backports also dump a "git show" command
- MINOR: build: Allow linking to device-atlas library file
- MINOR: stats: Escape equals sign on socket dump
Chad Lavoie [Tue, 4 Oct 2016 20:10:40 +0000 (16:10 -0400)]
MINOR: stats: Escape equals sign on socket dump
Greetings,
Was recently working with a stick table storing URL's and one had an
equals sign in it (e.g. 127.0.0.1/f=ab) which made it difficult to
easily split the key and value without a regex.
This patch will change it so that the key looks like
"key=127.0.0.1/f\=ab" instead of "key=127.0.0.1/f=ab".
Not very important given that there are ways to work around it.
Willy Tarreau [Tue, 25 Oct 2016 20:12:54 +0000 (22:12 +0200)]
SCRIPTS: make git-show-backports also dump a "git show" command
This is very convenient for backport reviews as in a single
command you get all the patches one at a time with their
changelog and backport instructions.
The consistent hash lookup is done as normal, then if balancing is
enabled, we progress through the hash ring until we find a server that
doesn't have "too much" load. In the case of equal weights for all
servers, the allowed number of requests for a server is either the
floor or the ceil of (num_requests * hash-balance-factor / num_servers);
with unequal weights things are somewhat more complicated, but the
spirit is the same -- a server should not be able to go too far above
(its relative weight times) the average load. Using the hash ring to
make the second/third/etc. choice maintains as much locality as
possible given the load limit.
Andrew Rodland [Tue, 25 Oct 2016 16:49:45 +0000 (12:49 -0400)]
MINOR: server: compute a "cumulative weight" to allow chash balancing to hit its target
For active servers, this is the sum of the eweights of all active
servers before this one in the backend, and
[srv->cumulative_weight .. srv_cumulative_weight + srv_eweight) is a
space occupied by this server in the range [0 .. lbprm.tot_wact), and
likewise for backup servers with tot_wbck. This allows choosing a
server or a range of servers proportional to their weight, by simple
integer comparison.
Andrew Rodland [Tue, 25 Oct 2016 16:49:05 +0000 (12:49 -0400)]
MINOR: backend: add hash-balance-factor option for hash-type consistent
0 will mean no balancing occurs; otherwise it represents the ratio
between the highest-loaded server and the average load, times 100 (i.e.
a value of 150 means a 1.5x ratio), assuming equal weights.
Willy Tarreau [Tue, 25 Oct 2016 15:20:24 +0000 (17:20 +0200)]
BUG/MEDIUM: systemd: let the wrapper know that haproxy has completed or failed
Pierre Cheynier found that there's a persistent issue with the systemd
wrapper. Too fast reloads can lead to certain old processes not being
signaled at all and continuing to run. The problem was tracked down as
a race between the startup and the signal processing : nothing prevents
the wrapper from starting new processes while others are still starting,
and the resulting pid file will only contain the latest pids in this
case. This can happen with large configs and/or when a lot of SSL
certificates are involved.
In order to solve this we want the wrapper to wait for the new processes
to complete their startup. But we also want to ensure it doesn't wait for
nothing in case of error.
The solution found here is to create a pipe between the wrapper and the
sub-processes. The wrapper waits on the pipe and the sub-processes are
expected to close this pipe once they completed their startup. That way
we don't queue up new processes until the previous ones have registered
their pids to the pid file. And if anything goes wrong, the wrapper is
immediately released. The only thing is that we need the sub-processes
to know the pipe's file descriptor. We pass it in an environment variable
called HAPROXY_WRAPPER_FD.
It was confirmed both by Pierre and myself that this completely solves
the "zombie" process issue so that only the new processes continue to
listen on the sockets.
It seems that in the future this stuff could be moved to the haproxy
master process, also getting rid of an environment variable.
Willy Tarreau [Tue, 25 Oct 2016 14:51:40 +0000 (16:51 +0200)]
MINOR: systemd: report it when execve() fails
It's important to know that a signal sent to the wrapper had no effect
because something failed during execve(). Ideally more info (strerror)
should be reported. It would be nice to backport this to 1.6 and 1.5.
Willy Tarreau [Tue, 25 Oct 2016 14:49:31 +0000 (16:49 +0200)]
BUG/MINOR: systemd: always restore signals before execve()
Since signals are inherited, we must restore them before calling execve()
and intercept them again after a failed execve(). In order to cleanly deal
with the SIGUSR2/SIGHUP loops where we re-exec the wrapper, we ignore these
two signals during a re-exec, and restore them to defaults when spawning
haproxy.
Willy Tarreau [Tue, 25 Oct 2016 13:50:47 +0000 (15:50 +0200)]
BUG/MINOR: systemd: make the wrapper return a non-null status code on error
When execv() fails to execute the haproxy executable, it's important to
return an error instead of pretending everything is cool. This fix should
be backported to 1.6 and 1.5 in order to improve the overall reliability
under systemd.
BUG/MINOR: ssl: prevent multiple entries for the same certificate
Today, the certificate are indexed int he SNI tree using their CN and the
list of thier AltNames. So, Some certificates have the same names in the
CN and one of the AltNames entries.
Typically Let's Encrypt duplicate the the DNS name in the CN and the
AltName.
This patch prevents the creation of identical entries in the trees. It
checks the same DNS name and the same SSL context.
If the same certificate is registered two time it will be duplicated.
This patch should be backported in the 1.6 and 1.5 version.
Add some debug trace when haproxy is configured in debug & verbose mode.
This is useful for openssl tests. Typically, the error "SSL handshake
failure" can be caused by a lot of protocol error. This patch details
the encountered error. For exemple:
OpenSSL error 0x1408a0c1: ssl3_get_client_hello: no shared cipher
Note that my compilator (gcc-4.7) refuse to considers the function
ssl_sock_dump_errors() as inline. The condition "if" ensure that the
content of the function is not executed in normal case. It should be
a pity to call a function just for testing its execution condition, so
I use the macro "forceinline".
Willy Tarreau [Fri, 21 Oct 2016 14:37:51 +0000 (16:37 +0200)]
MEDIUM: tcp: add registration and processing of TCP L5 rules
This commit introduces "tcp-request session" rules. These are very
much like "tcp-request connection" rules except that they're processed
after the handshake, so it is possible to consider SSL information and
addresses rewritten by the proxy protocol header in actions. This is
particularly useful to track proxied sources as this was not possible
before, given that tcp-request content rules are processed after each
HTTP request. Similarly it is possible to assign the proxied source
address or the client's cert to a variable.
Willy Tarreau [Fri, 21 Oct 2016 14:34:21 +0000 (16:34 +0200)]
CLEANUP: tcp rules: mention everywhere that tcp-conn rules are L4
This is in order to make integration of tcp-request-session cleaner :
- tcp_exec_req_rules() was renamed tcp_exec_l4_rules()
- LI_O_TCP_RULES was renamed LI_O_TCP_L4_RULES
(LI_O_*'s horrible indent was also fixed and a provision was left
for L5 rules).
Willy Tarreau [Fri, 21 Oct 2016 16:15:32 +0000 (18:15 +0200)]
MINOR: stats: output dcon
These are denied conns. Strangely this wasn't emitted while it used to be
available for a while. It corresponds to the number of connections blocked
by "tcp-request connection reject".
Willy Tarreau [Fri, 21 Oct 2016 15:17:18 +0000 (17:17 +0200)]
BUG/MINOR: vars: smp_fetch_var() doesn't depend on HTTP but on the session
Thus the SMP_USE_HTTP_ANY dependency is incorrect, we have to depend on
SMP_USE_L5_CLI (the session). It's particularly important for session-wide
variables which are kept across HTTP requests. For now there is no impact
but it will make a difference with tcp-request session rules.
Willy Tarreau [Fri, 21 Oct 2016 15:14:35 +0000 (17:14 +0200)]
BUG/MINOR: vars: make smp_fetch_var() more robust against misuses
smp_fetch_var() may be called from everywhere since it just reads a
variable. It must ensure that the stream exists before trying to return
a stream-dependant variable. For now there is no impact but it will
cause trouble with tcp-request session rules.
Willy Tarreau [Fri, 21 Oct 2016 13:07:45 +0000 (15:07 +0200)]
MINOR: tcp: make set-src/set-src-port and set-dst/set-dst-port commutative
When the tcp/http actions above were introduced in 1.7-dev4, we used to
proceed like this :
- set-src/set-dst would force the port to zero
- set-src-port/set-dst-port would not do anything if the address family is
neither AF_INET nor AF_INET6.
It was a stupid idea of mine to request this behaviour because it ensures
that these functions cannot be used in a wide number of situations. Because
of the first rule, it is necessary to save the source port one way or
another if only the address has to be changed (so you have to use an
variable). Due to the second rule, there's no way to set the source port
on a unix socket without first overwriting the address. And sometimes it's
really not convenient, especially when there's no way to guarantee that all
fields will properly be set.
In order to fix all this, this small change does the following :
- set-src/set-dst always preserve the original port even if the address
family changes. If the previous address family didn't have a port (eg:
AF_UNIX), then the port is set to zero ;
- set-src-port/set-dst-port always preserve the original address. If the
address doesn't have a port, then the family is forced to IPv4 and the
address to "0.0.0.0".
Thanks to this it now becomes possible to perform one action, the other or
both in any order.
MEDIUM: cli: register CLI keywords with cli_register_kw()
To register a new cli keyword, you need to declare a cli_kw_list
structure in your source file:
static struct cli_kw_list cli_kws = {{ },{
{ { "test", "list", NULL }, "test list : do some tests on the cli", test_parsing, NULL },
{ { NULL }, NULL, NULL, NULL, NULL }
}};
And then register it:
cli_register_kw(&cli_kws);
The first field is an array of 5 elements, where you declare the
keywords combination which will match, it must be ended by a NULL
element.
The second field is used as a usage message, it will appear in the help
of the cli, you can set it to NULL if you don't want to show it, it's a
good idea if you want to overwrite some existing keywords.
The two last fields are callbacks.
The first one is used at parsing time, you can use it to parse the
arguments of your keywords and print small messages. The function must
return 1 in case of a failure, otherwise 0:
if (!*args[2]) {
appctx->ctx.cli.msg = "Error: the 3rd argument is mandatory !";
appctx->st0 = STAT_CLI_PRINT;
return 1;
}
chunk_reset(&trash);
chunk_printf(&trash, "arg[3]: %s\n", args[2]);
chunk_init(&out, NULL, 0);
chunk_dup(&out, &trash);
appctx->ctx.cli.err = out.str;
appctx->st0 = STAT_CLI_PRINT_FREE; /* print and free in the default cli_io_handler */
return 0;
}
The last field is the IO handler callback, it can be set to NULL if you
want to use the default cli_io_handler() otherwise you can write your
own. You can use the private pointer in the appctx if you need to store
a context or some data. stats_dump_sess_to_buffer() is a good example of
IO handler, IO handlers often use the appctx->st2 variable for the state
machine. The handler must return 0 in case it have to be recall later
otherwise 1.
MEDIUM: peers: Fix a peer stick-tables synchronization issue.
During the stick-table teaching process which occurs at reloading/restart time,
expiration dates of stick-tables entries were not synchronized between peers.
This patch adds two new stick-table messages to provide such a synchronization feature.
As these new messages are not supported by older haproxy peers protocol versions,
this patch increments peers protol version, from 2.0 to 2.1, to help in detecting/supporting
such older peers protocol implementations so that new versions might still be able
to transparently communicate with a newer one.
[wt: technically speaking it would be nice to have this backported into 1.6
as some people who reload often are affected by this design limitation, but
it's not a totally transparent change that may make certain users feel
reluctant to upgrade older versions. Let's let it cook in 1.7 first and
decide later]
Willy Tarreau [Sat, 1 Oct 2016 07:20:32 +0000 (09:20 +0200)]
BUG/MEDIUM: dns: don't randomly crash on out-of-memory
dns_init_resolvers() tries to emit the current resolver's name in the
error message in case of out-of-memory condition. But it must not do
it when initializing the trash before even having such a resolver
otherwise the user is certain to get a dirty crash instead of the
error message. No backport is needed.
Willy Tarreau [Sat, 1 Oct 2016 07:12:08 +0000 (09:12 +0200)]
BUG/MINOR: stats: report the correct conn_time in backend's html output
An apparent copy-paste error resulted in backend's avg connection time
to report the average queue time instead in the HTML dump. Backport to
1.6 is desired.
BUG/MEDIUM: http/compression: Fix how chunked data are copied during the HTTP body parsing
When the compression is enable on HTTP responses, the chunked data are copied in
a temporary buffer during the HTTP body parsing and then compressed when
everything is forwarded to the client. But the amout of data that can be copied
was not correctly calculated. In many cases, it worked, else on the edge when
the channel buffer was almost full.
[wt: bug introduced by b77c5c26 in 1.7-dev, no backport needed]
Lukas Tribus [Tue, 13 Sep 2016 09:51:15 +0000 (09:51 +0000)]
MINOR: enable IP_BIND_ADDRESS_NO_PORT on backend connections
Enable IP_BIND_ADDRESS_NO_PORT on backend connections when the source
address is specified without port or port ranges. This is supported
since Linux 4.2/libc 2.23.
If the kernel supports it but the libc doesn't, we can define it at
build time:
make [...] DEFINE=-DIP_BIND_ADDRESS_NO_PORT=24
For more informations about this feature, see Linux commit 90c337da
Lukas Tribus [Mon, 12 Sep 2016 21:42:20 +0000 (21:42 +0000)]
MEDIUM: make SO_REUSEPORT configurable
With Linux officially introducing SO_REUSEPORT support in 3.9 and
its mainstream adoption we have seen more people running into strange
SO_REUSEPORT related issues (a process management issue turning into
hard to diagnose problems because the kernel load-balances between the
new and an obsolete haproxy instance).
Also some people simply want the guarantee that the bind fails when
the old process is still bound.
This change makes SO_REUSEPORT configurable, introducing the command
line argument "-dR" and the noreuseport configuration directive.
MINOR: dns: comments in types/dns.h about structures endianness
To avoid issues when porting code to some architecture, we need to know
the endianess the structures are currently used.
This patch simply had a short notice before those structures to report
endianess and ease contributor's job.
MINOR: dns: proper domain name validation when receiving DNS response
The analyse of CNAME resolution and request's domain name was performed
twice:
- when validating the response buffer
- when loading the right IP address from the response
Now DNS response are properly loaded into a DNS response structure, we
do the domain name validation when loading/validating the response in
the DNS strcucture and later processing of this task is now useless.
MINOR: dns: query type change when last record is a CNAME
DNS servers don't return A or AAAA record if the query points to a CNAME
not resolving to the right type.
We know it because the last record of the response is a CNAME. We can
trigger a new query, switching to a new query type, handled by the layer
above.
Baptiste Assmann [Sat, 14 May 2016 09:26:22 +0000 (11:26 +0200)]
MEDIUM: dns: new DNS response parser
New DNS response parser function which turn the DNS response from a
network buffer into a DNS structure, much easier for later analysis
by upper layer.
Memory is pre-allocated at start-up in a chunk dedicated to DNS
response store.
New error code to report a wrong number of queries in a DNS response.
Enrichment of the 'set server <b>/<s> addr' cli directive to allow changing
now a server's port.
The new syntax looks like:
set server <b>/<s> addr [port <port>]
MINOR: new update_server_addr_port() function to change both server's ADDR and service PORT
This function can replace update_server_addr() where the need to change the
server's port as well as the IP address is required.
It performs some validation before performing each type of change.
Baptiste Assmann [Thu, 11 Aug 2016 21:12:18 +0000 (23:12 +0200)]
MINOR: server: introduction of 3 new server flags
Introduction of 3 new server flags to remember if some parameters were set
during configuration parsing.
* SRV_F_CHECKADDR: this server has a check addr configured
* SRV_F_CHECKPORT: this server has a check port configured
* SRV_F_AGENTADDR: this server has a agent addr configured
Baptiste Assmann [Mon, 13 Jun 2016 12:15:41 +0000 (14:15 +0200)]
MAJOR: check: find out which port to use for health check at run time
HAProxy used to deduce port used for health checks when parsing configuration
at startup time.
Because of this way of working, it makes it complicated to change the port at
run time.
The current patch changes this behavior and makes HAProxy to choose the
port used for health checking when preparing the check task itself.
A new type of error is introduced and reported when no port can be found.
There won't be any impact on performance, since the process to find out the
port value is made of a few 'if' statements.
This patch also introduces a new check state CHK_ST_PORT_MISS: this flag is
used to report an error in the case when HAProxy needs to establish a TCP
connection to a server, to perform a health check but no TCP ports can be
found for it.
And last, it also introduces a new stream termination condition:
SF_ERR_CHK_PORT. Purpose of this flag is to report an error in the event when
HAProxy has to run a health check but no port can be found to perform it.
SOL_IPV6 is not defined on OSX, breaking the compile. Also libcrypt is
not available for installation neither in Macports nor as a Brew recipe,
so we're disabling implicit dependancy.
Chad Lavoie [Thu, 26 May 2016 20:42:25 +0000 (16:42 -0400)]
MINOR: cli: allow the semi-colon to be escaped on the CLI
Today I was working on an auto-update script for some ACLs, and found
that I couldn't load ACL entries with a semi-colon in them no matter
how I tried to escape it.
As such, I wrote this patch (this one is for 1.7dev, but it applies to
1.5 the same with just line numbers changed), which seems to allow me
to execute a command such as "add acl /etc/foo.lst foo\;bar" over the
socket. It's worth noting that stats_sock_parse_request() already uses
the backslash to escape spaces in words so it makes sense to use it as
well to escape the semi-colon.
Willy Tarreau [Tue, 30 Aug 2016 12:39:46 +0000 (14:39 +0200)]
BUG/MINOR: payload: fix SSLv2 version parser
A typo resulting from a copy-paste in the original req.ssl_ver code will
make certain SSLv2 hello messages not properly detected. The bug has been
present since the code was added in 1.3.16. In 1.3 and 1.4, this code was
in proto_tcp.c. In 1.5-dev0, it moved to acl.c, then later to payload.c.
This bug was tagged "minor" because SSLv2 is outdated and this encoding
was rarely (if at all) used, the shorter form starting with 0x80 being
more common.
This fix needs to be backported to all currently maintained branches.
In dump_servers_state(), srv_time_since_last_change, bk_f_forced_id, srv_f_forced_id variables
were firstly set to zero and immediately reassigned to another value while never been accessed in between.
Sounds like a useless initiazation. So let's make only the useful allocation.
Willy Tarreau [Mon, 29 Aug 2016 12:46:54 +0000 (14:46 +0200)]
BUG/MAJOR: stream: properly mark the server address as unset on connect retry
Bartosz Koninski reported that recent commit 568743a ("BUG/MEDIUM: stream-int:
completely detach connection on connect error") introduced a nasty side effect
during the connection retry, causing a reconnect attempt to be performed on a
random address, possibly another server from another backend as shown in his
reproducer.
The real reason is in fact that by calling si_release_endpoint() after a
failed connect attempt and not clearing the SN_ADDR_SET flag on the
stream, we indicate that we continue to trust the connection's current
address. So upon next connect attempt, a connection is picked from the
pool and the last target address is reused. It is not very easy to
reproduce it 100% reliably out of production traffic, but it's easier to
see when haproxy is started with -dM to force the pools to be filled with
junk, and where subsequent connections are not even attempted due to their
family being incorrect.
The correct fix consists in clearing the SN_ADDR_SET flag after calling
si_release_endpoint() during a retry. But it outlines a deeper issue which
is in fact that the target address is *stored* in the connection and its
validity is stored in the stream. Until we have a better solution to store
target addresses, it would be better to rely on the connection flags
(CO_FL_ADDR_TO_SET) for this purpose. But it also outlines the fact that
the same issue still exists in Lua sockets and in idle sockets, which
fortunately are not affected by this issue.
Thanks to Bartosz for providing all the elements needed to understand the
problem.
BUILD/MAJOR:updated 51d Trie implementation to incorperate latest update to 51Degrees.c
Trie now uses a dataset structure just like Pattern, so this has been
defined in includes/types/global.h for both Pattern and Trie where it
was just Pattern.
In src/51d.c all functions used by the Trie implementation which need a
dataset as an argument now use the global dataset. The
fiftyoneDegreesDestroy method has now been replaced with
fiftyoneDegreesDataSetFree which is common to Pattern and Trie. In
addition, two extra dataset init status' have been added to the switch
statement in init_51degrees.
Tq is the time between the instant the connection is accepted and a
complete valid request is received. This time includes the handshake
(SSL / Proxy-Protocol), the idle when the browser does preconnect and
the request reception.
This patch decomposes %Tq in 3 measurements names %Th, %Ti, and %TR
which returns respectively the handshake time, the idle time and the
duration of valid request reception. It also adds %Ta which reports
the request's active time, which is the total time without %Th nor %Ti.
It replaces %Tt as the total time, reporting accurate measurements for
HTTP persistent connections.
%Th is avalaible for TCP and HTTP sessions, %Ti, %TR and %Ta are only
avalaible for HTTP connections.
In addition to this, we have new timestamps %tr, %trg and %trl, which
log the date of start of receipt of the request, respectively in the
default format, in GMT time and in local time (by analogy with %t, %T
and %Tl). All of them are obviously only available for HTTP. These values
are more relevant as they more accurately represent the request date
without being skewed by a browser's preconnect nor a keep-alive idle
time.
The HTTP log format and the CLF log format have been modified to
use %tr, %TR, and %Ta respectively instead of %t, %Tq and %Tt. This
way the default log formats now produce the expected output for users
who don't want to manually fiddle with the log-format directive.
=> 6 ms of SSL handshake, 2375 waiting before sending the first char (in
fact the time to type the first line), 261 ms before the end of the request,
no time spent in queue, 1 ms spend connecting to the server, immediate
response, total active time for this request = 262ms. Total time from accept
to close : 2643 ms.
The timing now decomposes like this :
first request 2nd request
|<-------------------------------->|<-------------- ...
t tr t tr ...
---|----|----|----|----|----|----|----|----|--
: Th Ti TR Tw Tc Tr Td : Ti ...
:<---- Tq ---->: :
:<-------------- Tt -------------->:
:<--------- Ta --------->:
Willy Tarreau [Sun, 14 Aug 2016 10:25:21 +0000 (12:25 +0200)]
[RELEASE] Released version 1.7-dev4
Released version 1.7-dev4 with the following main changes :
- MINOR: add list_append_word function
- MEDIUM: init: use list_append_word in haproxy.c
- MEDIUM: init: allow directory as argument of -f
- CLEANUP: config: detect double registration of a config section
- MINOR: log: add the %Td log-format specifier
- MEDIUM: filters: Move HTTP headers filtering in its own callback
- MINOR: filters: Simplify calls to analyzers using 2 new macros
- MEDIUM: filters: Add pre and post analyzer callbacks
- DOC: filters: Update the filters documentation accordingly to recent changes
- BUG/MEDIUM: init: don't use environment locale
- SCRIPTS: teach git-show-backports how to report upstream commits
- SCRIPTS: make git-show-backports capable of limiting its history
- BUG/MAJOR: fix listening IP address storage for frontends
- BUG/MINOR: fix listening IP address storage for frontends (cont)
- DOC: Fix typo so fetch is properly parsed by Cyril's converter
- BUG/MAJOR: http: fix breakage of "reqdeny" causing random crashes
- BUG/MEDIUM: stick-tables: fix breakage in table converters
- MINOR: stick-table: change all stick-table converters' inputs to SMP_T_ANY
- BUG/MEDIUM: dns: unbreak DNS resolver after header fix
- BUILD: fix build on Solaris 11
- BUG/MEDIUM: config: fix multiple declaration of section parsers
- BUG/MEDIUM: stats: show servers state may show an servers from another backend
- BUG/MEDIUM: fix risk of segfault with "show tls-keys"
- MEDIUM: dumpstats: 'show tls-keys' is now able to show secrets
- DOC: update doc about tls-tickets-keys dump
- MEDIUM: tcp: add 'set-src' to 'tcp-request connection'
- MINOR: set the CO_FL_ADDR_FROM_SET flags with 'set-src'
- MEDIUM: tcp/http: add 'set-src-port' action
- MEDIUM: tcp/http: new set-dst/set-dst-port actions
- BUG/MEDIUM: sticktables: segfault in some configuration error cases
- BUILD/MEDIUM: rebuild everything when an include file is changed
- BUILD/MEDIUM: force a full rebuild if some build options change
- BUG/MEDIUM: lua: converters doesn't work
- BUG/MINOR: http: add-header: header name copied twice
- BUG/MEDIUM: http: add-header: buffer overwritten
- BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
- MINOR: stream: export the function 'smp_create_src_stkctr'
- BUG/MEDIUM: dumpstats: undefined behavior in stats_tlskeys_list()
- MEDIUM: dumpstats: make stats_tlskeys_list() yield-aware during tls-keys dump
- BUG/MINOR: http: url32+src should use the big endian version of url32
- BUG/MINOR: http: url32+src should check cli_conn before using it
- DOC: http: add documentation for url32 and url32+src
- BUG/MINOR: fix http-response set-log-level parsing error
- MINOR: systemd: Use variable for config and pidfile paths
- MINOR: systemd: Perform sanity check on config before reload
- MEDIUM: ssl: support SNI filters with multicerts
- MINOR: ssl: crt-list parsing factor
- BUILD: ssl: fix typo causing a build failure in the multicert patch
- MINOR: listener: add the "accept-netscaler-cip" option to the "bind" keyword
- MINOR: tcp: add "tcp-request connection expect-netscaler-cip layer4"
- BUG/MINOR: init: always ensure that global.rlimit_nofile matches actual limits
- BUG/MINOR: init: ensure that FD limit is raised to the max allowed
- BUG/MEDIUM: external-checks: close all FDs right after the fork()
- BUG/MAJOR: external-checks: use asynchronous signal delivery
- BUG/MINOR: external-checks: do not unblock undesired signals
- CLEANUP: external-check: don't block/unblock SIGCHLD when manipulating the list
- BUG/MEDIUM: filters: Fix data filtering when data are modified
- BUG/MINOR: filters: Fix HTTP parsing when a filter loops on data forwarding
- BUG/MINOR: srv-state: fix incorrect output of state file
- BUG/MINOR: ssl: close ssl key file on error
- BUG/MINOR: http: fix misleading error message for response captures
- BUG/BUILD: don't automatically run "make" on "make install"
- DOC: add missing doc for http-request deny [deny_status <status>]
- CLEANUP: dumpstats: u64 field is an unsigned type.
- BUG/MEDIUM: http: unbreak uri/header/url_param hashing
- BUG/MINOR: Rework slightly commit 9962f8fc to clean code and avoid mistakes
- MINOR: new function my_realloc2 = realloc + free upon failure
- CLEANUP: fixed some usages of realloc leading to memory leak
- Revert "BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()"
- CLEANUP: connection: using internal struct to hold source and dest port.
- DOC: spelling fixes
- BUG/MINOR: ssl: fix potential memory leak in ssl_sock_load_dh_params()
- BUG/MEDIUM: dns: fix alignment issues in the DNS response parser
- BUG/MINOR: Fix endiness issue in DNS header creation code
- BUG/MEDIUM: lua: the function txn_done() from sample fetches can crash
- BUG/MEDIUM: lua: the function txn_done() from action wrapper can crash
- MEDIUM: http: implement http-response track-sc* directive
- BUG/MINOR: peers: Fix peers data decoding issue
- BUG/MINOR: peers: don't count track-sc multiple times on errors
- MINOR: standard: add function "escape_string"
- BUG/MEDIUM: log: use function "escape_string" instead of "escape_chunk"
- MINOR: tcp: Return TCP statistics like RTT and RTT variance
- DOC: lua: remove old functions
- BUG/MEDIUM: lua: somme HTTP manipulation functions are called without valid requests
- DOC: fix json converter example and error message
- BUG/MEDIUM: stream-int: completely detach connection on connect error
- DOC: minor typo fixes to improve HTML parsing by haproxy-dconv
- BUILD: make proto_tcp.c compatible with musl library
- BUG/MAJOR: compression: initialize avail_in/next_in even during flush
- BUG/MEDIUM: samples: make smp_dup() always duplicate the sample
- MINOR: sample: implement smp_is_safe() and smp_make_safe()
- MINOR: sample: provide smp_is_rw() and smp_make_rw()
- BUG/MAJOR: server: the "sni" directive could randomly cause trouble
- BUG/MEDIUM: stick-tables: do not fail on string keys with no allocated size
- BUG/MEDIUM: stick-table: properly convert binary samples to keys
- MINOR: sample: use smp_make_rw() in upper/lower converters
- MINOR: tcp: add dst_is_local and src_is_local
- BUG/MINOR: peers: some updates are pushed twice after a resync.
- BUILD: protocol: fix some build errors on OpenBSD
- BUILD: log: iovec requires to include sys/uio.h on OpenBSD
- BUILD: tcp: do not include netinet/ip.h for IP_TTL
- BUILD: connection: fix build breakage on openbsd due to missing in_systm.h
- BUILD: checks: remove the last strcat and eliminate a warning on OpenBSD
- BUILD: tcp: define SOL_TCP when only IPPROTO_TCP exists
- BUILD: compression: remove a warning when no compression lib is used
- BUILD: poll: remove unused hap_fd_isset() which causes a warning with clang
- MINOR: tcp: add further tcp info fetchers
- BUG/MINOR: peers: empty chunks after a resync.
- BUG/MAJOR: stick-counters: possible crash when using sc_trackers with wrong table
- MINOR: standard.c: ipcmp() function to compare 2 IP addresses stored in 2 struct sockaddr_storage
- MINOR: standard.c: ipcpy() function to copy an IP address from a struct sockaddr_storage into an other one
- MAJOR: listen section: don't use first bind port anymore when no server ports are provided
Baptiste Assmann [Sat, 30 Jan 2016 23:27:17 +0000 (00:27 +0100)]
MINOR: standard.c: ipcpy() function to copy an IP address from a struct sockaddr_storage into an other one
The function ipcpy() simply duplicates the IP address found in one
struct sockaddr_storage into an other struct sockaddr_storage.
It also update the family on the destination structure.
Memory of destination structure must be allocated and cleared by the
caller.
Willy Tarreau [Sun, 14 Aug 2016 10:02:55 +0000 (12:02 +0200)]
BUG/MAJOR: stick-counters: possible crash when using sc_trackers with wrong table
Bryan Talbot reported a very interesting bug. The sc_trackers() sample
fetch seems to have escaped the sanitization that was performed during 1.5
to ensure all dereferences of stkctr_entry() were safe. Here if a tacker
is set on a backend and is then checked against a different backend where
the entry doesn't exist, stkctr_entry() returns NULL and this is dereferenced
to retrieve the ref count. Thanks to Bryan for his detailed bug report featuring
a working config and reproducer.
Emeric Brun [Fri, 12 Aug 2016 09:23:31 +0000 (11:23 +0200)]
BUG/MINOR: peers: empty chunks after a resync.
After pushing the last update related to a resync process, the teacher resets
the re-connection's origin to the saved one (pointer position when he receive
the resync request). But the last acknowledgement overwrites this pointer to
an inconsistent value. In peersv2, it results with empty table chunks
regularly pushed.
The fix consist to move the confirm code to assure that the confirm message is
always sent after the last acknowledgement related to the resync. And to reset
the re-connection's origin to the saved one when a confirm message is received.
This bug affects versions 1.6 and superior.
For older versions (peersv1), this inconsistent state should not generate side
effects because chunks are not used and a next acknowlegement message resets
the pointer in a valid state.
Joe Williams [Wed, 10 Aug 2016 14:06:44 +0000 (07:06 -0700)]
MINOR: tcp: add further tcp info fetchers
Adding on to Thierry's work (http://git.haproxy.org/?p=haproxy.git;h=6310bef5)
I have added a few more fetchers for counters based on the tcp_info struct
maintained by the kernel :
Two fields were not added because they're version-dependant :
fc_rcv_rtt, fc_total_retrans
The fields name depend on the operating system. FreeBSD and NetBSD prefix
all the field names with "__" so we have to rely on a few #ifdef for
portability.
Willy Tarreau [Wed, 10 Aug 2016 19:09:24 +0000 (21:09 +0200)]
BUILD: tcp: define SOL_TCP when only IPPROTO_TCP exists
FreeBSD prefers to use IPPROTO_TCP over SOL_TCP, just like it does
with their *_IP counterparts. It's worth noting that there are a few
inconsistencies between SOL_TCP and IPPROTO_TCP in the code, eg on
TCP_QUICKACK. The two values are the same but it's worth applying
what implementations recommend.
No backport is needed, this was uncovered by the recent tcp_info stuff.
Willy Tarreau [Wed, 10 Aug 2016 17:29:09 +0000 (19:29 +0200)]
BUILD: checks: remove the last strcat and eliminate a warning on OpenBSD
OpenBSD emits warnings on usages of strcpy, strcat and sprintf (and
probably a few others). Here we have a single such warning in all the code
reintroduced by commit 0ba0e4a ("MEDIUM: Support sending email alerts") in
1.6-dev1. Let's get rid of it, the open-coding of strcat is as small as its
usage and the the result is even more efficient.
Willy Tarreau [Wed, 10 Aug 2016 16:57:38 +0000 (18:57 +0200)]
BUILD: connection: fix build breakage on openbsd due to missing in_systm.h
Recent commit 93b227d ("MINOR: listener: add the "accept-netscaler-cip"
option to the "bind" keyword") introduced an include of netinet/ip.h
which requires in_systm.h on OpenBSD. No backport is needed.
Willy Tarreau [Wed, 10 Aug 2016 16:42:17 +0000 (18:42 +0200)]
BUILD: tcp: do not include netinet/ip.h for IP_TTL
On OpenBSD, netinet/ip.h fails unless in_systm.h is included. This
include was added by the silent-drop feature introduced with commit 2d392c2 ("MEDIUM: tcp: add new tcp action "silent-drop"") in 1.6-dev6,
but we don't need it, IP_TTL is defined in netinet/in.h, so let's drop
this useless include.
Willy Tarreau [Tue, 9 Aug 2016 14:46:18 +0000 (16:46 +0200)]
MINOR: tcp: add dst_is_local and src_is_local
It is sometimes needed in application server environments to easily tell
if a source is local to the machine or a remote one, without necessarily
knowing all the local addresses (dhcp, vrrp, etc). Similarly in transparent
proxy configurations it is sometimes desired to tell the difference between
local and remote destination addresses.
This patch adds two new sample fetch functions for this :
dst_is_local : boolean
Returns true if the destination address of the incoming connection is local
to the system, or false if the address doesn't exist on the system, meaning
that it was intercepted in transparent mode. It can be useful to apply
certain rules by default to forwarded traffic and other rules to the traffic
targetting the real address of the machine. For example the stats page could
be delivered only on this address, or SSH access could be locally redirected.
Please note that the check involves a few system calls, so it's better to do
it only once per connection.
src_is_local : boolean
Returns true if the source address of the incoming connection is local to the
system, or false if the address doesn't exist on the system, meaning that it
comes from a remote machine. Note that UNIX addresses are considered local.
It can be useful to apply certain access restrictions based on where the
client comes from (eg: require auth or https for remote machines). Please
note that the check involves a few system calls, so it's better to do it only
once per connection.
Willy Tarreau [Tue, 9 Aug 2016 12:29:38 +0000 (14:29 +0200)]
MINOR: sample: use smp_make_rw() in upper/lower converters
There's no point in always duplicating the sample, just ensure it's
writable, as was done prior to the smp_dup() change. This should be
backported to 1.6 to avoid a performance regression caused by this
change (about 30% more time for upper/lower due to the copy).
Willy Tarreau [Tue, 9 Aug 2016 10:08:41 +0000 (12:08 +0200)]
BUG/MEDIUM: stick-table: properly convert binary samples to keys
The binary sample to stick-table key conversion is wrong. It doesn't
check that the binary sample is writable before padding it. After a
quick audit, it doesn't look like any existing sample fetch function
can trigger this bug. The correct fix consists in calling smp_make_rw()
prior to padding the sample.
projects / thirdparty / haproxy.git / log
