MEDIUM: fd: merge fdtab[].ev and state for FD_EV_* and FD_POLL_* into state
For a long time we've had fdtab[].ev and fdtab[].state which contain two
arbitrary sets of information, one is mostly the configuration plus some
shutdown reports and the other one is the latest polling status report
which also contains some sticky error and shutdown reports.
These ones used to be stored into distinct chars, complicating certain
operations and not even allowing to clearly see concurrent accesses (e.g.
fd_delete_orphan() would set the state to zero while fd_insert() would
only set the event to zero).
This patch creates a single uint with the two sets in it, still delimited
at the byte level for better readability. The original FD_EV_* values
remained at the lowest bit levels as they are also known by their bit
value. The next step will consist in merging the remaining bits into it.
The whole bits are now cleared both in fd_insert() and _fd_delete_orphan()
because after a complete check, it is certain that in both cases these
functions are the only ones touching these areas. Indeed, for
_fd_delete_orphan(), the thread_mask has already been zeroed before a
poller can call fd_update_event() which would touch the state, so it
is certain that _fd_delete_orphan() is alone. Regarding fd_insert(),
only one thread will get an FD at any moment, and it as this FD has
already been released by _fd_delete_orphan() by definition it is certain
that previous users have definitely stopped touching it.
Strictly speaking there's no need for clearing the state again in
fd_insert() but it's cheap and will remove some doubts during some
troubleshooting sessions.
MEDIUM: fd: prepare FD_POLL_* to move to bits 8-15
In preparation of merging FD_POLL* and FD_EV*, this only changes the
value of FD_POLL_* to use bits 8-15 (the second byte). The size of the
field has been temporarily extended to 32 bits already, as well as
the temporary variables that carry the new composite value inside
fd_update_events(). The resulting fdtab entry becomes temporarily
unaligned. All places making access to .ev or FD_POLL_* were carefully
inspected to make sure they were safe regarding this change. Only one
temporary update was needed for the "show fd" code. The code was only
slightly inflated at this step.
CLEANUP: fd: remove FD_POLL_DATA and FD_POLL_STICKY
The former was not used and the second was used only as a positive mask
of the flags to keep instead of having the flags that are updated. Both
were removed in favor of a new FD_POLL_UPDT_MASK that only mentions the
updated flags. This will ease merging of state and ev later.
BUG/MEDIUM: log: fix config parse error logging on stdout/stderr or any raw fd
The regression was introduced by commit previous commit 94aab06:
MEDIUM: log: support tcp or stream addresses on log lines.
This previous patch tries to retrieve the used protocol parsing
the address using the str2sa_range function but forgets that
the raw file descriptor adresses don't specify a protocol
and str2sa_range probes an error.
This patch re-work the str2sa_range function to stop
probing error if an authorized RAW_FD address is parsed
whereas the caller request also a protocol.
It also modify the code of parse_logsrv to switch on stream
logservers only if a protocol was detected.
MEDIUM: log: support tcp or stream addresses on log lines.
An explicit stream address prefix such as "tcp6@" "tcp4@"
"stream+ipv6@" "stream+ipv4@" or "stream+unix@" will
allocate an implicit ring buffer with a forward server
targeting the given address.
This is usefull to simply send logs to a log server in tcp
and It doesn't need to declare a ring section in configuration.
MINOR: server/bind: add support of new prefixes for addresses.
Since the internal function str2sa_range is used to addresses
for different objects ('server', 'bind' but also 'log' or
'nameserver') we notice that some combinations are missing.
"ip@" is introduced to authorize the prefix "dgram+ip@" or
"stream+ip@" which dectects automatically IP version but
specify dgram or stream.
"tcp@" was introduced and is an alias for "stream+ip@".
"tcp6" and "tcp4" are now aliases for "stream+ipv6@" and
"stream+ipv4@".
"uxst@" and "uxdg@" are now aliases for "stream+unix@" and
"dgram+unix@".
This patch also adds a complete section in documentation to
describe adresses and their prefixes.
BUG/MINOR: tools: fix parsing "us" unit for timers
Commit c20ad0d8dbd1bb5707bbfe23632415c3062e046c (BUG/MINOR: tools: make
parse_time_err() more strict on the timer validity) broke parsing the "us"
unit in timers. It caused `parse_time_err()` to return the string "s",
which indicates an error.
Now if the "u" is followed by an "s" we properly continue processing the
time instead of immediately failing.
This fixes #1209. It must be backported to all stable versions.
BUG/MINOR: hlua: Detect end of request when reading data for an HTTP applet
When a script retrieves request data from an HTTP applet, line per line or
not, we must be sure to properly detect the end of the request by checking
HTX_FL_EOM flag when everything was consumed. Otherwise, the script may
hang.
It is pretty easy to reproduce the bug by calling applet:receive() without
specifying any length. If the request is not chunked, the function never
returns.
The bug was introduced when the EOM block was removed. Thus, it is specific
to the 2.4. This patch should fix the issue #1207. No backport needed.
HTTP_2.0 predefined macro returns true for HTTP/2 requests. HTTP/2 doen't
convey a version information, so this macro may seem a bit strange. But for
compatiblity reasons, internally, the "HTTP/2.0" version is set. Thus, it is
handy to rely on it to differenciate HTTP/1 and HTTP/2 requests.
MINOR: No longer rely on deprecated sample fetches for predefined ACLs
Some predefined ACLs were still based on deprecated sample fetches, like
req_proto_http or req_ver. Now, they use non-deprecated sample fetches. In
addition, the usage lines in the configuration manual have been updated to
be more explicit.
CLEANUP: dev/flags: remove useless test in the stdin number parser
The test on "if (err)" after parsing a number was meant to be
"if (*err)" but in practise it will always be true since we at least
have a '\n' there, so no need for testing before writing zero.
This fixes issue #1211.
Released version 2.4-dev15 with the following main changes :
- BUG/MINOR: payload: Wait for more data if buffer is empty in payload/payload_lv
- BUG/MINOR: stats: Apply proper styles in HTML status page.
- BUG/MEDIUM: time: make sure to always initialize the global tick
- BUG/MINOR: tcp: fix silent-drop workaround for IPv6
- BUILD: tcp: use IPPROTO_IPV6 instead of SOL_IPV6 on FreeBSD/MacOS
- CLEANUP: socket: replace SOL_IP/IPV6/TCP with IPPROTO_IP/IPV6/TCP
- BUG/MINOR: http_fetch: make hdr_ip() resistant to empty fields
- BUG/MINOR: mux-h2: Don't emit log twice if an error occurred on the preface
- MINOR: stream: Don't trigger errors on destructive HTTP upgrades
- MINOR: frontend: Create HTTP txn for HTX streams
- MINOR: stream: Be sure to set HTTP analysers when creating an HTX stream
- BUG/MINOR: stream: Properly handle TCP>H1>H2 upgrades in http_wait_for_request
- BUG/MINOR: config: Add warning for http-after-response rules in TCP mode
- MINOR: muxes: Add a flag to notify a mux does not support any upgrade
- MINOR: mux-h1: Don't perform implicit HTTP/2 upgrade if not supported by mux
- MINOR: mux-pt: Don't perform implicit HTTP upgrade if not supported by mux
- MEDIUM: mux-h1: Expose h1 in the list of supported mux protocols
- MEDIUM: mux-pt: Expose passthrough in the list of supported mux protocols
- MINOR: muxes: Show muxes flags when the mux list is displayed
- DOC: config: Improve documentation about proto/check-proto keywords
- MINOR: stream: Use stream type instead of proxy mode when appropriate
- MINOR: filters/http-ana: Decide to filter HTTP headers in HTTP analysers
- MINOR: http-ana: Simplify creation/destruction of HTTP transactions
- MINOR: stream: Handle stream HTTP upgrade in a dedicated function
- MEDIUM: Add tcp-request switch-mode action to perform HTTP upgrade
- MINOR: config/proxy: Don't warn for HTTP rules in TCP if 'switch-mode http' set
- MINOR: config/proxy: Warn if a TCP proxy without backend is upgradable to HTTP
- DOC: config: Add documentation about TCP to HTTP upgrades
- REGTESTS: Add script to tests TCP to HTTP upgrades
- BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX streams/checks
- MINOR: htx: Make internal.strm.is_htx an internal sample fetch
- MINOR: action: Use a generic function to check validity of an action rule list
- MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy
- MEDIUM: http-rules: Add wait-for-body action on request and response side
- REGTESTS: Add script to tests the wait-for-body HTTP action
- BUG/MINOR: http-fetch: Fix test on message state to capture the version
- CLEANUP: vars: always pre-initialize smp in vars_parse_cli_get_var()
- MINOR: global: define diagnostic mode of execution
- MINOR: cfgparse: diag for multiple nbthread statements
- MINOR: server: diag for 0 weight server
- MINOR: diag: create cfgdiag module
- MINOR: diag: diag if servers use the same cookie value
- MINOR: config: diag if global section after non-global
- TESTS: slightly reorganize the code in the tests/ directory
- TESTS: move tests/*.cfg to tests/config
- REGTESTS: ssl: "set ssl cert" and multi-certificates bundle
- REGTESTS: ssl: mark set_ssl_cert_bundle.vtc as broken
- CONTRIB: halog: fix issue with array of type char
- CONTRIB: tcploop: add a shutr command
- CONTRIB: debug: add the show-fd-to-flags script
- CONTRIB: debug: split poll from flags
- CONTRIB: move some dev-specific tools to dev/
- BUILD: makefile: always build the flags utility
- DEV: flags: replace the unneeded makefile with a README
- BUILD: makefile: integrate the hpack tools
- CONTRIB: merge ip6range with iprange
- CONTRIB: move some admin-related sub-projects to admin/
- CONTRIB: move halog to admin/
- ADMIN: halog: automatically enable USE_MEMCHR on the right glibc version
- BUILD: makefile: build halog with the correct flags
- BUILD: makefile: add a "USE_PROMEX" variable to ease building prometheus-exporter
- CONTRIB: move prometheus-exporter to addons/promex
- DOC: add a few words about USE_* and the addons directory
- CONTRIB: move 51Degrees to addons/51degrees
- CONTRIB: move src/da.c and contrib/deviceatlas to addons/deviceatlas
- CONTRIB: move src/wurfl.c and contrib/wurfl to addons/wurfl
- CONTRIB: move contrib/opentracing to addons/ot
- BUG/MINOR: opentracing: initialization after establishing daemon mode
- DOC: clarify that compression works for HTTP/2
Julien Pivotto [Mon, 29 Mar 2021 10:41:40 +0000 (12:41 +0200)]
DOC: clarify that compression works for HTTP/2
This patch clarifies that compression also works with HTTP/2. I have
picked the wording "HTTP/1.1 or above" because it is already used
elsewhere in the documentation.
I have tested that compression indeed works in HTTP/2.
BUG/MINOR: opentracing: initialization after establishing daemon mode
This patch solves the problem reported in github issue #1204, where the
OpenTracing filter cannot communicate with the selected tracer if HAProxy
is run in daemon mode. The author of the reported issue uses Zipkin
tracer, while in this example Jaeger tracer is used (see gdb output below).
The problem is that the OpenTracing library is initialized before HAProxy
initialize the daemon mode. Establishing this mode kills the OpenTracing
thread, after which the correct operation of the OpenTracing filter is no
longer possible. Also, HAProxy crashes on deinitialization of the
OpenTracing library.
The initialization of the OpenTracing library has been moved from the
flt_ot_init() function (which is started before switching the HAProxy to
daemon mode) to the flt_ot_init_per_thread() function (which is run after
switching the HAProxy to daemon mode).
Gdb output of crashed HAProxy process:
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `../../../haproxy -f sa/haproxy.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f8131fd5629 in pthread_join (threadid=140192831239936, thread_return=0x0) at pthread_join.c:45
45 pthread_join.c: No such file or directory.
(gdb) where
#0 0x00007f8131fd5629 in pthread_join (threadid=140192831239936, thread_return=0x0) at pthread_join.c:45
#1 0x00007f812f15abc7 in std::thread::join() ()
from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
#2 0x00007f812f0fb6f7 in jaegertracing::reporters::RemoteReporter::close() ()
from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
#3 0x00007f812f0b7055 in jaegertracing::reporters::CompositeReporter::close() ()
from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
#4 0x00007f812f0b9136 in jaegertracing::Tracer::Close() ()
from /tmp/haproxy-os-master/contrib/opentracing/test/libjaeger_opentracing_plugin-0.5.0.so
#5 0x00007f81309def32 in ot_tracer_close (tracer=0x55fb48057390) at ../../src/tracer.cpp:91
#6 0x000055fb41785705 in ot_close (tracer=0x55fb48061168) at contrib/opentracing/src/opentracing.c:208
#7 0x000055fb4177fc64 in flt_ot_deinit (p=<optimized out>, fconf=<optimized out>) at contrib/opentracing/src/filter.c:215
#8 0x000055fb418bc038 in flt_deinit (proxy=proxy@entry=0x55fb4805ce50) at src/filters.c:360
#9 0x000055fb41893ed1 in free_proxy (p=0x55fb4805ce50) at src/proxy.c:315
#10 0x000055fb41888809 in deinit () at src/haproxy.c:2217
#11 0x000055fb41889078 in deinit_and_exit (status=0) at src/haproxy.c:2343
#12 0x000055fb4173d809 in main (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:3230
This patch should be backported to all branches where the OpenTracing
filter is located.
This one is the last optional module to build with haproxy, so let's move
it to addons/. It was renamed to "ot" as it was the only one whose USE_*
option did not match the directory name, now this is consistent.
Few changes were required, only the Makefile, and doc were adjusted, as
the directory was already self-contained and relocatable.
Now it's much cleaner, both 51d.c and the dummy library live together and
are easier to spot and maintain. The build howto probably ought to be moved
there as well. Makefile, docs and MAINTAINERS were updated, as well as
the github CI's build matrix, travis CI's, and coverity checks.
CONTRIB: move prometheus-exporter to addons/promex
Let's start to better organize the addons by moving promex there (and
with an easier directory name). The makefile and maintainers files were
updated, as well as the CI's build matrix.
BUILD: makefile: add a "USE_PROMEX" variable to ease building prometheus-exporter
The Prometheus exporter has gained in popularity and deserves to be easier
to build. Let's add a standard "USE_PROMEX" variable to enable it without
having to modify EXTRA_OBJS nor fiddling with the build path. The readme
was updated to reflect this.
BUILD: makefile: build halog with the correct flags
halog currently emits lots of warnings because it does not benefit from
the default flags. Let's update the main makefile to build it by itself
and remove the other one. The sub-project's makefile was replaced with
A readme indicating how to build it.
ADMIN: halog: automatically enable USE_MEMCHR on the right glibc version
There has been a USE_MEMCHR option for ages that was mostly never enabled
because it was unclear when glibc became faster. A quick look at the code
indicates that this arrived with the SSE implementation of memchr() which
arrived at commit 093ecf92998de2 between 2.14 and 2.15, so let's automatically
turn this on on x86_64 with glibc >= 2.15.
This results in ~6GB of logs read per second (20 million lines) and ~2.5GB/s
(8 million lines) parsed for errors or status codes classification, or 1 GB/s
(3 million lines) for time percentiles.
The few hpack development tools are now integrated into the main
makefile, which allows to remove the original one which was causing
lots of build warnings. A README was added to explain how to build
instead.
DEV: flags: replace the unneeded makefile with a README
The makefile was not suited anymore as it didn't consider all
required compiler options and was causing way too many build
warnings with modern compilers. Let's just remove it and indicate
that this has to be built from the top of the project.
This utility is absolutely required for developers and not having it
built by default is a real pain that tends to encourage keeping an
outdated copy somewhere else. Let's have it built by default then,
since it has no dependency and is ultra-small.
The following directories were moved from contrib/ to dev/ to make their
use case a bit clearer. In short, only developers are expected to ever
go there. The makefile was updated to build and clean from these ones.
Now poll is its own project and doesn't share the "flags" Makefile
any more. One of the issues was that it was making references to the
haproxy include path which is not needed here.
I just noticed this in the windows build after moving the file to dev/:
In file included from include/import/ist.h:32,
from include/haproxy/connection-t.h:32,
from dev/flags/flags.c:5:
dev/flags/flags.c: In function `main':
dev/flags/flags.c:442:20: error: array subscript has type `char' [-Werror=char-subscripts]
442 | (isalnum(*err) && toupper(*err) != 'U' && toupper(*err) != 'L'))
| ^~~~
LD haproxy
cc1: all warnings being treated as errors
make: *** [Makefile:932: dev/flags/flags.o] Error 1
make: *** Waiting for unfinished jobs....
Error: Process completed with exit code 2.
Let's just cast it to uchar as is done everywhere else.
These are a collection of test files for a variety of features (old or
more recent). 2 or 3 files were found lying there non-committed and
were moved at the same time. A few deprecated or obsolete keywords were
updated to their recent equivalent. Many of these configurations are
made to trigger different parsing errors so it is normal that plenty
of them fail.
Now the tests directory is cleaner and easier to navigate through.
TESTS: slightly reorganize the code in the tests/ directory
The code that is there to run some unit tests on some internal features
was moved to tests/unit. Ideally it should be buildable from the main
makefile though this is not yet the case.
The code that is kept for experimentation purposes (hashes, syscall
optimization etc) as well as some captures of the results was moved
to tests/exp.
A few totally obsolete files which couldn't build anymore and were
not relevant to current versions were removed.
Amaury Denoyelle [Tue, 30 Mar 2021 15:35:19 +0000 (17:35 +0200)]
MINOR: diag: diag if servers use the same cookie value
Add a diagnostic to check that two servers of the same backend does not
use the same cookie value. Ignore backup servers as it is quite common
for them to share a cookie value with a primary one.
Amaury Denoyelle [Mon, 29 Mar 2021 08:29:07 +0000 (10:29 +0200)]
MINOR: global: define diagnostic mode of execution
Define MODE_DIAG which is used to run haproxy in diagnostic mode. This
mode is used to output extra warnings about possible configuration
blunder or sub-optimal usage. It can be activated with argument '-dD'.
A new output function ha_diag_warning is implemented reserved for
diagnostic output. It serves to standardize the format of diagnostic
messages.
A macro HA_DIAG_WARN_COND is also available to automatically check if
diagnostic mode is on before executing the diagnostic check.
CLEANUP: vars: always pre-initialize smp in vars_parse_cli_get_var()
In issue #1200 Coverity believes we may use an uninitialized field
smp.sess here while it's not possible because the returned variable
necessarily matches SCOPE_PROC hence smp.sess is not used. But it
cannot see this and it could be confusing if the code later evolved
into something more complex. That's not a critical path so let's
first reset the sample.
BUG/MINOR: http-fetch: Fix test on message state to capture the version
A bug was introduced when the legacy HTTP mode was removed. To capture the
HTTP version of the request or the response, we rely on the message state to
be sure the status line was received. However, the test is inverted. The
version can be captured if message headers were received, not the opposite.
MEDIUM: http-rules: Add wait-for-body action on request and response side
Historically, an option was added to wait for the request payload (option
http-buffer-request). This option has 2 drawbacks. First, it is an ON/OFF
option for the whole proxy. It cannot be enabled on demand depending on the
message. Then, as its name suggests, it only works on the request side. The
only option to wait for the response payload was to write a dedicated
filter. While it is an acceptable solution for complex applications, it is a
bit overkill to simply match strings in the body.
To make everyone happy, this patch adds a dedicated HTTP action to wait for
the message payload, for the request or the response depending it is used in
an http-request or an http-response ruleset. The time to wait is
configurable and, optionally, the minimum payload size to have before stop
to wait.
Both the http action and the old http analyzer rely on the same internal
function.
MINOR: payload/config: Warn if a L6 sample fetch is used from an HTTP proxy
L6 sample fetches are now ignored when called from an HTTP proxy. Thus, a
warning is emitted during the startup if such usage is detected. It is true
for most ACLs and for log-format strings. Unfortunately, it is a bit painful
to do so for sample expressions.
This patch relies on the commit "MINOR: action: Use a generic function to
check validity of an action rule list".
MINOR: action: Use a generic function to check validity of an action rule list
The check_action_rules() function is now used to check the validity of an
action rule list. It is used from check_config_validity() function to check
L5/6/7 rulesets.
MINOR: htx: Make internal.strm.is_htx an internal sample fetch
It is not really a context-less sample fetch, but it is internal. And it
only fails if no stream is attached to the sample. This way, it is still
possible to use it on an HTTP proxy (L6 sample fetches are ignored now for
HTTP proxies).
If the commit "BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX
streams/checks" is backported, it may be a good idea to backport this one
too. But only as far as 2.2.
BUG/MINOR: payload/htx: Ingore L6 sample fetches for HTX streams/checks
Use a L6 sample fetch on an HTX streams or a HTX health-check is meaningless
because data are not raw but structured. So now, these sample fetches fail
when called from an HTTP proxy. In addition, a warning has been added in the
configuration manual, at the begining of the L6 sample fetches section.
Note that req.len and res.len samples return the HTX data size instead of
failing. It is not accurate because it does not reflect the buffer size nor
the raw data length. But we keep it for backward compatibility purpose.
However it remains a bit strange to use it on an HTTP proxy.
This patch may be backported to all versions supporting the HTX, i.e as far
as 2.0. But the part about the health-checks is only valid for the 2.2 and
upper.
DOC: config: Add documentation about TCP to HTTP upgrades
This patch adds explanation about chaining a TCP frontend to an HTTP
backend. It also explain how the HTTP upgrades work in this context. A note
has also been added in "Fetching HTTP samples" section to warning about HTTP
content processing in TCP.
MINOR: config/proxy: Warn if a TCP proxy without backend is upgradable to HTTP
If a 'switch-mode http' tcp action is configured on a listener with no
backend, a warning is displayed to remember HTTP connections cannot be
routed to TCP servers. Indeed, backend connection is still established using
the proxy mode.
MEDIUM: Add tcp-request switch-mode action to perform HTTP upgrade
It is now possible to perform HTTP upgrades on a TCP stream from the
frontend side. To do so, a tcp-request content rule must be defined with the
switch-mode action, specifying the mode (for now, only http is supported)
and optionnaly the proto (h1 or h2).
This way it could be possible to set HTTP directives on a TCP frontend which
will only be evaluated if an upgrade is performed. This new way to perform
HTTP upgrades should replace progressively the old way, consisting to route
the request to an HTTP backend. And it should be also a good start to remove
all HTTP processing from tcp-request content rules.
This action is terminal, it stops the ruleset evaluation. It is only
available on proxy with the frontend capability.
The configuration manual has been updated accordingly.
MINOR: http-ana: Simplify creation/destruction of HTTP transactions
Now allocation and initialization of HTTP transactions are performed in a
unique function. Historically, there were two functions because the same TXN
was reset for K/A connections in the legacy HTTP mode. Now, in HTX, K/A
connections are handled at the mux level. A new stream, and thus a new TXN,
is created for each request. In addition, the function responsible to end
the TXN is now also reponsible to release it.
So, now, http_create_txn() and http_destroy_txn() must be used to create and
destroy an HTTP transaction.
MINOR: filters/http-ana: Decide to filter HTTP headers in HTTP analysers
It is just a small cleanup. AN_REQ_FLT_HTTP_HDRS and AN_RES_FLT_HTTP_HDRS
analysers are now set in HTTP analysers at the same place
AN_REQ_HTTP_XFER_BODY and AN_RES_HTTP_XFER_BODY are set.
MINOR: stream: Use stream type instead of proxy mode when appropriate
We now use the stream instead of the proxy to know if we are processing HTTP
data or not. If the stream is an HTX stream, it means we are dealing with
HTTP data. It is more accurate than the proxy mode because when an HTTP
upgrade is performed, the proxy is not changed and only the stream may be
used.
Note that it was not a problem to rely on the proxy because HTTP upgrades
may only happen when an HTTP backend was set. But, we will add the support
of HTTP upgrades on the frontend side, after te tcp-request rules
evaluation. In this context, we cannot rely on the proxy mode.
DOC: config: Improve documentation about proto/check-proto keywords
This patch adds a description about information provided by "haproxy -vv"
command regarding the available protocols. The description is adapted
depending the context (bind line, server line or health-check).
MEDIUM: mux-pt: Expose passthrough in the list of supported mux protocols
Add "none" in the list of supported mux protocols. It relies on the
passthrough multiplexer and use almost the same mux_ops structure. Only the
flags differ because this "new" mux does not support the upgrades. "none"
was chosen to explicitly stated there is not processing at the mux level.
Thus it is now possible to set "proto none" or "check-proto none" on
bind/server lines, depending on the context. However, when set, no upgrade
to HTTP is performed. It may be a way to disable HTTP upgrades per bind
line.
MEDIUM: mux-h1: Expose h1 in the list of supported mux protocols
Add "h1" in the list of supported mux protocols. It relies on the H1
multiplexer and use the almost the same mux_ops structure. Only the flags
differ because this "new" mux does not support the upgrades.
Thus it is now possible to set "proto h1" or "check-proto h1" on bind/server
lines, depending on the context. However, when set, no upgrade to HTTP/2 is
performed. It may be a way to disable implicit HTTP/2 upgrades per bind
line.
MINOR: muxes: Add a flag to notify a mux does not support any upgrade
MX_FL_NO_UPG flag may now be set on a multiplexer to explicitly disable
upgrades from this mux. For now, it is set on the FCGI multiplexer because
it is not supported and there is no upgrade on backend-only multiplexers. It
is also set on the H2 multiplexer because it is clearly not supported.
BUG/MINOR: config: Add warning for http-after-response rules in TCP mode
No warning is emitted if some http-after-response rules are configured on a
TCP proxy while such warning messages are emitted for other HTTP ruleset in
same condition. It is just an oversight.
BUG/MINOR: stream: Properly handle TCP>H1>H2 upgrades in http_wait_for_request
When a TCP stream is first upgraded to H1 and then to H2, we must be sure to
inhibit any connect and to properly handle the TCP stream destruction.
When the TCP stream is upgraded to H1, the HTTP analysers are set. Thus
http_wait_for_request() is called. In this case, the server connection must
be blocked, waiting for the request analysis. Otherwise, a server may be
assigned to the stream too early. It is especially a problem if the stream
is finally destroyed because of an implicit upgrade to H2.
In this case, the stream processing must be properly aborted to not have a
stalled stream. Thus, if a shutdown is detected in http_wait_for_request()
when an HTTP upgrade is performed, the stream is aborted.
MINOR: stream: Be sure to set HTTP analysers when creating an HTX stream
Always set frontend HTTP analysers when an HTX stream is created. It is only
useful in case a destructive HTTP upgrades (TCP>H2) because the frontend is
a TCP proxy.
In fact, to be strict, we must only set these analysers when the upgrade is
performed before setting the backend (it is not supported yet, but this
patch is required to do so), in the frontend part. If the upgrade happens
when the backend is set, it means the HTTP processing is just the backend
buisness. But there is no way to make the difference when a stream is
created, at least for now.
When an HTX stream is created, be sure to always create the HTTP txn object,
regardless of the ".http_needed" value of the frontend. That happens when a
destructive HTTP upgrades is performed (TCP>H2). The frontend is a TCP
proxy. If there is no dependency on the HTTP part, the HTTP transaction is
not created at this stage but only when the backend is set. For now, it is
not a problem. But an HTTP txn will be mandatory to fully support TCP to
HTTP upgrades after frontend tcp-request rules evaluation.
MINOR: stream: Don't trigger errors on destructive HTTP upgrades
When a TCP stream is upgraded to H2 stream, a destructive upgrade is
performed. It means the TCP stream is silently released while a new one is
created. It is of course more complicated but it is what we observe from the
stream point of view.
That was performed by returning an error when the backend was set. It is
neither really elegant nor accurate. So now, instead of returning an error
from stream_set_backend() in case of destructive HTTP upgrades, the TCP
stream processing is aborted and no error is reported. However, the result
is more or less the same.
Willy Tarreau [Wed, 31 Mar 2021 09:41:36 +0000 (11:41 +0200)]
BUG/MINOR: http_fetch: make hdr_ip() resistant to empty fields
The fix in commit 7b0e00d94 ("BUG/MINOR: http_fetch: make hdr_ip() reject
trailing characters") made hdr_ip() more sensitive to empty fields, for
example if a trusted proxy incorrectly sends the header with an empty
value, we could return 0.0.0.0 which is not correct. Let's make sure we
only assign an IPv4 type here when a non-empty address was found.
This should be backported to all branches where the fix above was
backported.
Willy Tarreau [Wed, 31 Mar 2021 06:45:47 +0000 (08:45 +0200)]
CLEANUP: socket: replace SOL_IP/IPV6/TCP with IPPROTO_IP/IPV6/TCP
Historically we've used SOL_IP/SOL_IPV6/SOL_TCP everywhere as the socket
level value in getsockopt() and setsockopt() but as we've seen over time
it regularly broke the build and required to have them defined to their
IPPROTO_* equivalent. The Linux ip(7) man page says:
Using the SOL_IP socket options level isn't portable; BSD-based
stacks use the IPPROTO_IP level.
And it indeed looks like a pure linuxism inherited from old examples and
documentation. strace also reports SOL_* instead of IPPROTO_*, which does
not help... A check to linux/in.h shows they have the same values. Only
SOL_SOCKET and other non-IP values make sense since there is no IPPROTO
equivalent.
Let's get rid of this annoying confusion by removing all redefinitions of
SOL_IP/IPV6/TCP and using IPPROTO_* instead, just like any other operating
system. This also removes duplicated tests for the same value.
Note that this should not result in exposing syscalls to other OSes
as the only ones that were still conditionned to SOL_IPV6 were for
IPV6_UNICAST_HOPS which already had an IPPROTO_IPV6 equivalent, and
IPV6_TRANSPARENT which is Linux-specific.
Willy Tarreau [Wed, 31 Mar 2021 06:29:27 +0000 (08:29 +0200)]
BUILD: tcp: use IPPROTO_IPV6 instead of SOL_IPV6 on FreeBSD/MacOS
Lukas reported in issue #1203 that the previous fix for silent-drop in
commit ab79ee8b1 ("BUG/MINOR: tcp: fix silent-drop workaround for IPv6")
breaks the build on FreeBSD/MacOS due to SOL_IPV6 not being defined. On
these platforms, IPPROTO_IPV6 must be used instead, so this should fix
it.
This needs to be backported to whatever version the fix above is backported
to.
Willy Tarreau [Tue, 30 Mar 2021 15:23:50 +0000 (17:23 +0200)]
BUG/MINOR: tcp: fix silent-drop workaround for IPv6
As reported in github issue #1203 the TTL-based workaround that is used
when permissions are insufficient for the TCP_REPAIR trick does not work
for IPv6 because we're using only SOL_IP with IP_TTL. In IPv6 we have to
use SOL_IPV6 and IPV6_UNICAST_HOPS. Let's pick the right one based on the
source address's family.
Willy Tarreau [Tue, 30 Mar 2021 16:13:26 +0000 (18:13 +0200)]
BUG/MEDIUM: time: make sure to always initialize the global tick
The issue with non-rotating freq counters was addressed in commit 8cc586c73
("BUG/MEDIUM: freq_ctr/threads: use the global_now_ms variable") using the
global date. But an issue remained with the comparison of the most recent
time. Since the initial time in the structure is zero, the tick_is_lt()
works on half of the periods depending on the first date an entry is
touched. And the wrapping happened last night:
$ date --date=@$(((($(date +%s) * 1000) & -0x8000000) / 1000))
Mon Mar 29 23:59:46 CEST 2021
So users of the last fix (backported to 2.3.8) may experience again an
always increasing rate for the next 24 days if they restart their process.
Let's always update the time if the latest date was not updated yet. It
will likely be simplified once the function is reorganized but this will
do the job for now.
Note that since this timer is only used by freq counters, no other
sub-system is affected. The bug can easily be tested with this config
during the right time period (i.e. today to today+24 days + N*49.7 days):
global
stats socket /tmp/sock1
frontend web
bind :8080
mode http
http-request track-sc0 src
stick-table type ip size 1m expire 1h store http_req_rate(2s)
Issuing 'socat - /tmp/sock1 <<< "show table web"' should show a stable
rate after 2 seconds.
The fix must be backported to 2.3 and any other version the fix above
goes into.
Thanks to Thomas SIMON and Sander Klein for quickly reporting this issue
with a working reproducer.
BUG/MINOR: stats: Apply proper styles in HTML status page.
When a backend is in status DOWN and going UP it is currently displayed
as yellow ("active UP, going down") instead of orange ("active DOWN, going
UP"). This patches restyles the table rows to actually match the
legend.
This may be backported to any version, the issue appeared in 1.7-dev2
with commit 0c378efe8 ("MEDIUM: stats: compute the color code only in
the HTML form").
BUG/MINOR: payload: Wait for more data if buffer is empty in payload/payload_lv
In payload() and payload_lv() sample fetches, if the buffer is empty, we
must wait for more data by setting SMP_F_MAY_CHANGE flag on the sample.
Otherwise, when it happens in an ACL, nothing is returned (because the
buffer is empty) and the ACL is considered as finished (success or failure
depending on the test).
As a workaround, the buffer length may be tested first. For instance :
Willy Tarreau [Sat, 27 Mar 2021 08:42:09 +0000 (09:42 +0100)]
[RELEASE] Released version 2.4-dev14
Released version 2.4-dev14 with the following main changes :
- MEDIUM: quic: Fix build.
- MEDIUM: quic: Fix build.
- CI: codespell: whitelist "Dragan Dosen"
- CLEANUP: assorted typo fixes in the code and comments
- CI: github actions: update LibreSSL to 3.2.5
- REGTESTS: revert workaround for a crash with recent libressl on http-reuse sni
- CLEANUP: mark defproxy as const on parse tune.fail-alloc
- REGTESTS: remove unneeded experimental-mode in cli add server test
- REGTESTS: wait for proper return of enable server in cli add server test
- MINOR: compression: use pool_alloc(), not pool_alloc_dirty()
- MINOR: spoe: use pool_alloc(), not pool_alloc_dirty()
- MINOR: fcgi-app: use pool_alloc(), not pool_alloc_dirty()
- MINOR: cache: use pool_alloc(), not pool_alloc_dirty()
- MINOR: ssl: use pool_alloc(), not pool_alloc_dirty()
- MINOR: opentracing: use pool_alloc(), not pool_alloc_dirty()
- MINOR: dynbuf: make b_alloc() always check if the buffer is allocated
- CLEANUP: compression: do not test for buffer before calling b_alloc()
- CLEANUP: l7-retries: do not test the buffer before calling b_alloc()
- MINOR: channel: simplify the channel's buffer allocation
- MEDIUM: dynbuf: remove last usages of b_alloc_margin()
- CLEANUP: dynbuf: remove b_alloc_margin()
- CLEANUP: dynbuf: remove the unused b_alloc_fast() function
- CLEANUP: pools: remove the unused pool_get_first() function
- MINOR: pools: make the pool allocator support a few flags
- MINOR: pools: add pool_zalloc() to return a zeroed area
- CLEANUP: connection: use pool_zalloc() in conn_alloc_hash_node()
- CLEANUP: filters: use pool_zalloc() in flt_stream_add_filter()
- CLEANUP: spoe: use pool_zalloc() instead of pool_alloc+memset
- CLEANUP: frontend: use pool_zalloc() in frontend_accept()
- CLEANUP: mailers: use pool_zalloc() in enqueue_one_email_alert()
- CLEANUP: resolvers: use pool_zalloc() in resolv_link_resolution()
- CLEANUP: ssl: use pool_zalloc() in ssl_init_keylog()
- CLEANUP: tcpcheck: use pool_zalloc() instead of pool_alloc+memset
- CLEANUP: quic: use pool_zalloc() instead of pool_alloc+memset
- MINOR: time: also provide a global, monotonic global_now_ms timer
- BUG/MEDIUM: freq_ctr/threads: use the global_now_ms variable
- MINOR: tools: introduce new option PA_O_DEFAULT_DGRAM on str2sa_range.
- BUILD: tools: fix build error with new PA_O_DEFAULT_DGRAM
- BUG/MINOR: ssl: Prevent disk access when using "add ssl crt-list"
- CLEANUP: ssl: remove unused definitions
- BUILD: ssl: guard ecdh functions with SSL_CTX_set_tmp_ecdh macro
- MINOR: lua: Slightly improve function dumping the lua traceback
- BUG/MEDIUM: debug/lua: Use internal hlua function to dump the lua traceback
- BUG/MEDIUM: lua: Always init the lua stack before referencing the context
- MINOR: fd: make fd_clr_running() return the remaining running mask
- MINOR: fd: remove the unneeded running bit from fd_insert()
- BUG/MEDIUM: fd: do not wait on FD removal in fd_delete()
- CLEANUP: fd: remove unused fd_set_running_excl()
- CLEANUP: fd: slightly simplify up _fd_delete_orphan()
- BUG/MEDIUM: fd: Take the fd_mig_lock when closing if no DWCAS is available.
- BUG/MEDIUM: release lock on idle conn killing on reached pool high count
- BUG/MEDIUM: thread: Fix a deadlock if an isolated thread is marked as harmless
- MINOR: tools: make url2ipv4 return the exact number of bytes parsed
- BUG/MINOR: http_fetch: make hdr_ip() reject trailing characters
- BUG/MEDIUM: mux-h1: make h1_shutw_conn() idempotent
- BUG/MINOR: ssl: Fix update of default certificate
- BUG/MINOR: ssl: Prevent removal of crt-list line if the instance is a default one
- BUILD: ssl: introduce fine guard for ssl random extraction functions
- REORG: global: move initcall register code in a dedicated file
- REORG: global: move free acl/action in their related source files
- REORG: split proxy allocation functions
- MINOR: proxy: implement a free_proxy function
- MINOR: proxy: define cap PR_CAP_LUA
- MINOR: lua: properly allocate the lua Socket proxy
- MINOR: lua: properly allocate the lua Socket servers
- MINOR: vars: make get_vars() allow the session to be null
- MINOR: vars: make the var() sample fetch keyword depend on nothing
- CLEANUP: sample: remove duplicate "stopping" sample fetch keyword
- MINOR: sample: make smp_resolve_args() return an allocate error message
- MINOR: sample: add a new SMP_SRC_CONST sample capability
- MINOR: sample: mark the truly constant sample fetch keywords as such
- MINOR: sample: add a new CFG_PARSER context for samples
- MINOR: action: add a new ACT_F_CFG_PARSER origin designation
- MEDIUM: vars: add support for a "set-var" global directive
- REGTESTS: add a basic reg-test for some "set-var" commands
- MINOR: sample: add a new CLI_PARSER context for samples
- MINOR: action: add a new ACT_F_CLI_PARSER origin designation
- MINOR: vars/cli: add a "get var" CLI command to retrieve global variables
- MEDIUM: cli: add a new experimental "set var" command
- MINOR: compat: add short aliases for a few very commonly used types
- BUILD: ssl: use EVP_CIPH_GCM_MODE macro instead of HA_OPENSSL_VERSION
- MEDIUM: backend: use a trylock to grab a connection on high FD counts as well
Willy Tarreau [Fri, 26 Mar 2021 19:52:10 +0000 (20:52 +0100)]
MEDIUM: backend: use a trylock to grab a connection on high FD counts as well
Commit b1adf03df ("MEDIUM: backend: use a trylock when trying to grab an
idle connection") solved a contention issue on the backend under normal
condition, but there is another one further, which only happens when the
number of FDs in use is considered too high, and which obviously causes
random crashes with just 16 threads once the number of FDs is about to be
exhausted.
Like the aforementioned patch, this one should be backported to 2.3.
Willy Tarreau [Fri, 26 Mar 2021 16:28:47 +0000 (17:28 +0100)]
MINOR: compat: add short aliases for a few very commonly used types
Very often we use "int" where negative numbers are not needed (and can
further cause trouble) just because it's painful to type "unsigned int"
or "unsigned", or ugly to use in function arguments. Similarly sometimes
chars would absolutely need to be signed but nobody types "signed char".
Let's add a few aliases for such types and make them part of the standard
internal API so that over time we can get used to them and get rid of
horrible definitions. A comment also reminds some commonly available
types and their properties regarding other types.
Willy Tarreau [Fri, 26 Mar 2021 14:19:50 +0000 (15:19 +0100)]
MEDIUM: cli: add a new experimental "set var" command
set var <name> <expression>
Allows to set or overwrite the process-wide variable 'name' with the result
of expression <expression>. Only process-wide variables may be used, so the
name must begin with 'proc.' otherwise no variable will be set. The
<expression> may only involve "internal" sample fetch keywords and converters
even though the most likely useful ones will be str('something') or int().
Note that the command line parser doesn't know about quotes, so any space in
the expression must be preceeded by a backslash. This command requires levels
"operator" or "admin". This command is only supported on a CLI connection
running in experimental mode (see "experimental-mode on").
Just like for "set-var" in the global section, the command uses a temporary
dummy proxy to create a temporary "set-var(name)" rule to assign the value.
The reg test was updated to verify that an updated global variable is properly
reflected in subsequent HTTP responses.
Willy Tarreau [Fri, 26 Mar 2021 13:51:31 +0000 (14:51 +0100)]
MINOR: vars/cli: add a "get var" CLI command to retrieve global variables
Process-wide variables can now be displayed from the CLI using "get var"
followed by the variable name. They must all start with "proc." otherwise
they will not be found. The output is very similar to the one of the
debug converter, with a type and value being reported for the embedded
sample.
This command is limited to clients with the level "operator" or higher,
since it can possibly expose traffic-related data.
Willy Tarreau [Fri, 26 Mar 2021 14:36:44 +0000 (15:36 +0100)]
MINOR: action: add a new ACT_F_CLI_PARSER origin designation
In order to process samples from the command line interface we'll need
rules as well, and these rules will have to be marked as coming from
the CLI parser. This new origin is used for this.
Willy Tarreau [Fri, 26 Mar 2021 14:29:35 +0000 (15:29 +0100)]
MINOR: sample: add a new CLI_PARSER context for samples
In order to prepare for supporting calling sample expressions from the
CLI, let's create a new CLI_PARSER parsing context. This one supports
constants and internal samples only.
Willy Tarreau [Fri, 26 Mar 2021 13:03:57 +0000 (14:03 +0100)]
REGTESTS: add a basic reg-test for some "set-var" commands
This reg-test tests "set-var" in the global section, with some overlapping
variables and using a few samples and converters, then at the TCP and HTTP
levels using proc/sess/req variables.
Willy Tarreau [Fri, 26 Mar 2021 10:38:08 +0000 (11:38 +0100)]
MEDIUM: vars: add support for a "set-var" global directive
While we do support process-wide variables ("proc.<name>"), there was
no way to preset them from the configuration. This was particularly
limiting their usefulness since configs involving them always had to
first check if the variable was set prior to performing an operation.
This patch adds a new "set-var" directive in the global section that
supports setting the proc.<name> variables from an expression, like
other set-var actions do. The syntax however follows what is already
being done for setenv, which consists in having one argument for the
variable name and another one for the expression.
Only "constant" expressions are allowed here, such as "int", "str"
etc, combined with arithmetic or string converters, and variable
lookups. A few extra sample fetch keywords like "date", "rand" and
"uuid" are also part of the constant expressions and may make sense
to allow to create a random key or differentiate processes.
The way it was done consists in parsing a dummy rule an executing the
expression in the CFG_PARSE context, then releasing the expression.
This is safe because the sample that variables store does not hold a
back pointer to expression that created them.
Willy Tarreau [Fri, 26 Mar 2021 10:11:34 +0000 (11:11 +0100)]
MINOR: action: add a new ACT_F_CFG_PARSER origin designation
In order to process samples from the config file we'll need rules as
well, and these rules will have to be marked as coming from the
config parser. This new origin is used for this.
Willy Tarreau [Fri, 26 Mar 2021 10:09:38 +0000 (11:09 +0100)]
MINOR: sample: add a new CFG_PARSER context for samples
We'd sometimes like to be able to process samples while parsing
the configuration based on purely internal thing but that's not
possible right now. Let's add a new CFG_PARSER context for samples
which only permits constant samples (i.e. those which do not change
in the process' life and which are stable during config parsing).
Willy Tarreau [Fri, 26 Mar 2021 11:03:11 +0000 (12:03 +0100)]
MINOR: sample: mark the truly constant sample fetch keywords as such
A number of keywords are really constant and safe to use at config
time. This is the case for str(), int() etc but also env(), hostname(),
nbproc() etc. By extension a few other ones which can be useful to
preset values in a configuration were enabled as well, like data(),
rand() or uuid(). At the moment this doesn't change anything as they
are still only usable from runtime rules.
The "var()" keyword was also marked as const as it can definitely
return stable stuff at boot time.
Willy Tarreau [Fri, 26 Mar 2021 10:56:11 +0000 (11:56 +0100)]
MINOR: sample: add a new SMP_SRC_CONST sample capability
This level indicates that everything it constant in the expression during
the whole process' life and that it may safely be used at config parsing
time.
Willy Tarreau [Fri, 26 Mar 2021 15:11:55 +0000 (16:11 +0100)]
MINOR: sample: make smp_resolve_args() return an allocate error message
For now smp_resolve_args() complains on stderr via ha_alert(), but if we
want to make it a bit more dynamic, we need it to return errors in an
allocated message. Let's pass it an error pointer and have it fill it.
On return we indent the output if it contains more than one line.
The "stopping" sample fetch keyword was accidently duplicated in 1.9
by commit 70fe94419 ("MINOR: sample: add cpu_calls, cpu_ns_avg,
cpu_ns_tot, lat_ns_avg, lat_ns_tot"). This has no effect so no
backport is needed.
Willy Tarreau [Fri, 26 Mar 2021 11:08:06 +0000 (12:08 +0100)]
MINOR: vars: make the var() sample fetch keyword depend on nothing
This sample fetch doesn't require any L4 client session in practice, as
get_var() now checks for the session. This is important to remove this
dependency in order to support accessing variables in scope "proc" from
anywhere.
projects / thirdparty / haproxy.git / log
