Willy Tarreau [Wed, 17 Feb 2021 15:26:55 +0000 (16:26 +0100)]
OPTIM: lb-leastconn: do not unlink the server if it did not change
Due to the two-phase server reservation, there are 3 calls to
fwlc_srv_reposition() per request, one during assign_server() to reserve
the slot, one in connect_server() to commit it, and one in process_stream()
to release it. However only one of the first two will change the key, so
it's needlessly costly to take the lock, remove a server and insert it
again at the same place when we can already figure we ought not to even
have taken the lock.
Furthermore, even when the server needs to move, there can be quite some
contention on the lbprm lock forcing the thread to wait. During this time
the served and nbpend server values might have changed, just like the
lb_node.key itself. Thus we measure the values again under the lock
before updating the tree. Measurements have shown that under contention
with 16 servers and 16 threads, 50% of the updates can be avoided there.
This patch makes the function compute the new key and compare it to
the current one before deciding to move the entry (and does it again
under the lock forthe second test).
This removes between 40 and 50% of the tree updates depending on the
thread contention and the number of servers. The performance gain due
to this (on 16 threads) was:
One point worth thinking about is that it's not logic to update the
tree 2-3 times per request while it's only read once. half to 2/3 of
these updates are not needed. An experiment consisting in subscribing
the server to a list and letting the readers reinsert them on the fly
showed further degradation instead of an improvement.
A better approach would probably consist in avoinding writes to shared
cache lines by having the leastconn nodes distinct from the servers,
with one node per value, and having them hold an mt-list with all the
servers having that number of connections. The connection count tree
would then be read-mostly instead of facing heavy writes, and most
write operations would be performed on 1-3 list heads which are way
cheaper to migrate than a tree node, and do not require updating the
last two updated neighbors' cache lines.
Willy Tarreau [Wed, 17 Feb 2021 15:14:00 +0000 (16:14 +0100)]
OPTIM: lb-leastconn: do not take the server lock on take_conn/drop_conn
The operations are only an insert and a delete into the LB tree, which
doesn't require the server's lock at all as the lbprm lock is already
held. Let's drop it. Just for the sake of cleanness, given that the
served and nbpend values used to be atomically updated, we'll use an
atomic load to read them.
Willy Tarreau [Wed, 17 Feb 2021 15:15:23 +0000 (16:15 +0100)]
OPTIM: lb-first: do not take the server lock on take_conn/drop_conn
The operations are only an insert and a delete into the LB tree, which
doesn't require the server's lock at all as the lbprm lock is already
held. Let's drop it.
Willy Tarreau [Wed, 17 Feb 2021 15:01:37 +0000 (16:01 +0100)]
MINOR: lb/api: let callers of take_conn/drop_conn tell if they have the lock
The two algos defining these functions (first and leastconn) do not need the
server's lock. However it's already present in pendconn_process_next_strm()
so the API must be updated so that the functions may take it if needed and
that the callers indicate whether they already own it.
As such, the call places (backend.c and stream.c) now do not take it
anymore, queue.c was unchanged since it's already held, and both "first"
and "leastconn" were updated to take it if not already held.
A quick test on the "first" algo showed a jump from 432 to 565k rps by
just dropping the lock in stream.c!
Repeated tests have shown a small perforamnce degradation of ~1.8%
caused by this patch at high request rates on 16 threads. The exact
cause is not yet perfectly known but it probably stems in slower
accesses for non-64-bit aligned atomic accesses.
Willy Tarreau [Wed, 17 Feb 2021 12:33:24 +0000 (13:33 +0100)]
OPTIM: server: switch the actconn list to an mt-list
The remaining contention on the server lock solely comes from
sess_change_server() which takes the lock to add and remove a
stream from the server's actconn list. This is both expensive
and pointless since we have mt-lists, and this list is only
used by the CLI's "shutdown server sessions" command!
Let's migrate to an mt-list and remove the need for this costly
lock. By doing so, the request rate increased by ~1.8%.
Willy Tarreau [Wed, 17 Feb 2021 13:33:58 +0000 (14:33 +0100)]
DEBUG: thread: add 5 extra lock labels for statistics and debugging
Since OTHER_LOCK is commonly used it's become much more difficult to
profile lock contention by temporarily changing a lock label. Let's
add DEBUG1..5 to serve only for debugging. These ones must not be
used in committed code. We could decide to only define them when
DEBUG_THREAD is set but that would complicate attempts at measuring
performance with debugging turned off.
Willy Tarreau [Wed, 17 Feb 2021 14:20:19 +0000 (15:20 +0100)]
BUG/MEDIUM: checks: don't needlessly take the server lock in health_adjust()
The server lock was taken preventively for anything in health_adjust(),
including the static config checks needed to detect that the lock was not
needed, while the function is always called on the response path to update
a server's status. This was responsible for huge contention causing a
performance drop of about 17% on 16 threads. Let's move the lock only
where it should be, i.e. inside the function around the critical sections
only. By doing this, a 16-thread process jumped back from 575 to 675 krps.
This should be backported to 2.3 as the situation degraded there, and
maybe later to 2.2.
Willy Tarreau [Wed, 17 Feb 2021 14:15:15 +0000 (15:15 +0100)]
BUG/MINOR: checks: properly handle wrapping time in __health_adjust()
There's an issue when a server state changes, we use an integer comparison
to decide whether or not to reschedule a test instead of using a wrapping
timer comparison. This will cause some health-checks not to be immediately
triggered half of the time, and some unneeded calls to task_queue() to be
performed in other cases.
This bug has always been there as it was introduced with the commit that
added the feature, 97f07b832 ("[MEDIUM] Decrease server health based on
http responses / events, version 3"). This may be backported everywhere.
Amaury Denoyelle [Wed, 17 Feb 2021 14:59:02 +0000 (15:59 +0100)]
BUG/MINOR: backend: do not call smp_make_safe for sni conn hash
conn_hash_prehash does not need a nul-terminated string, thus it is only
needed to test if the sni sample is not null before using it as
connection hash input.
Moreover, a bug could be introduced between smp_make_safe and
ssl_sock_set_servername call. Indeed, smp_make_safe may call smp_dup
which duplicates the sample in the trash buffer. If another function
manipulates the trash buffer before the call to ssl_sock_set_servername,
the sni sample might be erased. Currently, no function seems to do that
except make_proxy_line in case proxy protocol is used simultaneously
with the sni on the server.
Willy Tarreau [Tue, 16 Feb 2021 17:08:12 +0000 (18:08 +0100)]
BUG/MINOR: session: atomically increment the tracked sessions counter
In session_count_new() the tracked counter was still incremented with
a "++" outside of any lock, resulting in occasional slightly off values
such as the following:
Amaury Denoyelle [Tue, 16 Feb 2021 14:16:17 +0000 (15:16 +0100)]
BUG/MAJOR: connection: prevent double free if conn selected for removal
Always try to remove a connexion from its toremove_list in conn_free.
This prevents a double-free in case the connection is freed but was
already added in toremove_list.
This bug was easily reproduced by running 4-5 runs of inject on a
single-thread instance of haproxy :
$ inject -u 10000 -d 10 -G 127.0.0.1:20080
A crash would soon be triggered in srv_cleanup_toremove_connections.
Emeric Brun [Mon, 15 Feb 2021 13:28:27 +0000 (14:28 +0100)]
BUG/MINOR: dns: dns_connect_server must return -1 unsupported nameserver's type
This patch fix returns code in case of dns_connect_server is called
on unsupported type (which should not happen). Doing this we have
the warranty that after a return 0 the fd is never -1.
This patch should fix github issues #1127, #1128 and #1130
Emeric Brun [Mon, 15 Feb 2021 12:58:06 +0000 (13:58 +0100)]
BUG/MINOR: dns: add test on result getting value from buffer into ring.
This patch adds a missing test in dns_session_io_handler, getting
the query id from the buffer of the ring. An error should never
happen since messages are completely added atomically.
this was a missing piece for a while now even though it was planned. This
patch adds listen stats.
Nothing in particular but we make use of the status helper previously
added. `promex_st_metrics` diff also looks scary, but I had to realign
all lines.
William Dauchy [Sun, 14 Feb 2021 22:22:55 +0000 (23:22 +0100)]
MINOR: stats: add helper to get status string
move listen status to a helper, defining both status enum and string
definition.
this will be helpful to be reused in prometheus code. It also removes
this hard-to-read nested ternary.
William Dauchy [Sun, 14 Feb 2021 22:22:54 +0000 (23:22 +0100)]
MEDIUM: stats: allow to select one field in `stats_fill_li_stats`
prometheus approach requires to output all values for a given metric
name; meaning we iterate through all metrics, and then iterate in the
inner loop on all objects for this metric.
In order to allow more code reuse, adapt the stats API to be able to
select one field or fill them all otherwise.
From this patch it should be possible to add support for listen stats in
prometheus.
BUG/MINOR: server: Remove RMAINT from admin state when loading server state
The RMAINT admin state is dynamic and should be remove from the
srv_admin_state parameter when a server state is loaded from a server-state
file. Otherwise an erorr is reported, the server-state line is ignored and
the server state is not updated.
This patch should fix the issue #576. It must be backported as far as 1.8.
Willy Tarreau [Sat, 13 Feb 2021 09:17:27 +0000 (10:17 +0100)]
[RELEASE] Released version 2.4-dev8
Released version 2.4-dev8 with the following main changes :
- BUILD: ssl: fix typo in HAVE_SSL_CTX_ADD_SERVER_CUSTOM_EXT macro
- BUILD: ssl: guard SSL_CTX_add_server_custom_ext with special macro
- BUG/MINOR: mux-h1: Don't emit extra CRLF for empty chunked messages
- MINOR: contrib/prometheus-exporter: use stats desc when possible followup
- MEDIUM: contrib/prometheus-exporter: export base stick table stats
- CLEANUP: assorted typo fixes in the code and comments
- CLEANUP: check: fix some typo in comments
- CLEANUP: tools: typo in `strl2irc` mention
- BUILD: ssl: guard SSL_CTX_set_msg_callback with SSL_CTRL_SET_MSG_CALLBACK macro
- MEDIUM: ssl: add a rwlock for SSL server session cache
- BUG/MINOR: intops: fix mul32hi()'s off-by-one
- BUG/MINOR: freq_ctr: fix a wrong delay calculation in next_event_delay()
- MINOR: stick-tables/counters: add http_fail_cnt and http_fail_rate data types
- MINOR: ssl: add SSL_SERVER_LOCK label in threads.h
- BUG/MINOR: mux-h1: Don't increment HTTP error counter for 408/500/501 errors
- BUG/MINOR: http-ana: Don't increment HTTP error counter on internal errors
- BUG/MEDIUM: mux-h1: Always set CS_FL_EOI for response in MSG_DONE state
- BUG/MINOR: mux-h1: Fix data skipping for bodyless responses
- BUG/MINOR: mux-h1: Don't blindly skip EOT block for non-chunked messages
- BUG/MEDIUM: mux-h2: Add EOT block when EOM flag is set on an empty HTX message
- MINOR: mux-h1: Be sure EOM flag is set when processing end of outgoing message
- REGTESTS: Add a script to test payload skipping for bodyless HTTP responses
- BUG/MINOR: server: re-align state file fields number
- CLEANUP: muxes: Remove useless calls to b_realign_if_empty()
- BUG/MINOR: tools: Fix a memory leak on error path in parse_dotted_uints()
- CLEANUP: remove unused variable assigned found by Coverity
- CLEANUP: queue: Remove useless tests on p or pp in pendconn_process_next_strm()
- BUG/MINOR: backend: hold correctly lock when killing idle conn
- MEDIUM: connection: protect idle conn lists with locks
- MEDIUM: connection: replace idle conn lists by eb trees
- MINOR: backend: search conn in idle/safe trees after available
- MINOR: backend: search conn in idle tree after safe on always reuse
- MINOR: connection: prepare hash calcul for server conns
- MINOR: connection: use the srv pointer for the srv conn hash
- MINOR: backend: compare conn hash for session conn reuse
- MINOR: connection: use sni as parameter for srv conn hash
- MINOR: reg-tests: test http-reuse with sni
- MINOR: backend: rewrite alloc of stream target address
- MINOR: connection: use dst addr as parameter for srv conn hash
- MINOR: reg-test: test http-reuse with specific dst addr
- MINOR: backend: rewrite alloc of connection src address
- MINOR: connection: use src addr as parameter for srv conn hash
- MINOR: connection: use proxy protocol as parameter for srv conn hash
- MINOR: reg-tests: test http-reuse with proxy protocol
- MINOR: doc: update http reuse for new eligilible connections
- BUG/MINOR: backend: fix compilation without ssl
- REGTESTS: adjust http_reuse_conn_hash requirements
- REGTESTS: deactivate a failed test on CI in http_reuse_conn_hash
- REGTESTS: fix sni used in http_reuse_conn_hash for libressl 3.3.0
- CI: cirrus: update FreeBSD image to 12.2
- MEDIUM: cli: add check-addr command
- MEDIUM: cli: add agent-port command
- MEDIUM: server: add server-states version 2
- MEDIUM: server: support {check,agent}_addr, agent_port in server state
- MINOR: server: enhance error precision when applying server state
- BUG/MINOR: server: Fix server-state-file-name directive
- CLEANUP: deinit: release global and per-proxy server-state variables on deinit
- BUG/MEDIUM: config: don't pick unset values from last defaults section
- BUG/MINOR: stats: revert the change on ST_CONVDONE
- BUG/MINOR: cfgparse: do not mention "addr:port" as supported on proxy lines
- BUG/MINOR: http-htx: defpx must be a const in proxy_dup_default_conf_errors()
- BUG/MINOR: tcpheck: the source list must be a const in dup_tcpcheck_var()
- BUILD: proxy: add missing compression-t.h to proxy-t.h
- REORG: move init_default_instance() to proxy.c and pass it the defproxy pointer
- REORG: proxy: centralize the proxy allocation code into alloc_new_proxy()
- MEDIUM: proxy: only take defaults when a default proxy is passed.
- MINOR: proxy: move the defproxy freeing code to proxy.c
- MINOR: proxy: always properly reset the just freed default instance pointers
- BUG/MINOR: extcheck: proxy_parse_extcheck() must take a const for the defproxy
- BUG/MINOR: tcpcheck: proxy_parse_*check*() must take a const for the defproxy
- BUG/MINOR: server: parse_server() must take a const for the defproxy
- MINOR: cfgparse: move defproxy to cfgparse-listen as a static
- MINOR: proxy: add a new capability PR_CAP_DEF
- MINOR: cfgparse: check PR_CAP_DEF instead of comparing poiner against defproxy
- MINOR: cfgparse: use a pointer to the current default proxy
- MINOR: proxy: also store the name for a defaults section
- MINOR: proxy: support storing defaults sections into their own tree
- MEDIUM: proxy: store the default proxies in a tree by name
- MEDIUM: cfgparse: allow a proxy to designate the defaults section to use
- MINOR: http: add baseq sample fetch
- CLEANUP: tcpcheck: Remove a useless test on port variable
- BUG/MINOR: server: Don't call fopen() with server-state filepath set to NULL
- CLEANUP: server: Remove useless "filepath" variable in apply_server_state()
- MINOR: peers/cli: do not dump the peers dictionaries by default on "show peers"
- MINOR: cfgparse: implement a simple if/elif/else/endif macro block handler
- DOC: tune: explain the origin of block size for ssl.cachesize
- MINOR: tcp: add support for defer-accept on FreeBSD.
- MINOR: ring: adds new ring_init function.
- CLEANUP: channel: fix comment in ci_putblk.
- BUG/MINOR: dns: add missing sent counter and parent id to dns counters.
- BUG/MINOR: resolvers: fix attribute packed struct for dns
- MINOR: resolvers: renames some resolvers internal types and removes dns prefix
- MINOR: resolvers: renames type dns_resolvers to resolvers.
- MINOR: resolvers: renames some resolvers specific types to not use dns prefix
- MINOR: resolvers: renames some dns prefixed types using resolv prefix.
- MINOR: resolvers: renames resolvers DNS_RESP_* errcodes RSLV_RESP_*
- MINOR: resolvers: renames resolvers DNS_UPD_* returncodes to RSLV_UPD_*
- MINOR: resolvers: rework prototype suffixes to split resolving and dns.
- MEDIUM: resolvers: move resolvers section parsing from cfgparse.c to dns.c
- MINOR: resolvers: replace nameserver's resolver ref by generic parent pointer
- MINOR: resolvers: rework dns stats prototype because specific to resolvers
- MEDIUM: resolvers: split resolving and dns message exchange layers.
- MEDIUM: resolvers/dns: split dns.c into dns.c and resolvers.c
- MEDIUM: dns: adds code to support pipelined DNS requests over TCP.
- MEDIUM: resolvers: add supports of TCP nameservers in resolvers.
Emeric Brun [Fri, 12 Feb 2021 19:05:45 +0000 (20:05 +0100)]
MEDIUM: resolvers: add supports of TCP nameservers in resolvers.
This patch introduce the new line "server" to set a TCP
nameserver in a "resolvers" section:
server <name> <address> [param*]
Used to configure a DNS TCP or stream server. This supports for all
"server" parameters found in 5.2 paragraph. Some of these parameters
are irrelevant for DNS resolving. Note: currently 4 queries are pipelined
on the same connections. A batch of idle connections are removed every
5 seconds. "maxconn" can be configured to limit the amount of those
concurrent connections and TLS should also usable if the server supports
. The current implementation limits to 4 pipelined
The name of the line in configuration is open to discussion
and could be changed before the next release.
Emeric Brun [Fri, 12 Feb 2021 19:03:38 +0000 (20:03 +0100)]
MEDIUM: dns: adds code to support pipelined DNS requests over TCP.
This patch introduce the "dns_stream_nameserver" to use DNS over
TCP on strict nameservers. For the upper layer it is analog to
the api used with udp nameservers except that the user que switch
the name server in "stream" mode at the init using "dns_stream_init".
The fallback from UDP to TCP is not handled and this is not the
purpose of this feature. This is done to choose the transport layer
during the initialization.
Currently there is a hardcoded limit of 4 pipelined transactions
per TCP connections. A batch of idle connections is expired every 5s.
This code is designed to support a maximum DNS message size on TCP: 64k.
Note: this code won't perform retry on unanswered queries this
should be handled by the upper layer
Emeric Brun [Fri, 12 Feb 2021 18:42:55 +0000 (19:42 +0100)]
MEDIUM: resolvers/dns: split dns.c into dns.c and resolvers.c
This patch splits current dns.c into two files:
The first dns.c contains code related to DNS message exchange over UDP
and in future other TCP. We try to remove depencies to resolving
to make it usable by other stuff as DNS load balancing.
The new resolvers.c inherit of the code specific to the actual
resolvers.
Note:
It was really difficult to obtain a clean diff dur to the amount
of moved code.
Note2:
Counters and stuff related to stats is not cleany separated because
currently counters for both layers are merged and hard to separate
for now.
Emeric Brun [Mon, 4 Jan 2021 12:32:20 +0000 (13:32 +0100)]
MEDIUM: resolvers: split resolving and dns message exchange layers.
This patch splits recv and send functions in two layers. the
lowest is responsible of DNS message transactions over
the network. Doing this we could use DNS message layer
for something else than resolving. Load balancing for instance.
This patch also re-works the way to init a nameserver and
introduce the new struct dns_dgram_server to prepare the arrival
of dns_stream_server and the support of DNS over TCP.
The way to retry a send failure of a request because of EAGAIN
was re-worked. Previously there was no control and all "pending"
queries were re-played each time it reaches a EAGAIN. This
patch introduce a ring to stack messages in case of sent
failure. This patch is emptied if poller shows that the
socket is ready again to push messages.
Emeric Brun [Wed, 6 Jan 2021 15:01:02 +0000 (16:01 +0100)]
MINOR: resolvers: rework dns stats prototype because specific to resolvers
Counters are currently stored into lowlevel nameservers struct but
most of them are resolving layer data and increased in the upper layer
So this patch renames the prototype used to allocate/dump them with prefix
'resolv' waiting for a clean split.
Emeric Brun [Mon, 4 Jan 2021 09:40:46 +0000 (10:40 +0100)]
BUG/MINOR: dns: add missing sent counter and parent id to dns counters.
Resolv callbacks are also updated to rely on counters and not on
nameservers.
"show stat domain dns" will now show the parent id (i.e. resolvers
section name).
David Carlier [Sat, 6 Feb 2021 12:11:11 +0000 (12:11 +0000)]
MINOR: tcp: add support for defer-accept on FreeBSD.
FreeBSD has a kernel feature (accf) and a sockopt flag similar to the
Linux's TCP_DEFER_ACCEPT to filter incoming data upon ACK. The main
difference is the filter needs to be placed when the socket actually
listens.
William Dauchy [Fri, 12 Feb 2021 14:58:46 +0000 (15:58 +0100)]
DOC: tune: explain the origin of block size for ssl.cachesize
A user could eventually ask himself where those 200 bytes block size are
coming from. This patch tries to better explain the origin in case
people are curious or want to double check the reality.
Willy Tarreau [Fri, 12 Feb 2021 16:59:10 +0000 (17:59 +0100)]
MINOR: cfgparse: implement a simple if/elif/else/endif macro block handler
Very often, especially since reg-tests, it would be desirable to be able
to conditionally comment out a config block, such as removing an SSL
binding when SSL is disabled, or enabling HTX only for certain versions,
etc.
This patch introduces a very simple nested block management which takes
".if", ".elif", ".else" and ".endif" directives to take or ignore a block.
For now the conditions are limited to empty string or "0" for false versus
a non-nul integer for true, which already suffices to test environment
variables. Still, it needs to be a bit more advanced with defines, versions
etc.
A set of ".notice", ".warning" and ".alert" statements are provided to
emit messages, often in order to provide advice about how to fix certain
conditions.
Willy Tarreau [Fri, 12 Feb 2021 15:56:22 +0000 (16:56 +0100)]
MINOR: peers/cli: do not dump the peers dictionaries by default on "show peers"
The "show peers" output has become huge due to the dictionaries making it
less readable. Now this feature has reached a certain level of maturity
which doesn't warrant to dump it all the time, given that it was essentially
needed by developers. Let's make it optional, and disabled by default, only
when "show peers dict" is requested. The default output reminds about the
command. The output has been divided by 5 :
CLEANUP: server: Remove useless "filepath" variable in apply_server_state()
This variable is now only used to point on the local server-state file. When
the server-state is global, it is unused. So, we now use "localfilepath"
instead. Thus, the "filepath" variable can safely be removed.
BUG/MINOR: server: Don't call fopen() with server-state filepath set to NULL
When a local server-state file is loaded, if its name is too long, the error
is not properly handled, resulting to a call to fopen() with the "filepath"
variable set to NULL. To fix the bug, when this error occurs, we jump to the
next proxy, via a "continue" statement. And we take case to set "filepath"
variable after the error handling to be sure.
This patch should fix the issue #1111. It must be backported as far as 1.6.
CLEANUP: tcpcheck: Remove a useless test on port variable
When a connect rule is evaluated a test is performed on the "port" variable
while it is set to 0 just on the line just above. Just remove this useless
test to make ccpcheck happy.
Willy Tarreau [Fri, 12 Feb 2021 13:58:08 +0000 (14:58 +0100)]
MEDIUM: cfgparse: allow a proxy to designate the defaults section to use
Now it becomes possible to specify "from foo" on a frontend/listen/backend
or even on a "defaults" line, to mention that defaults section "foo" needs
to be used to preset the proxy's settings.
When not set, the last section remains used. In case the designated name
is found at multiple places, it is rejected and an error indicates two
occurrences of the same name. Similarly, if the section name is found,
its name must only use valid characters. This allows multiple named
defaults section to continue to coexist without the risk that they will
cause trouble by accident.
When it comes to "defaults" relying on another defaults, what happens is
just that a new defaults section is created from the designated one. This
will make it possible for example to reuse some settings such as log-format
like below:
A small corner case remains in the error detection, if a second defaults
section appears with the same name after the point where it was used, and
nobody references it, the duplicate will not be detected. This could be
addressed by performing the syntactic checks in check_config_validity(),
and by postponing the freeing of the defaults, after tagging a defaults
section as explicitly looked up by another section. This doesn't seem
that important at the moment though.
Willy Tarreau [Fri, 12 Feb 2021 13:08:31 +0000 (14:08 +0100)]
MEDIUM: proxy: store the default proxies in a tree by name
Now default proxies are stored into a dedicated tree, sorted by name.
Only unnamed entries are not kept upon new section creation. The very
first call to cfg_parse_listen() will automatically allocate a dummy
defaults section which corresponds to the previous static one, since
the code requires to have one at a few places.
The first immediately visible benefit is that it allows to reuse
alloc_new_proxy() to allocate a defaults section instead of doing it by
hand. And the secret goal is to allow to keep multiple named defaults
section in memory to reuse them from various proxies.
Willy Tarreau [Fri, 12 Feb 2021 12:52:11 +0000 (13:52 +0100)]
MINOR: proxy: support storing defaults sections into their own tree
Now we'll have a tree of named defaults sections. The regular insertion
and lookup functions take care of the capability in order to select the
appropriate tree. A new function proxy_destroy_defaults() removes a
proxy from this tree and frees it entirely.
Willy Tarreau [Fri, 12 Feb 2021 11:17:30 +0000 (12:17 +0100)]
MINOR: cfgparse: use a pointer to the current default proxy
In order to make the default proxy configurable, we'll need to have a
pointer to it which might differ from &defproxy. cfg_parse_listen()
now gets curr_defproxy for this.
Willy Tarreau [Fri, 12 Feb 2021 11:29:28 +0000 (12:29 +0100)]
MINOR: cfgparse: move defproxy to cfgparse-listen as a static
We don't want to expose this one anymore as we'll soon keep multiple
default proxies. Let's move it inside the parser which is the only
place which still uses it, and initialize it on the fly once needed
instead of doing it at boot time.
Willy Tarreau [Fri, 12 Feb 2021 11:15:05 +0000 (12:15 +0100)]
BUG/MINOR: server: parse_server() must take a const for the defproxy
The default proxy was passed as a variable, which in addition to being
a PITA to deal with in the config parser, doesn't feel safe to use when
it ought to be const.
This will only affect new code so no backport is needed.
Willy Tarreau [Fri, 12 Feb 2021 11:09:38 +0000 (12:09 +0100)]
BUG/MINOR: tcpcheck: proxy_parse_*check*() must take a const for the defproxy
The default proxy was passed as a variable, which in addition to being
a PITA to deal with in the config parser, doesn't feel safe to use when
it ought to be const.
This will only affect new code so no backport is needed.
Willy Tarreau [Fri, 12 Feb 2021 11:07:38 +0000 (12:07 +0100)]
BUG/MINOR: extcheck: proxy_parse_extcheck() must take a const for the defproxy
The default proxy was passed as a variable, which in addition to being
a PITA to deal with in the config parser, doesn't feel safe to use when
it ought to be const.
This will only affect new code so no backport is needed.
Willy Tarreau [Fri, 12 Feb 2021 09:48:53 +0000 (10:48 +0100)]
MINOR: proxy: always properly reset the just freed default instance pointers
In proxy_free_defaults(); none of the free() calls was followed by a
pointer reset. Not only it's hard to figure if one of them is duplicated,
but this code started to call other functions which might or might not
rely on such just freed pointers. Let's reset them as they should be to
make sure there will never be any case of use-after-free. The 3 functions
called there were inspected and are all unaffected by this so this remains
safe to do right now.
Willy Tarreau [Fri, 12 Feb 2021 09:38:49 +0000 (10:38 +0100)]
MINOR: proxy: move the defproxy freeing code to proxy.c
This used to be open-coded in cfgparse-listen.c when facing a "defaults"
keyword. Let's move this into proxy_free_defaults(). This code is ugly and
doesn't even reset the just freed pointers. Let's not change this yet.
This code should probably be merged with a generic proxy deinit function
called from deinit(). However there's a catch on uri_auth which cannot be
freed because it might be used by one or several proxies. We definitely
need refcounts there!
Willy Tarreau [Fri, 12 Feb 2021 08:15:16 +0000 (09:15 +0100)]
MEDIUM: proxy: only take defaults when a default proxy is passed.
The proxy initialization code relies on three phases, allocation,
pre-initialization, and assignments from defaults. This last part is
entirely taken from the defaults proxy when arguments are set. This
sensibly complexifies the initialization code as it requires to always
have a default proxy.
This patch instead first applies the original default settings on a
proxy, and then uses those from a default proxy only if one such is
used. This will allow to initialize a proxy out of any default proxy
while still using valid defaults. A careful inspection of the function
showed that only 4 fields used to be set regardless of the default
proxy, and those were moved to init_new_proxy() where they ought to
have been in the first place.
Willy Tarreau [Fri, 12 Feb 2021 07:49:47 +0000 (08:49 +0100)]
REORG: proxy: centralize the proxy allocation code into alloc_new_proxy()
This new function takes over the old open-coding that used to be done
for too long in cfg_parse_listen() and it now does everything at once
in a proxy-centric function. The function does all the job of allocating
the structure, initializing it, presetting its defaults from the default
proxy and checking for errors. The code was almost unchanged except for
defproxy being passed as a pointer, and the error message being passed
using memprintf().
This change will be needed to ease reuse of multiple default proxies,
or to create dynamic backends in a distant future.
Willy Tarreau [Fri, 12 Feb 2021 07:19:01 +0000 (08:19 +0100)]
REORG: move init_default_instance() to proxy.c and pass it the defproxy pointer
init_default_instance() was still left in cfgparse.c which is not the
best place to pre-initialize a proxy. Let's place it in proxy.c just
after init_new_proxy(), take this opportunity for renaming it to
proxy_preset_defaults() and taking out init_new_proxy() from it, and
let's pass it the pointer to the default proxy to be initialized instead
of implicitly assuming defproxy. We'll soon be able to exploit this.
Only two call places had to be updated.
Willy Tarreau [Fri, 12 Feb 2021 07:46:01 +0000 (08:46 +0100)]
BUILD: proxy: add missing compression-t.h to proxy-t.h
struct comp is used in struct proxy but never declared prior to this
so depending on where proxy.h is included, touching the <comp> field
can break the build.
Willy Tarreau [Fri, 12 Feb 2021 07:42:30 +0000 (08:42 +0100)]
BUG/MINOR: tcpheck: the source list must be a const in dup_tcpcheck_var()
This is just an API bug but it's annoying when trying to tidy the code.
The source list passed in argument must be a const and not a variable,
as it's typically the list head from a default proxy and must obviously
not be modified by the function. No backport is needed as it only impacts
new code.
Willy Tarreau [Fri, 12 Feb 2021 07:40:29 +0000 (08:40 +0100)]
BUG/MINOR: http-htx: defpx must be a const in proxy_dup_default_conf_errors()
This is just an API bug but it's annoying when trying to tidy the code.
The default proxy passed in argument must be a const and not a variable.
No backport is needed as it only impacts new code.
Willy Tarreau [Fri, 12 Feb 2021 12:28:22 +0000 (13:28 +0100)]
BUG/MINOR: cfgparse: do not mention "addr:port" as supported on proxy lines
The very old error message indicating that a proxy name is mandatory
still had a reference to the optional addr:port argument while this one
is explicitly rejected a few lines later since at least 1.9.
This is harmless but confusing. This can be backported to 2.0.
Willy Tarreau [Fri, 12 Feb 2021 10:49:25 +0000 (11:49 +0100)]
BUG/MINOR: stats: revert the change on ST_CONVDONE
In 2.1, commit ee4f5f83d ("MINOR: stats: get rid of the ST_CONVDONE flag")
introduced a subtle bug. By testing curproxy against defproxy in
check_config_validity(), it tried to eliminate the need for a flag
to indicate that stats authentication rules were already compiled,
but by doing so it left the issue opened for the case where a new
defaults section appears after the two proxies sharing the first
one:
defaults
mode http
stats auth foo:bar
listen l1
bind :8080
listen l2
bind :8181
defaults
# just to break above
This config results in:
[ALERT] 042/113725 (3121) : proxy 'f2': stats 'auth'/'realm' and 'http-request' can't be used at the same time.
[ALERT] 042/113725 (3121) : Fatal errors found in configuration.
Removing the last defaults remains OK. It turns out that the cleanups
that followed that patch render it useless, so the best fix is to revert
the change (with the up-to-date flags instead). The flag was marked as
belonging to the config. It's not exact but it's the closest to the
reality, as it's not there to configure the behavior but ti mention
that the config parser did its job.
This could be backported as far as 2.1, but in practice it looks like
nobody ever hit it.
Willy Tarreau [Fri, 12 Feb 2021 10:14:35 +0000 (11:14 +0100)]
BUG/MEDIUM: config: don't pick unset values from last defaults section
Since commit 1.3.14 with commit 1fa3126ec ("[MEDIUM] introduce separation
between contimeout, and tarpit + queue"), check_config_validity() looks
at the last defaults section to update all proxies' queue and tarpit
timeouts if they were not set!
This was apparently an attempt to properly set them on the fallback values,
except that the fallback values were taken from the default proxy before
looking at the current proxy itself. The worst part of it is that it might
have randomly worked by accident for some configurations when there was a
single defaults section, but has certainly caused too short queue
expirations once another defaults section was added later in the file with
these explicitly defined.
Let's remove the defproxy part and keep only the curproxy ones. This could
be backported everywhere, the bug has been there for 13 years.
Since the beginning, this directive is documented to accept an optional file
name. But it should also be possible to use it without any argument to use
the backend name as file name. However, when no argument is provided, an
error is reported during the configuration parsing requesting an argument, a
file name or "use-backend-name". And This last special argument is not
documented.
So, to respect the documentation and to avoid configuration breakages, all
modes are now supported. If this directive is called with no argument or
with "use-backend-name", the backend name is use as file name for the
server-state file. Otherwise, the provided string is used.
In addition, we take care to release any previously allocated file name in
case this directive is defines multiple times in the same backend. And an
error is reported if more than one argument are defined. Finally, the
documentation is updated accordingly. Sections supporting this directive are
also mentioned.
William Dauchy [Thu, 11 Feb 2021 21:51:27 +0000 (22:51 +0100)]
MINOR: server: enhance error precision when applying server state
server health checks and agent parameters are written the same way as
others to be able to enahcne code reuse: basically we make use of
parsing and assignment at the same place. It makes it difficult for
error handling to know whether srv object was modified partially or not.
The problem was already present with SRV resolution though.
I was a bit puzzled about the approach to take to be honest, and I did
not wanted to go into a full refactor, so I assumed it was ok to simply
notify whether the line was failed or partially applied.
William Dauchy [Thu, 11 Feb 2021 21:51:26 +0000 (22:51 +0100)]
MEDIUM: server: support {check,agent}_addr, agent_port in server state
logical followup from cli commands addition, so that the state server
file stays compatible with the changes made at runtime; use previously
added helper to load server attributes.
also alloc a specific chunk to avoid mixing with other called functions
using it
William Dauchy [Thu, 11 Feb 2021 21:51:25 +0000 (22:51 +0100)]
MEDIUM: server: add server-states version 2
Even if it is possibly too much work for the current usage, it makes
sure we don't break states file from v2.3 to v2.4; indeed, since v2.3,
we introduced two new fields, so we put them aside to guarantee we can
easily reload from a version 1.
The diff seems huge but there is no specific change apart from:
- introduce v2 where it is needed (parsing, update)
- move away from switch/case in update to be able to reuse code
- move srv lock to the whole function to make it easier
this patch confirm how painful it is to maintain this functionality.
William Dauchy [Thu, 11 Feb 2021 21:51:24 +0000 (22:51 +0100)]
MEDIUM: cli: add agent-port command
this patch allows to set agent port at runtime. In order to align with
both `addr` and `check-addr` commands, also add the possibility to
optionnaly set port on `agent-addr` command. This led to a small
refactor in order to use the same function for both `agent-addr` and
`agent-port` commands.
William Dauchy [Thu, 11 Feb 2021 21:51:23 +0000 (22:51 +0100)]
MEDIUM: cli: add check-addr command
this patch allows to set server health check address at runtime. In
order to align with `addr` command, also allow to set port optionnaly.
This led to a small refactor in order to use the same function for both
`check-addr` and `check-port` commands.
for `check-port`, we however don't permit the change anymore if checks
are not enabled on the server.
This command becomes more and more useful for people having a consul
like architecture:
- the backend server is located on a container with its own IP
- the health checks are done the consul instance located on the host
with the host IP
Amaury Denoyelle [Fri, 12 Feb 2021 14:22:43 +0000 (15:22 +0100)]
REGTESTS: fix sni used in http_reuse_conn_hash for libressl 3.3.0
libressl 3.3.0 is stricter on the sni field and fails if it contains
illegal characters such as the underscore. Replace sni field with proper
name to pass the test on the CI environment.
Amaury Denoyelle [Thu, 14 Jan 2021 09:15:29 +0000 (10:15 +0100)]
MINOR: connection: use proxy protocol as parameter for srv conn hash
Use the proxy protocol frame if proxy protocol is activated on the
server line. Do not add anymore these connections in the private list.
If some requests are made with the same proxy fields, they can reuse
the idle connection.
The reg-tests proxy_protocol_send_unique_id must be adapted has it
relied on the side effect behavior that every requests from a same
connection reused a private server connection. Now, a new connection is
created as expected if the proxy protocol fields differ.
Amaury Denoyelle [Thu, 11 Feb 2021 18:45:19 +0000 (19:45 +0100)]
MINOR: connection: use src addr as parameter for srv conn hash
The source address is used as an input to the the server connection hash. The
address and port are used as separate hash inputs. Do not add anymore these
connections in the private list.
This parameter is set only if used in the transparent-proxy mode.
Amaury Denoyelle [Thu, 11 Feb 2021 15:46:53 +0000 (16:46 +0100)]
MINOR: connection: use dst addr as parameter for srv conn hash
The destination address is used as an input to the server connection hash. The
address and port are used as separated hash inputs. Note that they are not used
when statically specified on the server line. This is only useful for dynamic
destination address.
This is typically used when the server address is dynamically set via the
set-dst action. The address and port are separated hash parameters.
Most notably, it should fixed set-dst use case (cf github issue #947).
Amaury Denoyelle [Mon, 11 Jan 2021 14:24:31 +0000 (15:24 +0100)]
MINOR: backend: rewrite alloc of stream target address
Change the API of the function used to allocate the stream target
address. This is done in order to be able to allocate the destination
address and use it to reuse a connection sharing with the same address.
In particular, the flag stream SF_ADDR_SET is now set outside of the
function.
MINOR: connection: use sni as parameter for srv conn hash
The sni parameter is an input to the server connection hash. Do not add
anymore connections with dynamic sni in the private list. Thus, it is
now possible to reuse a server connection if they use the same sni.
Amaury Denoyelle [Mon, 25 Jan 2021 09:29:35 +0000 (10:29 +0100)]
MINOR: backend: compare conn hash for session conn reuse
Compare the connection hash when reusing a connection from the session.
This ensures that a private connection is reused only if it shares the
same set of parameters.
Amaury Denoyelle [Fri, 22 Jan 2021 15:47:46 +0000 (16:47 +0100)]
MINOR: connection: use the srv pointer for the srv conn hash
The pointer of the target server is used as a first parameter for the
server connection hash calcul. This prevents the hash to be null when no
specific parameters are present, and can serve as a simple defense
against an attacker trying to reuse a non-conform connection.
Amaury Denoyelle [Mon, 18 Jan 2021 13:57:50 +0000 (14:57 +0100)]
MINOR: connection: prepare hash calcul for server conns
This is a preliminary work for the calcul of the backend connection
hash. A structure conn_hash_params is the input for the operation,
containing the various specific parameters of a connection.
The high bits of the hash will reflect the parameters present as input.
A set of macros is written to manipulate the connection hash and extract
the parameters/payload.
Amaury Denoyelle [Mon, 25 Jan 2021 13:43:17 +0000 (14:43 +0100)]
MINOR: backend: search conn in idle tree after safe on always reuse
With http-reuse always, if no matching safe connection is found, check
in idle tree for a matching one. This is needed because now idle
connections can be differentiated from each other.
If only the safe tree was checked because not empty, but did not contain
a matching connection, we could miss matching entry in idle tree.