Since the new master-worker model in 3.1, signals are registered in
step_init_3(). However, those signals were supposed to be registered
only for the worker or the standalone mode. It would call the wrong
callback in the master even during configuration parsing.
The patch set the signals handler to NULL for the master so it does
nothing until they really are registered.
BUG/MEDIUM: mworker: signals inconsistencies during startup and reload
Since haproxy 3.1, the master-worker mode changed to let the worker
parse the configuration instead of the master.
Previously, signals were blocked during configuration parsing and
unblocked before entering the polling loop of the master. This way it
was impossible to start a reload during the configuration parsing.
But with the new model, the polling loop is started in the master before
the configuration parsing is finished, and the signals are still
unblocked at this step. Meaning that it is possible to start a reload
while the configuration is parsing.
This patch reintroduce the behavior of blocking the signals during
configuration parsing adapted to the new model:
- Before the exec() of the reload, signals are blocked.
- When entering the polling loop, the SIGCHLD is unblocked because it is
required to get a failure during configuration parsing in the worker
- Once the configuration is parsed, upon success in _send_status() or
upon failure in run_master_in_recovery_mode() every signals are unblocked.
BUG/MEDIUM: quic-be: prevent use of MUX for 0-RTT sessions without secrets
The QUIC backend crashes when its peer does not support 0-RTT. In this case,
when the sessions are reused, no early-data level secrets are derived by
the TLS stack. This leads to crashes from qc_send_mux() which does not suppose
that both early-data level (qc->eel) and application level (qc->ael) cipher levels
could be non initialized.
To fix this:
- prevent qc_send_mux() to send data if these two encryption level are not
intialized. In this case it returns QUIC_TX_ERR_NONE;
- avoid waking up the MUX from XPRT ->start() callback if the MUX is ready
but without early-data level secrets to send them;
- ensure the MUX is woken up by qc_ssl_do_handshake() after handshake completion
if it is ready calling qc_notify_send()
Thank you to @InputOutputZ for having reported this issue in GH #3188.
No need to backport because QUIC backends is a current 3.3 development feature.
MEDIUM: mworker: set the mworker-max-reloads to 50
There was no mworker-max-reload value by default, it was set to INT_MAX
so this was impossible to reach.
The default value is now 50, which is still high, but no workers should
undergo that much reloads. Meaning that a worker will be killed with
SIGTERM if it reach this much reloads.
Amaury Denoyelle [Wed, 12 Nov 2025 10:36:09 +0000 (11:36 +0100)]
MINOR: quic: remove <ipv4> arg from qc_new_conn()
Remove <ipv4> argument from qc_new_conn(). This parameter is unnecessary
as it can be derived from the family type of the addresses also passed
as argument.
Amaury Denoyelle [Wed, 12 Nov 2025 10:35:25 +0000 (11:35 +0100)]
MINOR: quic: refactor qc_new_conn() prototype
The objective of this patch is to streamline qc_new_conn() usage so that
it is similar for frontend and backend sides.
Previously, several parameters were set only for frontend connections.
These arguments are replaced by a single quic_rx_packet argument, which
represents the INITIAL packet triggering the connection allocation on
the server side. For a QUIC client endpoint, it remains NULL. This usage
is consider more explicit.
As a minor change, <target> is moved as the first argument of the
function. This is considered useful as this argument determines whether
the connection is a frontend or backend entry.
Along with these changes, qc_new_conn() documentation has been reworded
so that it is now up-to-date with the newest usage.
Amaury Denoyelle [Wed, 12 Nov 2025 15:58:46 +0000 (16:58 +0100)]
MINOR: quic: try to clarify quic_conn CIDs fields direction
quic_conn has two fields named <dcid> and <scid>. It may cause confusion
as it is not obvious how these fields are related to the connection
direction. Try to improve this by extending the documentation of these
two fields.
MINOR: quic: support multiple random CID generation for BE side
When a new backend connection is instantiated, a CID is first randomly
generated. It will serve as the first DCID for incoming packets from the
server. Prior to this patch, if the generated CID caused a collision
with an other entries from another connection, an error is reported and
the connection cannot be allocated.
This patch improves this procedure by implementing retries when a
collision occurs. Now, at most three attemps will be performed before
giving up. This is the same procedure already performed for CIDs
instantiated after RETIRE_CONNECTION_ID frame parsing.
Along with this functional change, qc_new_conn() is refactored for
backend instantiation. The CID generation is extracted from it and the
value is passed as an argument. This is considered cleaner as the code
is more similar between frontend and backend sides.
Amaury Denoyelle [Mon, 10 Nov 2025 13:38:45 +0000 (14:38 +0100)]
MINOR: quic: do not use quic_newcid_from_hash64 on BE side
quic_newcid_from_hash64 is an external callback. If defined, it serves
as a CID method generation, as an alternative to the default random
implementation.
This mechanism was not correctly implemented on the backend side.
Indeed, <hash64> quic_conn member is only setted for frontend
connections. The simplest solution would be to properly define it also
for backend ones. However, quic_newcid_from_hash64 derivation is really
only useful for the frontend side for now. Thus, this patch disables
using it on the backend side in favor of the default random generator.
To implement this, quic_cid_generate() is splitted in two functions, for
both methods of CIDs generation. This is the responsibility of the
caller to select the proper method. On backend side, only random
implementation is now used.
MINOR: stick-tables: Rename stksess shards to use buckets
The shard keyword is already used by the peers and on the server lines. And
it is unrelated with the session keys distribution. So instead of talking
about shard for the session key hashing, we now use the term "bucket".
Willy Tarreau [Fri, 14 Nov 2025 18:22:46 +0000 (19:22 +0100)]
[RELEASE] Released version 3.3-dev13
Released version 3.3-dev13 with the following main changes :
- BUG/MEDIUM: config: for word expansion, empty or non-existing are the same
- BUG/MINOR: quic: close connection on CID alloc failure
- MINOR: quic: adjust CID conn tree alloc in qc_new_conn()
- MINOR: quic: split CID alloc/generation function
- BUG/MEDIUM: quic: handle collision on CID generation
- MINOR: quic: extend traces on CID allocation
- MEDIUM/OPTIM: quic: alloc quic_conn after CID collision check
- MINOR: stats-proxy: ensure future-proof FN_AGE manipulation in me_generate_field()
- BUG/MEDIUM: stats-file: fix shm-stats-file preload not working anymore
- BUG/MINOR: do not account backend connections into maxconn
- BUG/MEDIUM: init: 'devnullfd' not properly closed for master
- BUG/MINOR: acme: more explicit error when BIO_new_file()
- BUG/MEDIUM: quic-be: do not launch the connection migration process
- MINOR: quic-be: Parse the NEW_TOKEN frame
- MEDIUM: quic-be: Parse, store and reuse tokens provided by NEW_TOKEN
- MINOR: quic-be: helper functions to save/restore transport params (0-RTT)
- MINOR: quic-be: helper quic_reuse_srv_params() function to reuse server params (0-RTT)
- MINOR: quic-be: Save the backend 0-RTT parameters
- MEDIUM: quic-be: modify ssl_sock_srv_try_reuse_sess() to reuse backend sessions (0-RTT)
- MINOR: quic-be: allow the preparation of 0-RTT packets
- MINOR: quic-be: Send post handshake frames from list of frames (0-RTT)
- MEDIUM: quic-be: qc_send_mux() adaptation for 0-RTT
- MINOR: quic-be: discard the 0-RTT keys
- MEDIUM: quic-be: enable the use of 0-RTT
- MINOR: quic-be: validate the 0-RTT transport parameters
- MINOR: quic-be: do not create the mux after handshake completion (for 0-RTT)
- MINOR: quic-be: avoid a useless I/O callback wakeup for 0-RTT sessions
- BUG/MEDIUM: acme: move from mt_list to a rwlock + ebmbtree
- BUG/MINOR: acme: can't override the default resolver
- MINOR: ssl/sample: expose ssl_*c_curve for AWS-LC
- MINOR: check: delay MUX init when SSL ALPN is used
- MINOR: cfgdiag: adjust diag on servers
- BUG/MINOR: check: only try connection reuse for http-check rulesets
- BUG/MINOR: check: fix reuse-pool if MUX inherited from server
- MINOR: check: clarify check-reuse-pool interaction with reuse policy
- DOC: configuration: add missing ssllib_name_startswith()
- DOC: configuration: add missing openssl_version predicates
- MINOR: cfgcond: add "awslc_api_atleast" and "awslc_api_before"
- REGTESTS: ssl: activate ssl_curve_name.vtc for AWS-LC
- BUILD: ech: fix clang warnings
- BUG/MEDIUM: stick-tables: Always return the good stksess from stktable_set_entry
- BUG/MINOR: stick-tables: Fix return value for __stksess_kill()
- CLEANUP: stick-tables: Don't needlessly compute shard number in stksess_free()
- MINOR: h1: h1_release() should return if it destroyed the connection
- BUG/MEDIUM: h1: prevent a crash on HTTP/2 upgrade
- MINOR: check: use auto SNI for QUIC checks
- MINOR: check: ensure QUIC checks configuration coherency
- CLEANUP: peers: remove an unneeded null check
- Revert "BUG/MEDIUM: connections: permit to permanently remove an idle conn"
- BUG/MEDIUM: connection: do not reinsert a purgeable conn in idle list
- DEBUG: extend DEBUG_STRESS to ease testing and turn on extra checks
- DEBUG: add BUG_ON_STRESS(): a BUG_ON() implemented only when DEBUG_STRESS > 0
- DEBUG: servers: add a few checks for stress-testing idle conns
- BUG/MINOR: check: fix QUIC check test when QUIC disabled
- BUG/MINOR: quic-be: missing version negotiation
- CLEANUP: quic: Missing succesful SSL handshake backend trace (OpenSSL 3.5)
- BUG/MINOR: quic-be: backend SSL session reuse fix (OpenSSL 3.5)
- REGTEST: quic: quic/ssl_reuse.vtc supports OpenSSL 3.5 QUIC API
This bug impacts only the QUIC backends when haproxy is compiled against
OpenSSL 3.5 with QUIC API(HAVE_OPENSSL_QUIC).
The QUIC clients could not reuse their SSL session because the TLS tickets
received from the servers could not be provided to the TLS stack. This should
be done when the stack calls ha_quic_ossl_crypto_recv_rcd()
(OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_RECV_RCD callback).
According to OpenSSL team, an SSL_read() call must be done after the handshake
completion. It seems the correct location is at the same level as for
SSL_process_quic_post_handshake() for quictls.
Thank you to @mattcaswell, @Sashan and @vdukhovni for having helped in solving
this issue.
This bug impacts only the QUIC clients (or backends). The version negotiation
was not supported at all for them. This is an oversight.
Contrary to the QUIC server which choose the negotiated version after having
received the transport parameters (into ClientHello message) the client selects
the negotiated version from the first Initial packet version field. Indeed, the
server transport parameters are inside the ServerHello messages ciphered
into Handshake packets.
This non intrusive patch does not impact the QUIC server implementation.
It only selects the negotiated version from the first Initial packet
received from the server and consequently initializes the TLS cipher context.
Thank you to @InputOutputZ for having reporte this issue in GH #3178.
No need to backport because the QUIC backends support arrives with 3.3.
Willy Tarreau [Fri, 14 Nov 2025 16:27:53 +0000 (17:27 +0100)]
BUG/MINOR: check: fix QUIC check test when QUIC disabled
Latest commit ef206d441c ("MINOR: check: ensure QUIC checks configuration
coherency") introduced a regression when QUIC is not compiled in. Indeed,
not specifying a check proto sets mux_proto to NULL, which also happens to
be the value of get_mux_proto("QUIC"), so it complains about QUIC. Let's
add a non-null check in addition to this.
Willy Tarreau [Fri, 14 Nov 2025 15:48:16 +0000 (16:48 +0100)]
DEBUG: servers: add a few checks for stress-testing idle conns
The latest idle conns fix 9481cef948 ("BUG/MEDIUM: connection: do not
reinsert a purgeable conn in idle list") addresses a very hard-to-hit
case which manifests itself with an attempt to reuse a connection fails
because conn->mux is NULL:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000655410b8642c in conn_backend_get (reuse_mode=4, srv=srv@entry=0x6554378a7140,
sess=sess@entry=0x7cfe140948a0, is_safe=is_safe@entry=0,
hash=hash@entry=910818338996668161) at src/backend.c:1390
1390 if (conn->mux->takeover && conn->mux->takeover(conn, i, 0) == 0) {
However the condition that leads to this situation can be detected
earlier, by the presence of the connection in the toremove_list, whose
race window is much larger and easier to detect.
This patch adds a few BUG_ON_STRESS() at selected places that an detect
this condition. When built with -DDEBUG_STRESS and run under stress with
two distinct processes communicating over H2 over SSL, under a stress of
400-500k req/s, the front process usually crashes in the first 10-30s
triggering in _srv_add_idle() if the fix above is reverted (and it does
not crash with the fix).
This is mainly included to serve as an illustration of how to instrument
the code for seamless stress testing.
Willy Tarreau [Fri, 14 Nov 2025 15:25:27 +0000 (16:25 +0100)]
DEBUG: add BUG_ON_STRESS(): a BUG_ON() implemented only when DEBUG_STRESS > 0
The purpose of this new BUG_ON is beyond BUG_ON_HOT(). While BUG_ON_HOT()
is meant to be light but placed on very hot code paths, BUG_ON_STRESS()
might be heavy and only used under stress-testing, to try to detect early
that something bad is starting to happen. This one is not even type-checked
when not defined because we don't want to risk the compiler emitting the
slightest piece of code there in production mode, so as to give enough
freedom to the developers.
Willy Tarreau [Fri, 14 Nov 2025 14:58:06 +0000 (15:58 +0100)]
DEBUG: extend DEBUG_STRESS to ease testing and turn on extra checks
DEBUG_STRESS is currently used only to expose "stress-level". With this
patch, we go a bit further, by automatically forcing DEBUG_STRICT and
DEBUG_STRICT_ACTION to their highest values in order to enable all
BUG_ON levels, and make all of them result in a crash. In addition,
care is taken to always only have 0 or 1 in the macro, so that it can be
tested using "#if DEBUG_STRESS > 0" as well as "if (DEBUG_STRESS) { }"
everywhere.
The goal will be to ease insertion of extra tests for builds dedicated
to stress-testing that enable possibly expensive extra checks on certain
code paths that cannot reasonably be compiled in for production code
right now.
Amaury Denoyelle [Wed, 12 Nov 2025 16:44:36 +0000 (17:44 +0100)]
BUG/MEDIUM: connection: do not reinsert a purgeable conn in idle list
A recent patch was introduced to fix a rare race condition in idle
connection code which would result in a crash. The issue is when MUX IO
handler run on top of connection moved in the purgeable list. The
connection would be considered as present in the idle list instead, and
reinserted in it at the end of the handler while still in the purge
list.
This patch solves the described issue. However, it introduces another
bug as it may clear connection flag when removing a connection from its
parent list. However, these flags now serve primarily as a status which
indicate that the connection is accounted by the server. When a backend
connection is freed, server idle/used counters are decremented
accordingly to these flags. With the above patch, an incorrect counter
could be adjusted and thus wrapping would occured.
The first impact of this bug is that it may distort the estimated number
of connections needed by servers, which would result either in poor
reuse rate or too many idle connections kept. Another noticeable impact
is that it may prevent server deletion.
The main problem of the original and current issues is that connection
flags are misinterpreted as telling if a connection is present in the
idle list. As already described here, in fact these flags are solely a
status which indicate that the connection is accounted in server
counters. Thus, here are the definitive conclusion that can be learned
here :
* (conn->flags & CO_FL_LIST_MASK) == 1:
the connection is accounted by the server
it may or may not be present in the idle list
* (conn->flags & CO_FL_LIST_MASK) == 0
the connection is not accounted and not present in idle list
The discussion above does not mention session list, but a similar
pattern can be observed when CO_FL_SESS_IDLE flag is set.
To keep the original issue solved and fix the current one, IO MUX
handlers prologue are rewritten. Now, flags are not checked anymore for
list appartenance and LIST_INLIST macro is used instead. This is
definitely clearer with conn_in_list purpose here.
On IO MUX handlers end, conn idle flags may be checked if conn_in_list
was true, to reinsert the connection either in idle or safe list. This
is considered safe as no function should modify idle flags when a
connection is not stored in a list, except during conn_free() operation.
This patch must be backported to every stable versions after revert of
the above commit. It should be appliable up to 3.0 without any issue. On
2.8 and below, <idle_list> connection member does not exist. It should
be safe to check <leaf_p> tree node as a replacement.
Amaury Denoyelle [Wed, 12 Nov 2025 16:38:08 +0000 (17:38 +0100)]
Revert "BUG/MEDIUM: connections: permit to permanently remove an idle conn"
The target patch fixes a rare race condition which happen when a MUX IO
handler is working on a connection already moved into the purge list. In
this case, the handler will incorrectly moved back the connection into
the idle list.
To fix this, conn_delete_from_tree() was extended to remove flags along
with the connection from the idle list. This was performed when the
connection is moved into the purge list. However, it introduces another
issue related to the idle server connection accounting. Thus it is
necessary to revert it prior to the incoming newer fix.
This patch must be backported to every version where the original commit
is.
Willy Tarreau [Fri, 14 Nov 2025 12:46:00 +0000 (13:46 +0100)]
CLEANUP: peers: remove an unneeded null check
Coverity reported in GH #3181 that a NULL test was useless, in
peers_trace(), which is true since the peer always belongs to a
peers section and it was already dereferenced. Let's just remove
the test to avoid the confusion.
QUIC is now supported on the backend side, thus it is possible to use it
with server checks. However, checks configuration can be quite
extensive, differing greatly from the server settings.
This patch ensures that QUIC checks are always performed under a
controlled context. Objectives are to avoid any crashes and ensure that
there is no suprise for users in respect to the configuration.
The first part of this patch ensures that QUIC checks can only be
activated on QUIC servers. Indeed, QUIC requires dedicated
initialization steps prior to its usage.
The other part of this patch disables QUIC usage when one or multiple
specific check connection settings are specified in the configuration,
diverging from the server settings. This is the simplest solution for
now and ensure that there is no hidden behavior to users. This means
that it's currently impossible to perform QUIC checks if other endpoints
that the server itself. However for now there is no real use-case for
this scenario.
Along with these changes, check-proto documentation is updated to
clarify QUIC checks behavior.
Amaury Denoyelle [Thu, 30 Oct 2025 10:39:42 +0000 (11:39 +0100)]
MINOR: check: use auto SNI for QUIC checks
By default, check SNI is set to the Host header when an HTTPS check is
performed. This patch extends this mode so that it is also active when
QUIC checks are executed.
This patch should improve reuse rate with checks. Indeed, SNI is also
already automatically set for normal traffic. The same value must be
used during check so that a connection hash match can be found.
Olivier Houchard [Fri, 14 Nov 2025 11:42:38 +0000 (12:42 +0100)]
BUG/MEDIUM: h1: prevent a crash on HTTP/2 upgrade
Change h1_process() to return -2 when the mux is destroyed but the
connection is not, so that we can differentiate between "both mux and
connection were destroyed" and "only the mux was destroyed".
It can happen that only the mux gets destroyed, and the connection is
still alive, if we did upgrade it to HTTP/2.
In h1_wake(), if the connection is alive, then return 0, as the wake
methods should only return -1 if the connection is dead.
This fixes a bug where the ssl xprt would consider the connection
destroyed, and thus would consider its tasklet should die, and return
NULL, and its TASK_RUNNING flag would never be removed, leading to an
infinite loop later on. This would happen anytime an HTTP/2 upgrade was
successful.
This should be backported up to 2.8. While the bug by commit 00f43b7c8b136515653bcb2fc014b0832ec32d61, it was not triggered before
only by chance, and exists in previous releases too.
Olivier Houchard [Fri, 14 Nov 2025 11:33:33 +0000 (12:33 +0100)]
MINOR: h1: h1_release() should return if it destroyed the connection
h1_release() is called to destroy everything related to the mux h1,
usually even the connection. However, it handles upgrades to HTTP/2 too,
in which case the h1 mux will be destroyed, but the connection will
still be alive. So make it so it returns 0 if everything is destroyed,
and -1 if the connection is still alive.
This should be backported up to 2.8, as a future bugfix will depend on
it.
CLEANUP: stick-tables: Don't needlessly compute shard number in stksess_free()
Since commit 0bda33a3e ("MINOR: stick-tables: remove the uneeded read lock
in stksess_free()"), the lock on the shard is no longer acquired. So it is
useless to still compture the shard number. The result is never used and can
be safely removed.
BUG/MINOR: stick-tables: Fix return value for __stksess_kill()
The commit 9938fb9c7 ("BUG/MEDIUM: stick-tables: Fix race with peers when
killing a sticky session") introduced a regression.
__stksess_kill() must always return 0 if the session cannot be released. But
when the ref_cnt is tested under the update lock, a success is reported if
the session is still in-used. 0 must be returned in that case.
This bug is harmless because callers never use the return value of
__stksess_kill() or stksess_kill().
BUG/MEDIUM: stick-tables: Always return the good stksess from stktable_set_entry
In stktable_set_entry(), the return value of __stktable_store() is not
tested while it is possible to get an existing session with the same key
instead of the one we want to insert. It happens when we fails to upgrade
the read lock on the bucket to an write lock. In that case, we release the
lock for a short time to get a write lock.
So, to fix the bug, we must check the session returned by __stktable_store()
and take care to return this one.
The bug was introduced by the commit e62885237c ("MEDIUM: stick-table: make
stktable_set_entry() look up under a read lock"). It must be backported as
far as 2.8.
MINOR: cfgcond: add "awslc_api_atleast" and "awslc_api_before"
AWS-LC features are not easily tested with just the openssl version
constant. AWS-LC uses its own API versioning stored in the
AWSLC_API_VERSION constant.
This patch add the two awslc_api_atleast and awslc_api_before predicates
that help to check the AWS-LC API.
Amaury Denoyelle [Thu, 13 Nov 2025 16:45:18 +0000 (17:45 +0100)]
MINOR: check: clarify check-reuse-pool interaction with reuse policy
check-reuse-pool can only perform as expected if reuse policy on the
backend is set to aggressive or higher. Update the documentation to
reflect this and implement a server diag warning.
Amaury Denoyelle [Wed, 29 Oct 2025 17:25:55 +0000 (18:25 +0100)]
BUG/MINOR: check: fix reuse-pool if MUX inherited from server
Check reuse is only performed if no specific check connect options are
specified on the configuration. This ensures that reuse won't be
performed if intending to use different connection parameters from the
default traffic.
This relies on tcpcheck_use_nondefault_connect() which indicates if the
check has any specific connection parameters. One of them if check
<mux_proto> field. However, this field may be automatically set during
init_srv_check() in some specific conditions without any explicit
configuration, most notably when using http-check rulesets on an HTTP
backend. Thus, it prevents connection reuse for these checks.
This commit fixes this by adjuting tcpcheck_use_nondefault_connect().
Beside checking check <mux_proto> field, it also detects if it is
different from the server configuration. This is sufficient to know if
the value is derived from the configuration or automatically calculated
in init_srv_check().
Note that this patch introduces a small behavior change. Prior to it,
check reuse were never performed if "check-proto" is explicitely
configured. Now, check reuse will be performed if the configured value
is identical to the server MUX protocol. This is considered as
acceptable as connection reuse is safe when using a similar MUX
protocol.
Amaury Denoyelle [Wed, 29 Oct 2025 15:20:11 +0000 (16:20 +0100)]
BUG/MINOR: check: only try connection reuse for http-check rulesets
In 3.2, a new server keyword "check-reuse-pool" has been introduced. It
allows to reuse a connection for a new check, instead of always
initializing a new one. This is only performed if the check does not
rely on specific connection parameters differing from the server.
This patch further restricts reuse for checks only when an HTTP ruleset
is used at the backend level. Indeed, reusing a connection outside of
HTTP is an undefined behavior. The impact of this bug is unknown and
depends on the proxy/server configuration. In the case of an HTTP
backend with non-HTTP checks, check-reuse-pool would probably cause a
drop in reuse rate.
Along this change, implement a new diagnostic warning on servers to
report that check-reuse-pool cannot apply due to an incompatible check
type.
Amaury Denoyelle [Fri, 14 Nov 2025 08:40:07 +0000 (09:40 +0100)]
MINOR: cfgdiag: adjust diag on servers
Adjust code dealing with diagnostics performed on server. The objective
is to extract the check on duplicate cookies in a dedicated function
outside of the proxies/servers loop.
This does not have any noticeable impact. This patch is merely a code
improvment to implement easily new future diagnostics on servers.
Amaury Denoyelle [Thu, 30 Oct 2025 10:08:21 +0000 (11:08 +0100)]
MINOR: check: delay MUX init when SSL ALPN is used
When instantiating a new connection for check, its MUX may be
initialized early. This was not performed though if SSL ALPN negotiation
will be used, except if check MUX is already fixed.
However, this method of initialization is problematic when QUIC MUX is
used. Indeed, this multiplexer must only be instantiated after the above
application protocol is known, which is derived from the ALPN
negotiation. If this is not the case a crash will occur in qmux_init().
In fact, a similar problem was already encountered for normal traffic.
Thus, a change was performed in connect_server() : MUX early
initialization is now always skipped if SSL ALPN negotiation is active,
even if MUX is already fixed. This patch introduces a similar change for
checks.
Without this patch, it is not possible to perform check on QUIC servers
as expected. Indeed, when http-check ruleset is active a crash would
occur prior to it.
Damien Claisse [Thu, 13 Nov 2025 15:39:58 +0000 (15:39 +0000)]
MINOR: ssl/sample: expose ssl_*c_curve for AWS-LC
The underlying SSL_get_negotiated_group function has been backported
into AWS-LC [1], so expose the feature for users of this TLS stack
as well. Note that even though it was actually added in AWS-LC 1.56.0,
we require AWSLC_API_VERSION >= 35 which was released in AWS-LC 1.57.0,
because API version wasn't incremented after this change. As the delta
is one minor version (less than two weeks), I consider this acceptable
to avoid relying on a proxy constant like TLSEXT_nid_unknown which
might be removed at some point.
BUG/MINOR: acme: can't override the default resolver
httpclient_acme_init() was called in cfg_parse_acme() which is at
section parsing. httpclient_acme_init() also calls
httpclient_create_proxy() which could create a "default" resolvers
section if it doesn't exists.
If one tries to override the default resolvers section after an ACME
section, the resolvers section parsing will fail because the section was
already created by httpclient_create_proxy().
This patch fixes the issue by moving the initialization of the ACME
proxy to a pre_check callback, which is called just before
check_config_validity().
BUG/MEDIUM: acme: move from mt_list to a rwlock + ebmbtree
The current ACME scheduler suffers from problems due to the way the
tasks are stored:
- MT_LIST are not scalables when having a lot of ACME tasks and having
to look for a specific one.
- the acme_task pointer was stored in the ckch_store in order to not
passing through the whole list. But a ckch_store can be updated and
the pointer lost in the previous one.
- when a task fails, the ptr in the ckch_store was not removed because
we only work with a copy of the original ckch_store, it would need to
lock the ckchs_tree and remove this pointer.
This patch fixes the issues by removing the MT_LIST-based architecture,
and replacing it by a simple ebmbtree + rwlock design.
The pointer to the task is not stored anymore in the ckch_store, but
instead it is stored in the acme_tasks tree. Finding a task is done by
doing a lookup on this tree with a RDLOCK.
Instead of checking if store->acme_task is not NULL, a lookup is also
done.
This allow to remove the stuck "acme_task" pointer in the store, which
was preventing to restart an acme task when the previous failed for this
specific certificate.
MINOR: quic-be: avoid a useless I/O callback wakeup for 0-RTT sessions
For backends and 0-RTT sessions, this patch modifies the ->start() callback to
wake up the I/O callback only if the connection (and the mux) is not ready. Note that
connect_server() has been modified to call this xprt callback just after having
created the mux and installed the mux. Contrary to 1-RTT session, for 0-RTT sessions,
the connections are always ready before calling this ->start xprt callback.
MINOR: quic-be: do not create the mux after handshake completion (for 0-RTT)
This is required during connection with 0-RTT support, to prevent two mux creations.
Indeed, for 0-RTT sessions, the QUIC mux is already started very soon from
connect_server() (src/backend.c).
MINOR: quic-be: validate the 0-RTT transport parameters
During 0-RTT sessions, some server transport parameters are reused after having
been save from previous sessions. These parameters must not be reduced
when it resends them. The client must check this is the case when some early data
are accepted by the server. This is what is implemented by this patch.
Implement qc_early_tranport_params_validate() which checks the new server parameters
are not reduced.
Also implement qc_ssl_eary_data_accepted() which was not implemented for TLS
stack without 0-RTT support (for instance wolfssl). That said this function
was no more used. This is why the compilation against wolfssl could not fail.
This patch allows the use of 0-RTT feature on QUIC server lines with "allow-0rtt"
option. In fact 0-RTT is really enabled only if ssl_sock_srv_try_reuse_sess()
successfully manages to reuse the SSL session and the chosen application protocol
from previous connections.
Note that, at this time, 0-RTT works only with quictls and aws-lc as TLS stack.
(0-RTT does not work at all (even for QUIC frontends) with libressl).
MEDIUM: quic-be: qc_send_mux() adaptation for 0-RTT
When entering this function, a selection is done about the encryption level
to be used to send data. For a client, the early data encryption level
is used to send 0-RTT if this encryption level is initialized.
The Initial encryption is also registered to the send list for clients if there
is Initial crypto data to send. This allow Initial and 0-RTT packets to
be coalesced by datagrams.
MINOR: quic-be: Send post handshake frames from list of frames (0-RTT)
This patch is required to make 0-RTT work. It modifies the prototype of
quic_build_post_handshake_frames() to send post handshake frames from a
list of frames in place of the application encryption level (used
as <qc->ael> local variable).
This patch does not modify at all the current QUIC stack behavior (even for
QUIC frontends). It must be considered as a preparation for the code
to come about 0-RTT support for QUIC backends.
MEDIUM: quic-be: modify ssl_sock_srv_try_reuse_sess() to reuse backend sessions (0-RTT)
This function is called for both TCP and QUIC connections to reuse SSL sessions
saved by ssl_sess_new_srv_cb() callback called upon new SSL session creation.
In addition to this, a QUIC SSL session must reuse the ALPN and some specific QUIC
transport parameters. This is what is added by this patch for QUIC 0-RTT sessions.
Note that for now on, ssl_sock_srv_try_reuse_sess() may fail for QUIC connections
if it did not managed to reuse the ALPN. The caller must be informed of such an
issue. It must not enable 0-RTT for the current session in this case. This is
impossible without ALPN which is required to start a mux.
ssl_sock_srv_try_reuse_sess() is modified to always succeeds for TCP connections.
For both TCP and QUIC connections, this is ssl_sess_new_srv_cb() callback which
is called when a new SSL session is created. Its role is to save the session to
be reused for the next sessions.
This patch modifies this callback to save the QUIC parameters to be reused
for the next 0-RTT sessions (or during SSL session resumption).
The already existing path_params->nego_alpn member is used to store the ALPN as
this is done for TCP alongside path_params->tps new quic_early_transport_params
struct used to save the QUIC transport parameters to be reused for 0-RTT sessions.
MINOR: quic-be: helper quic_reuse_srv_params() function to reuse server params (0-RTT)
Implement quic_reuse_srv_params() whose role is to reuse the ALPN negotiated
during a first connection to a QUIC backend alongside its transport parameters.
MINOR: quic-be: helper functions to save/restore transport params (0-RTT)
Define quic_early_transport_params new struct for QUIC transport parameters
in relation with 0-RTT. This parameters must be saved during a first session to
be reused for 0-RTT next sessions.
qc_early_transport_params_cpy() copies the 0-RTT transport parameters to be
saved during a first connection to a backend. The copy is made from
a quic_transport_params struct to a quic_ealy_transport_params struct.
On the contrary, qc_early_transport_params_reuse() copies the transport parameters
to be reused for a 0-RTT session from a previous one. The copy is made
from a quic_early_transport_params strcut to a quic_transport_params struct.
Also add QUIC_EV_EARLY_TRANSP_PARAMS trace event to dump such 0-RTT
transport parameters from traces.
MEDIUM: quic-be: Parse, store and reuse tokens provided by NEW_TOKEN
Add a per thread ist struct to srv_per_thread struct to store the QUIC token to
be reused for subsequent sessions.
Parse at packet level (from qc_parse_ptk_frms()) these tokens and store
them calling qc_try_store_new_token() newly implemented function. This is
this new function which does its best (may fail) to update the tokens.
Modify qc_do_build_pkt() to resend these tokens calling quic_enc_token()
implemented by this patch.
Rename ->data qf_new_token struct field to ->w_data to distinguish it from
->r_data new field used to parse the NEW_TOKEN frame. Indeed to build the
NEW_TOKEN we need to write it to a static buffer into the frame struct. To
parse it we only need to store the address of the token field into the
RX buffer.
BUG/MEDIUM: quic-be: do not launch the connection migration process
At this time the connection migration is not supported by QUIC backends.
This patch prevents this process to be launched for connections to QUIC backends.
Furthermore, the connection migration process could be started systematically
when connecting a backend to INADDR_ANY, leading to crashes into qc_handle_conn_migration()
(when referencing qc->li).
Thank you to @InputOutputZ for having reported this issue in GH #3178.
This patch simply checks the connection type (listener or not) before checking if
a connection migration must be started.
No need to backport because support for QUIC backends is available from 3.3.
BUG/MINOR: acme: more explicit error when BIO_new_file()
Replace the error message of BIO_new_file() when the account-key cannot
be created on disk by "acme: cannot create the file '%s'". It was
previously "acme: out of memory." Which is unclear.
BUG/MEDIUM: init: 'devnullfd' not properly closed for master
Since commit "1ec59d3 MINOR: init: Make devnullfd global and create it
earlier in init" the devnullfd pointing towards /dev/null gets created
early in the init process but it was closed after the call to
"mworker_run_master". The master process never got to the FD closing
code and we had an FD leak.
Amaury Denoyelle [Mon, 10 Nov 2025 14:24:35 +0000 (15:24 +0100)]
BUG/MINOR: do not account backend connections into maxconn
Remove QUIC backend connections from global actconn accounting. Indeed,
this counter is only used on the frontend side. This is required to
ensure maxconn coherence.
BUG/MEDIUM: stats-file: fix shm-stats-file preload not working anymore
Due to recent commit 5c299dee ("MEDIUM: stats: consider that shared stats
pointers may be NULL") shm-stats-file preloading suddenly stopped working
In fact preloading should be considered as an initializing step so the
counters may be assigned there without checking for NULL first.
Indeed there are supposed to be NULL because preloading occurs before
counters_{fe,be}_shared_prepare() which takes care of setting the pointers
for counters if they weren't set before.
Obviously this corner-case was overlooked during 5c299dee writing and
testing. Thanks to Nick Ramirez for having reported the issue.
No backport needed, this issue is specific to 3.3.
MINOR: stats-proxy: ensure future-proof FN_AGE manipulation in me_generate_field()
Commit ad1bdc33 ("BUG/MAJOR: stats-file: fix crash on non-x86 platform
caused by unaligned cast") revealed an ambiguity in me_generate_field()
around FN_AGE manipulation. For now FN_AGE can only be stored as u32 or
s32, but in the future we could also support 64bit FN_AGES, and the
current code assumes 32bits types and performs and explicit unsigned int
cast. Instead we group current 32 bits operations for FF_U32 and FF_S32
formats, and let room for potential future formats for FN_AGE.
Commit ad1bdc33 also suggested that the fix was temporary and the approach
must change, but after a code review it turns out the current approach
(generic types manipulation under me_generate_field()) is legit. The
introduction of shm-stats-file feature didn't change the logic which
was initially implemented in 3.0. It only extended it and since shared
stats are now spread over thread-groups since 3.3, the use of atomic
operations made typecasting errors more visible, and structure mapping
change from d655ed5f14 ("BUG/MAJOR: stats-file: ensure
shm_stats_file_object struct mapping consistency (2nd attempt)") was in
fact the only change to blame for the crash on non-x86 platforms.
With ambiguities removed in me_generate_field(), let's hope we don't face
similar bugs in the future. Indeed, with generic counters, and more
specifically shared ones (which leverage atomic ops), great care must be
taken when changing their underlying types as me_generate_field() solely
relies on stat_col descriptor to know how to read the stat from a generic
pointer, so any breaking change must be reflected in that function as well
Amaury Denoyelle [Mon, 10 Nov 2025 10:02:45 +0000 (11:02 +0100)]
MEDIUM/OPTIM: quic: alloc quic_conn after CID collision check
On Initial packet parsing, a new quic_conn instance is allocated via
qc_new_conn(). Then a CID is allocated with its value derivated from
client ODCID. On CID tree insert, a collision can occur if another
thread was already parsing an Initial packet from the same client. In
this case, the connection is released and the packet will be requeued to
the other thread.
Originally, CID collision check was performed prior to quic_conn
allocation. This was changed by the commit below, as this could cause
issue on quic_conn alloc failure.
However, this procedure is less optimal. Indeed, qc_new_conn() performs
many steps, thus it could be better to skip it on Initial CID collision,
which can happen frequently. This patch restores the older order of
operations, with CID collision check prior to quic_conn allocation.
To ensure this does not cause again the same bug, the CID is removed in
case of quic_conn alloc failure. This should prevent any loop as it
ensures that a CID found in the global tree does not point to a NULL
quic_conn, unless if CID is attach to a foreign thread. When this thread
will parse a re-enqueued packet, either the quic_conn is already
allocated or the CID has been removed, triggering a fresh CID and
quic_conn allocation procedure.
BUG/MEDIUM: quic: handle collision on CID generation
CIDs are provided by haproxy so that the peer can use them as DCID of
its packets. Their value is set via a random generator. It happens on
several occasions during connection lifetime:
* via ODCID derivation if haproxy is the server
* on quic_conn init if haproxy is the client
* during post-handshake if haproxy is the server
* on RETIRE_CONNECTION_ID frame parsing
CIDs are stored in a global tree. On ODCID derivation, a check is
performed to ensure the CID is not a duplicate value. This is mandatory
to properly handle multiple INITIAL packets from the same client on
different thread.
However, for the other cases, no check is performed for CID collision.
As _quic_cid_insert() is silent, the issue is not detected at all. This
results in a CID advertized to the peer but not stored in the global
one. In the end, this may cause two issues. The first one is that
packets from the client which use the new CID will be rejected by
haproxy, most probably with a STATELESS_RESET. The second issue is that
it can cause a crash during quic_conn release. Indeed, the CID is stored
in the quic_conn local tree and thus eb_delete() for the global tree
will be performed. As <leaf_p> member is uninit, this results in a
segfault.
Note that this issue is pretty rare. It can only be observed if running
with a high number of concurrent connections in parallel, so that the
random generator will provide duplicate values. Patch is still labelled
as MEDIUM as this modifies code paths used frequently.
To fix this, _quic_cid_insert() unsafe function is completely removed.
Instead, quic_cid_insert() can be used, which reports an error code if a
collision happens. CID are then stored in the quic_conn tree only after
global tree insert success. Here is the solution for each steps if a
collision occurs :
* on init as client: the connection is completely released
* post-handshake: the CID is immediately released. The connection is
kept, but it will miss an extra CID.
* on RETIRE_CONNECTION_ID parsing: a loop is implemented to retry random
generation. It it fails several times, the connection is closed in
error.
A small convenience change is made to quic_cid_insert(). Output
parameter <new_tid> can now be NULL, which is useful as most of the
times caller do not care about it.
Split new_quic_cid() function into multiple ones. This patch should not
introduce any visible change. The objective is to render CID allocation
and generation more modular.
The first advantage of this patch is to bring code simplication. In
particular, conn CID sequence number increment and insertion into
connection tree is simpler than before. Another improvment is also that
errors could now be handled easier at each different steps of the CID
init.
This patch is a prerequisite for the fix on CID collision, thus it must
be backported prior to it to every affected version.
MINOR: quic: adjust CID conn tree alloc in qc_new_conn()
Change qc_new_conn() so that the connection CID tree is allocated
earlier in the function. This patch does not introduce a behavior
change. Its objective is to facilitate future evolutions on CIDs
handling.
This patch is a prerequisite for the fix on CID collision, thus it must
be backported prior to it to every affected version.
BUG/MINOR: quic: close connection on CID alloc failure
During RETIRE_CONNECTION_ID frame parsing, a new connection ID is
immediately reallocated after the release of the previous one. This is
done to ensure that the peer will never run out of DCID.
Prior to this patch, a CID allocation failure was be silently ignored.
This prevent the emission of a new CID, which could prevent the peer to
emit packets if it had no other CIDs available for use. Now, such error
is considered fatal to the connection. This is the safest solution as
it's better to close connections when memory is running low.
Willy Tarreau [Mon, 10 Nov 2025 10:55:33 +0000 (11:55 +0100)]
BUG/MEDIUM: config: for word expansion, empty or non-existing are the same
Amaury reported a case where "${FOO[*]}" still produces an empty field.
It happens if the variable is defined but does not contain any non-space
characters. The reason is that we special-case word expansion only on
non-existing vars. Let's change the ordering of operations so that word-
expanded vars always pretend the current arg is not an empty quote, so
that we don't make any difference between a non-existing var and an
empty one.
No backport is needed unless commit 1968731765 ("BUG/MEDIUM: config:
solve the empty argument problem again") is.
Willy Tarreau [Sat, 8 Nov 2025 11:12:00 +0000 (12:12 +0100)]
[RELEASE] Released version 3.3-dev12
Released version 3.3-dev12 with the following main changes :
- MINOR: quic: enable SSL on QUIC servers automatically
- MINOR: quic: reject conf with QUIC servers if not compiled
- OPTIM: quic: adjust automatic ALPN setting for QUIC servers
- MINOR: sample: optional AAD parameter support to aes_gcm_enc/dec
- REGTESTS: converters: check USE_OPENSSL in aes_gcm.vtc
- BUG/MINOR: resolvers: ensure fair round robin iteration
- BUG/MAJOR: stats-file: fix crash on non-x86 platform caused by unaligned cast
- OPTIM: backend: skip conn reuse for incompatible proxies
- SCRIPTS: build-ssl: allow to build a FIPS version without FIPS
- OPTIM: proxy: move atomically access fields out of the read-only ones
- SCRIPTS: build-ssl: fix rpath in AWS-LC install for openssl and bssl bin
- CI: github: update to macos-26
- BUG/MINOR: quic: fix crash on client handshake abort
- MINOR: quic: do not set conn member if ssl_sock_ctx
- MINOR: quic: remove connection arg from qc_new_conn()
- BUG/MEDIUM: server: Add a rwlock to path parameter
- BUG/MEDIUM: server: Also call srv_reset_path_parameters() on srv up
- BUG/MEDIUM: mux-h1: fix 414 / 431 status code reporting
- BUG/MEDIUM: mux-h2: make sure not to move a dead connection to idle
- BUG/MEDIUM: connections: permit to permanently remove an idle conn
- MEDIUM: cfgparse: deprecate 'master-worker' keyword alone
- MEDIUM: cfgparse: 'daemon' not compatible with -Ws
- DOC: configuration: deprecate the master-worker keyword
- MINOR: quic: remove <mux_state> field
- BUG/MEDIUM: stick-tables: Make sure we handle expiration on all tables
- MEDIUM: stick-tables: Optimize the expiration process a bit.
- MEDIUM: ssl/ckch: use ckch_store instead of ckch_data for ckch_conf_kws
- MINOR: acme: generate a temporary key pair
- MEDIUM: acme: generate a key pair when no file are available
- BUILD: ssl/ckch: wrong function name in ckch_conf_kws
- BUILD: acme: acme_gen_tmp_x509() signedness and unused variables
- BUG/MINOR: acme: fix initialization issue in acme_gen_tmp_x509()
- BUILD: ssl/ckch: fix ckch_conf_kws parsing without ACME
- MINOR: server: move the lock inside srv_add_idle()
- DOC: acme: crt-store allows you to start without a certificate
- BUG/MINOR: acme: allow 'key' when generating cert
- MINOR: stconn: Add counters to SC to know number of bytes received and sent
- MINOR: stream: Add samples to get number of bytes received or sent on each side
- MINOR: counters: Add req_in/req_out/res_in/res_out counters for fe/be/srv/li
- MINOR: stream: Remove bytes_in and bytes_out counters from stream
- MINOR: counters: Remove bytes_in and bytes_out counter from fe/be/srv/li
- MINOR: stats: Add stats about request and response bytes received and sent
- MINOR: applet: Add function to get amount of data in the output buffer
- MINOR: channel: Remove total field from channels
- DEBUG: stream: Add bytes_in/bytes_out value for both SC in session dump
- MEDIUM: stktables: Limit the number of stick counters to 100
- BUG/MINOR: config: Limit "tune.maxpollevents" parameter to 1000000
- BUG/MEDIUM: server: close a race around ready_srv when deleting a server
- BUG/MINOR: config: emit warning for empty args when *not* in discovery mode
- BUG/MEDIUM: config: solve the empty argument problem again
- MEDIUM: config: now reject configs with empty arguments
- MINOR: tools: add support for ist to the word fingerprinting functions
- MINOR: tools: add env_suggest() to suggest alternate variable names
- MINOR: tools: have parse_line's error pointer point to unknown variable names
- MINOR: cfgparse: try to suggest correct variable names on errors
- IMPORT: cebtree: Replace offset calculation with offsetof to avoid UB
- BUG/MINOR: acme: wrong dns-01 challenge in the log
- MEDIUM: backend: Defer conn_xprt_start() after mux creation
- MINOR: peers: Improve traces for peers
- MEDIUM: peers: No longer ack updates during a full resync
- MEDIUM: peers: Remove commitupdate field on stick-tables
- BUG/MEDIUM: peers: Fix update message parsing during a full resync
- MINOR: sample/stats: Add "bytes" in req_{in,out} and res_{in,out} names
- BUG/MEDIUM: stick-tables: Make sure updates are seen as local
- BUG/MEDIUM: proxy: use aligned allocations for struct proxy
- BUG/MEDIUM: proxy: use aligned allocations for struct proxy_per_tgroup
- BUG/MINOR: acme: avoid a possible crash on error paths
Willy Tarreau [Fri, 7 Nov 2025 21:27:25 +0000 (22:27 +0100)]
BUG/MINOR: acme: avoid a possible crash on error paths
In acme_EVP_PKEY_gen(), an error message is printed if *errmsg is set,
however, since commit 546c67d13 ("MINOR: acme: generate a temporary key
pair"), errmsg is passed as NULL in at least one occurrence, leading
the compiler to issue a NULL deref warning at -O3. And indeed, if the
errors are encountered, a crash will occur. No backport is needed.
Willy Tarreau [Fri, 7 Nov 2025 21:10:45 +0000 (22:10 +0100)]
BUG/MEDIUM: proxy: use aligned allocations for struct proxy_per_tgroup
In 3.2, commit f879b9a18 ("MINOR: proxies: Add a per-thread group field
to struct proxy") introduced struct proxy_per_tgroup that is declared as
thread_aligned, but is allocated using calloc(). Thus it is at risk of
crashing on machines using instructions requiring 64-byte alignment such
as AVX512. Let's use ha_aligned_zalloc_typed() instead of malloc().
For 3.2, we don't have aligned allocations, so instead the THREAD_ALIGNED()
will have to be removed from the struct definition. Alternately, we could
manually align it as is done for fdtab.
Willy Tarreau [Fri, 7 Nov 2025 21:05:21 +0000 (22:05 +0100)]
BUG/MEDIUM: proxy: use aligned allocations for struct proxy
Commit fd012b6c5 ("OPTIM: proxy: move atomically access fields out of
the read-only ones") caused the proxy struct to be 64-byte aligned,
which allows the compiler to use optimizations such as AVX512 to zero
certain fields. However the struct was allocated using calloc() so it
was not necessarily aligned, causing segv on startup on compatible
machines. Let's just use ha_aligned_zalloc_typed() to allocate the
struct.
BUG/MEDIUM: stick-tables: Make sure updates are seen as local
In stktable_touch_with_exp, if it is a local update, add it to the
pending update list even if it's already in the tree as a remote update,
otherwise it will never be communicated to other peers;
It used to work before 3.2 because of the ordering of operations, but
it's been broken by adding an extra step with the pending update list,
so we now have to explicitely check for that.
MINOR: sample/stats: Add "bytes" in req_{in,out} and res_{in,out} names
Number of bytes received or sent by a client or a server are now
saved. Sample fetches and stats fields to retrieve these informations are
renamed to add "bytes" in names to avoid any ambiguity with number of
requests and responses.
BUG/MEDIUM: peers: Fix update message parsing during a full resync
The commit 590c5ff2e ("MEDIUM: peers: No longer ack updates during a full
resync") introduced a regression. During a full resync, the ID of an update
message is not parsed at all. Thus, the parsing of the whole message in
desynchronized.
On full resync the update id itself is ignored, to not be acked, but it must
be parsed. It is now fixed.
MEDIUM: peers: Remove commitupdate field on stick-tables
This stick-table field was atomically updated with the last update id pushed
and dumped on the CLI but never used otherwise. And all peer sessions share
the same id because it is a stick-table info. So the info in peers dump is
pretty limited.
MEDIUM: peers: No longer ack updates during a full resync
ACK messages received by a peer sending updates during a full resync are
ignored. So, on the other side, there is no reason to still send these ACK
messages. Let's skip them.
In addition, the received updates during this stage are not considered as to
be acked. It is important to be sure to properly emit ACK messages once the
full sync finished.
Trace messages for peers were only protocol oriented and information
provided were quite light. With this patch, the traces were
improved. information about the peer, its applet and the section are
dumped. Several verbosities are now available and messages are dumped at
different levels depending on the context. It should easier to track issues
in the peers.
MEDIUM: backend: Defer conn_xprt_start() after mux creation
In connect_server(), defer the call to conn_xprt_start() until after we
had a chance to create the mux. The xprt can behave differently
depending on if a mux is or is not available at this point, as if it is,
it may want to wait until some data comes from the mux.
BUG/MINOR: acme: wrong dns-01 challenge in the log
Since 861fe532046 ("MINOR: acme: add the dns-01-record field to the
sink"), the dns-01 challenge is output in the dns_record trash, instead
of the global trash.
The send_log string was never updated with this change, and dumps some
data from the global trash instead. Since the last data emitted in the
trash seems to be the dns-01 token from the authorization object, it
looks like the response to the challenge.
Ben Kallus [Wed, 29 Oct 2025 12:38:51 +0000 (08:38 -0400)]
IMPORT: cebtree: Replace offset calculation with offsetof to avoid UB
This is the same as the equivalent fix in ebtree:
The C standard specifies that it's undefined behavior to dereference
NULL (even if you use & right after). The hand-rolled offsetof idiom
&(((s*)NULL)->f) is thus technically undefined. This clutters the
output of UBSan and is simple to fix: just use the real offsetof when
it's available.
Willy Tarreau [Tue, 24 Jun 2025 15:25:55 +0000 (17:25 +0200)]
MINOR: cfgparse: try to suggest correct variable names on errors
When an empty argument comes from the use of a non-existing variable,
we'll now detect the difference with an empty variable (error pointer
points to the variable's name instead), and submit it to env_suggest()
to see if another variable looks likely to be the right one or not.
This can be quite useful to quickly figure how to fix misspelled variable
names. Currently only series of letters, digits and underscores are
attempted to be resolved as a name. A typical example is:
peer "${HAPROXY_LOCAL_PEER}" 127.0.0.1:10000
which produces:
[ALERT] (24231) : config : parsing [bug-argv4.cfg:2]: argument number 1 at position 13 is empty and marks the end of the argument list:
peer "${HAPROXY_LOCAL_PEER}" 127.0.0.1:10000
^
[NOTICE] (24231) : config : Hint: maybe you meant HAPROXY_LOCALPEER instead ?
Willy Tarreau [Tue, 24 Jun 2025 15:20:33 +0000 (17:20 +0200)]
MINOR: tools: have parse_line's error pointer point to unknown variable names
When an argument is empty, parse_line() currently returns a pointer to
the empty string itself. This is convenient, but it's only actionable by
the user who will see for example "${HAPROXY_LOCALPEER}" and figure what
is wrong. Here we slightly change the reported pointer so that if an empty
argument results from the evaluation of an empty variable (meaning that
all variables in string are empty and no other char is present), then
instead of pointing to the opening quote, we'll return a pointer to the
first character of the variable's name. This will allow to make a
difference between an empty variable and an unknown variable, and for
the caller to take action based on this.
I.e. before we would get:
log "${LOG_SERVER_IP}" local0
^
if LOG_SERVER_IP is not set, and now instead we'll get this:
Willy Tarreau [Tue, 24 Jun 2025 15:18:52 +0000 (17:18 +0200)]
MINOR: tools: add env_suggest() to suggest alternate variable names
The purpose here is to look in the environment for a variable whose
name looks like the provided one. This will be used to try to auto-
correct misspelled environment variables that would silently be turned
to an empty string.
Willy Tarreau [Tue, 24 Jun 2025 15:14:47 +0000 (17:14 +0200)]
MINOR: tools: add support for ist to the word fingerprinting functions
The word fingerprinting functions are used to compare similar words to
suggest a correctly spelled one that looks like what the user proposed.
Currently the functions only support const char*, but there's no reason
for this, and it would be convenient to support substrings extracted
from random pieces of configurations. Here we're adding new variants
"_with_len" that take these ISTs and which are in fact a slight change
of the original ones that the old ones now rely on.
Willy Tarreau [Tue, 24 Jun 2025 06:24:28 +0000 (08:24 +0200)]
MEDIUM: config: now reject configs with empty arguments
As prepared during 3.2, we must error on empty arguments because they
mark the end of the line and cause subsequent arguments to be silently
ignored. It was too late in 3.2 to turn that into an error so it's a
warning, but for 3.3 it needed to be an alert.
This patch does that. It doesn't instantly break, instead it counts
one fatal error per violating line. This allows to emit several errors
at once, which can often be caused by the same variable being missed,
or a group of variables sharing a same misspelled prefix for example.
Tests show that it helps locate them better. It also explains what to
look for in the config manual for help with variables expansion.
Willy Tarreau [Tue, 24 Jun 2025 16:18:18 +0000 (18:18 +0200)]
BUG/MEDIUM: config: solve the empty argument problem again
This mostly reverts commit ff8db5a85 ("BUG/MINOR: config: Stopped parsing
upon unmatched environment variables").
As explained in commit #2367, finally the fix above was incorrect because
it causes other trouble such as this:
log "192.168.100.${NODE}" "local0"
being resolved to this:
log 192.168.100.local0
when NODE does not exist due to the loss of the spaces. In fact, while none
of us was well aware of this, when the user had:
server app 127.0.0.1:80 "${NO_CHECK}" weight 123
in fact they should have written it this way:
server app 127.0.0.1:80 "${NO_CHECK[*]}" weight 123
so that the variable is expanded to zero, one or multiple words, leaving
no empty arg (like in shell). This is supported since 2.3 with commit fa41cb6 so the right fix is in the config, let's revert the fix and
properly address the issue.
Some changes are necessary however, since after that patch, the in_arg
checks were added and are now inserting an empty argument even for
proper error reporting. For example, the following statement:
acl foo path "/a" "${FOO[*]}" "/b"
would complain about an empty arg at FOO due to in_arg=1, while dropping
this in_arg=1 with the following config:
acl foo path "/a" "${FOO}" "/b"
would silently stop after "/a" instead of complaining about an empty
field. So the approach here consists in noting whether or not something
was written since the quotes were emitted, in order to decide whether
or not to produce an argument. This way, "" continues to be an explicitly
empty arg, just like the same with an unknown variable, while "${FOO[*]}"
is allowed to prevent the creation of an argument if empty.
This should be backported to *some* versions, but the risk that some
configs were altered to rely on the broken fix is not null. At least
recent LTS should be reverted. Note that this requires previous commit:
BUG/MINOR: config: emit warning for empty args when *not* in discovery mode
otherwise this will break again configs relying on HAPROXY_LOCALPEER and
maybe a few other variables set at the end of discovery.
Willy Tarreau [Thu, 6 Nov 2025 16:57:37 +0000 (17:57 +0100)]
BUG/MINOR: config: emit warning for empty args when *not* in discovery mode
This actually reverses the condition of commit 5f1fad1690 ("BUG/MINOR:
config: emit warning for empty args only in discovery mode"). Indeed,
some variables are not known in discovery mode (e.g. HAPROXY_LOCALPEER),
and statements like:
peer "${HAPROXY_LOCALPEER}" 127.0.0.1:10000
are broken during discovery mode. It turns out that the warning is
currently hidden by commit ff8db5a85d ("BUG/MINOR: config: Stopped
parsing upon unmatched environment variables") since it silently drops
empty args which is sufficient to hide the warning, but it also breaks
other configs and needs to be reverted, which will break configs like
above again.
In issue #2995 we were not fully decided about discovery mode or not,
and already suspected some possible issues without being able to guess
which ones. The only downside of not displaying them in discovery mode
is that certain empty fields on the rare keywords specific to master
mode might remain silent until used. Let's just flip the condition to
check for empty args in normal mode only.
This should be backported to 3.2 after some time of observation.
Willy Tarreau [Thu, 6 Nov 2025 18:52:10 +0000 (19:52 +0100)]
BUG/MEDIUM: server: close a race around ready_srv when deleting a server
When a server is being disabled or deleted, in case it matches the
backend's ready_srv, this one is reset. However it's currently done in
a non-atomic way when the server goes down, and that could occasionally
reset the entry matching another server, but more importantly if in
parallel some requests are dequeued for that server, it may re-appear
there after having been removed, leading to a possible crash once it
is fully removed, as shown in issue #3177.
Let's make sure we reset the pointer when detaching the server from
the proxy, and use a CAS in both cases to only reset this server.
This fix needs to be backported to 3.2. There, srv_detach() is in
server.c instead of server.h. Thanks to Basha Mougamadou for the
detailed report and the useful backtraces.
BUG/MINOR: config: Limit "tune.maxpollevents" parameter to 1000000
"tune.maxpollevents" global parameter was not limited. It was possible to
set any integer value. But this value is used to allocate the array of
events used by epoll. With a huge value, it seems the allocation silently
fail, making haproxy totally unresponsive.
So let's to limit its value to 1 million. It is pretty high and it should
not be an issue to forbid greater values. The documentation was updated
accordingly.
This patch could be backported to all stable branches.
MEDIUM: stktables: Limit the number of stick counters to 100
"tune.stick-counters" global parameter was accepting any positive integer
value. But the maximum value is incredibly high. Setting a huge value has
signitifcant impact on memory and CPU usage. To avoid any issue, this value
is now limited to 100. It should be greater enough to all usage.
MINOR: applet: Add function to get amount of data in the output buffer
The helper function applet_output_data() returns the amount of data in the
output buffer of an applet. For applets using the new API, it is based on
data present in the outbuf buffer. For legacy applets, it is based on input
data present in the input channel's buffer. The HTX version,
applet_htx_output_data(), is also available
MINOR: stats: Add stats about request and response bytes received and sent
In previous patches, these counters were added per frontend, backend, server
and listener. With this patch, these counters are reported on stats,
including promex.
Note that the stats file minor version was incremented by one because the
shm_stats_file_object struct size has changed.
MINOR: counters: Remove bytes_in and bytes_out counter from fe/be/srv/li
bytes_in and bytes_out counters per frontend, backend, listener and server
were removed and we now rely on, respectively on, req_in and res_in
counters.
MINOR: stream: Remove bytes_in and bytes_out counters from stream
per-stream bytes_in and bytes_out counters was removed and replaced by
req.in and res.in. Coorresponding samples still exists but replies on new
counters.
MINOR: counters: Add req_in/req_out/res_in/res_out counters for fe/be/srv/li
Thanks to the previous patch, and based on info available on the stream, it
is now possible to have counters for frontends, backends, servers and
listeners to report number of bytes received and sent on both sides.
projects / thirdparty / haproxy.git / log
