BUILD: mworker: fix ignoring return value of ‘read’
Fix read return value unused result.
src/haproxy.c: In function ‘main’:
src/haproxy.c:3630:17: error: ignoring return value of ‘read’ declared with attribute ‘warn_unused_result’ [-Werror=unused-result]
3630 | read(sock_pair[1], &c, 1);
| ^~~~~~~~~~~~~~~~~~~~~~~~~
BUG/MEDIUM: mworker: fix startup and reload on macOS
Since the mworker rework in haproxy 3.1, the worker need to tell the
master that it is ready. This is done using the sockpair protocol by
sending a _send_status message to the master.
It seems that the sockpair protocol is buggy on macOS because of a known
issue around fd transfer documented in sendmsg(2):
Because sendmsg() does not necessarily block until the data has been
transferred, it is possible to transfer an open file descriptor across
an AF_UNIX domain socket (see recv(2)), then close() it before it has
actually been sent, the result being that the receiver gets a closed
file descriptor. It is left to the application to implement an
acknowledgment mechanism to prevent this from happening.
Indeed the recv side of the sockpair is closed on the send side just
after the send_fd_uxst(), which does not implement an acknowledgment
mechanism. So the master might never recv the _send_status message.
In order to implement an acknowledgment mechanism, a blocking read() is
done before closing the recv fd on the sending side, so we are sure that
the message was read on the other side.
This was only reproduced on macOS, meaning the master CLI is also
impacted on macOS. But no solution was found on macOS for it.
Implementing an acknowledgment mechanism would complexify too much the
protocol in non-blocking mode.
The problem was reported in ticket #3045, reproduced and analyzed by
@cognet.
BUG/MINOR: acl: set arg_list->kw to aclkw->kw string literal if aclkw is found
During configuration parsing *args can contain different addresses, it is
changing from line to line. smp_resolve_args() is called after the
configuration parsing, it uses arg_list->kw to create an error message, if a
userlist referenced in some ACL is absent. This leads to wrong keyword names
reported in such message or some garbage is printed.
It does not happen in the case of sample fetches. In this case arg_list->kw is
assigned to a string literal from the sample_fetch struct returned by
find_sample_fetch(). Let's do the same in parse_acl_expr(), when find_acl_kw()
lookup returns a corresponding acl_keyword structure.
This fixes the issue #3088 at GitHub.
This should be backported in all stable versions since 2.6 including 2.6.
BUG/MINOR: mux-quic: trace with non initialized qcc
This issue leads to crashes when the QUIC mux traces are enabled and could be
reproduced with -dMfail. When the qcc allocation fails (qcc_init()) haproxy
crashes into qmux_dump_qcc_info() because ->conn qcc member is initialized:
Program terminated with signal SIGSEGV, Segmentation fault.
at src/qmux_trace.c:146
146 const struct quic_conn *qc = qcc->conn->handle.qc;
[Current thread is 1 (LWP 1448960)]
(gdb) p qcc
$1 = (const struct qcc *) 0x7f9c63719fa0
(gdb) p qcc->conn
$2 = (struct connection *) 0x155550508
(gdb)
This patch simply fixes the TRACE() call concerned to avoid <qcc> object
dereferencing when it is NULL.
MINOR: quic: remove ->offset qf_crypto struct field
This patch follows this previous bug fix:
BUG/MINOR: quic: reorder fragmented RX CRYPTO frames by their offsets
where a ebtree node has been added to qf_crypto struct. It has the same
meaning and type as ->offset_node.key field with ->offset_node an eb64tree node.
This patch simply removes ->offset which is no more useful.
This patch should be easily backported as far as 2.6 as the one mentioned above
to ease any further backport to come.
MINOR: ssl: diagnostic warning when both 'default-crt' and 'strict-sni' are used
It possible to use both 'strict-sni' and 'default-crt' on the same bind
line, which does not make much sense.
This patch implements a check which will look for default certificates
in the sni_w tree when strict-sni is used. (Referenced by their empty
sni ""). default-crt sets the CKCH_INST_EXPL_DEFAULT flag in
ckch_inst->is_default, so its possible to differenciate explicits
default from implicit default.
BUG/MINOR: quic: reorder fragmented RX CRYPTO frames by their offsets
This issue impacts the QUIC listeners. It is the same as the one fixed by this
commit:
BUG/MINOR: quic: repeat packet parsing to deal with fragmented CRYPTO
As chrome, ngtcp2 client decided to fragment its CRYPTO frames but in a much
more agressive way. This could be fixed with a list local to qc_parse_pkt_frms()
to please chrome thanks to the commit above. But this is not sufficient for
ngtcp2 which often splits its ClientHello message into more than 10 fragments
with very small ones. This leads the packet parser to interrupt the CRYPTO frames
parsing due to the ncbuf gap size limit.
To fix this, this patch approximatively proceeds the same way but with an
ebtree to reorder the CRYPTO by their offsets. These frames are directly
inserted into a local ebtree. Then this ebtree is reused to provide the
reordered CRYPTO data to the underlying ncbuf (non contiguous buffer). This way
there are very few less chances for the ncbufs used to store CRYPTO data
to reach a too much fragmented state.
BUG/MEDIUM: quic-be: avoid crashes when releasing Initial pktns
This bug arrived with this fix:
BUG/MINOR: quic-be: missing Initial packet number space discarding
leading to crashes when dereferencing ->ipktns.
Such crashes could be reproduced with -dMfail option. To reach them, the
memory allocations must fail. So, this is relatively rare, except on systems
with limited memory.
To fix this, do not call quic_pktns_discard() if ->ipktns is NULL.
MINOR: proxy: handle shared listener counters preparation from proxy_postcheck()
We used to allocate and prepare listener counters from
check_config_validity() all at once. But it isn't correct, since at that
time listeners's guid are not inserted yet, thus
counters_fe_shared_prepare() cannot work correctly, and so does
shm_stats_file_preload() which is meant to be called even earlier.
Thus in this commit (and to prepare for upcoming shm shared counters
preloading patches), we handle the shared listener counters prep in
proxy_postcheck(), which means that between the allocation and the
prep there is the proper window for listener's guid insertion and shm
counters preloading.
No change of behavior expected when shm shared counters are not
actually used.
MEDIUM: server: split srv_init() in srv_preinit() + srv_postinit()
We actually need more granularity to split srv postparsing init tasks:
Some of them are required to be run BEFORE the config is checked, and
some of them AFTER the config is checked.
Thus we push the logic from 368d0136 ("MEDIUM: server: add and use
srv_init() function") a little bit further and split the function
in two distinct ones, one of them executed under check_config_validity()
and the other one using REGISTER_POST_SERVER_CHECK() hook.
SRV_F_CHECKED flag was removed because it is no longer needed,
srv_preinit() is only called once, and so is srv_postinit().
MINOR: haproxy: abort config parsing on fatal errors for post parsing hooks
When pre-check and post-check postparsing hooks= are evaluated in
step_init_2() potential fatal errors are ignored during the iteration
and are only taken into account at the end of the loop. This is not ideal
because some errors (ie: memory errors) could cause multiple alert
messages in a row, which could make troubleshooting harder for the user.
Let's stop as soon as a fatal error is encountered for post parsing
hooks, as we use to do everywhere else.
BUG/MEDIUM: spoe: Improve error detection in SPOE applet on client abort
It is possible to interrupt a SPOE applet without reporting an error. For
instance, when the client of the parent stream aborts. Thanks to this patch,
we take care to report an error on the SPOE applet to be sure to interrupt
the processing. It is especially important if the connection to the agent is
queued. Thanks to 886a248be ("BUG/MEDIUM: mux-spop: Reject connection
attempts from a non-spop frontend"), it is no longer an issue. But there is
no reason to continue to process if the parent stream is gone.
In addition, in the SPOE filter, if the processing is interrupted when the
filter is destroyed, no specific status code was set. It is not a big deal
because it cannot be logged at this stage. But it can be used to notify the SPOE
applet. So better to set it.
BUG/MEDIUM: mux-spop: Reject connection attempts from a non-spop frontend
It is possible to crash the process by initializing a connection to a SPOP
server from a non-spop frontend. It is of course unexpected and invalid. And
there are some checks to prevent that when the configuration is
loaded. However, it is not possible to handle all cases, especially the
"use_backend" rules relying on log-format strings.
It could be good to improve the backend selection by checking the mode
compatibility (for now, it is only performed for the HTTP).
But at the end, this can also be handled by the SPOP multiplexer when it is
initialized. If the opposite SD is not attached to an SPOE agent, we should
fail the mux initialization and return an internal error.
MEDIUM: applet: Set .rcv_buf and .snd_buf functions on default ones if not set
Based on the applet flags, it is possible to set .rcv_buf and .snd_buf
callback functions if necessary. If these functions are not defined for an
applet using the new API, it means the default functions must be used.
We also take care to choose the raw version or the htx version, depending on
the applet flags.
MINOR: applet: Make some applet functions HTX aware
applet_output_room() and applet_input_data() are now HTX aware. These
functions automatically rely on htx versions if APPLET_FL_HTX flag is set
for the applet.
MINOR: applet: Add a flag to know an applet is using HTX buffers
Multiplexers already explicitly announce their HTX support. Now it is
possible to set flags on applet, it could be handy to do the same. So, now,
HTX aware applets must set the APPLET_FL_HTX flag.
MINOR: applet: Add function to test applet flags from the appctx
appctx_app_test() function can now be used to test the applet flags using an
appctx. This simplify a bit tests on applet flags. For now, this function is
used to test APPLET_FL_NEW_API flag.
MINOR: applet: Rely on applet flag to detect the new api
Instead of setting a flag on the applet context by checking the defined
callback functions of the applet to know if an applet is using the new API
or not, we can now rely on the applet flags itself. By checking
APPLET_FL_NEW_API flag, it does the job. APPCTX_FL_INOUT_BUFS flag is thus
removed.
BUG/MEDIUM: http_ana: handle yield for "stats http-request" evaluation
stats http-request rules evaluation is handled separately in
http_process_req_common(). Because of that, if a rule requires yielding,
the evaluation is interrupted as (F)YIELD verdict return values are not
handled there.
Since 3.2 with the introduction of costly ruleset interruption in 0846638 ("MEDIUM: stream: interrupt costly rulesets after too many
evaluations"), the issue started being more visible because stats
http-request rules would be interrupted when the evaluation counters
reached tune.max-rules-at-once, but the evaluation would never be
resumed, and the request would continue to be handled as if the
evaluation was complete. Note however that the issue already existed
in the past for actions that could return ACT_RET_YIELD such as
"pause" for instance.
This issue was reported by GH user @Wahnes in #3087, thanks to him for
providing useful repro and details.
To fix the issue, we merge rule vedict handling in
http_process_req_common() so that "stats http-request" evaluation benefits
from all return values already supported for the current ruleset.
It should be backported in 3.2 with 0846638 ("MEDIUM: stream: interrupt
costly rulesets after too many evaluations"), and probably even further
(all stable versions) if the patch adaptation is not to complex (before
HTTP_RULE_RES_FYIELD was introduced) because it is still relevant.
MINOR: http_ana: fix typo in http_res_get_intercept_rule
HTTP_RULE_RES_YIELD was used where HTTP_RULE_RES_FYIELD should be used.
Hopefully, aside from debug traces, both return values were treated
equally. Let's fix that to prevent confusion and from causing bugs
in the future.
It may be backported in 3.2 with 0846638 ("MEDIUM: stream: interrupt
costly rulesets after too many evaluations") if it easily applies
Amaury Denoyelle [Tue, 12 Aug 2025 08:40:06 +0000 (10:40 +0200)]
MINOR: quic: centralize padding for HP sampling on packet building
The below patch has simplified INITIAL padding on emission. Now,
qc_prep_pkts() is responsible to activate padding for this case, and
there is no more special case in qc_do_build_pkt() needed.
However, qc_do_build_pkt() may still activate padding on its own, to
ensure that a packet is big enough so that header protection decryption
can be performed by the peer. HP decryption is performed by extracting a
sample from the ciphered packet, starting 4 bytes after PN offset.
Sample length is 16 bytes as defined by TLS algos used by QUIC. Thus, a
QUIC sender must ensures that length of packet number plus payload
fields to be at least 4 bytes long. This is enough given that each
packet is completed by a 16 bytes AEAD tag which can be part of the HP
sample.
This patch simplifies qc_do_build_pkt() by centralizing padding for this
case in a single location. This is performed at the end of the function
after payload is completed. The code is thus simpler.
This is not a bug. However, it may be interesting to backport this patch
up to 2.6, as qc_do_build_pkt() is a tedious function, in particular
when dealing with padding generation, thus it may benefit greatly from
simplification.
Amaury Denoyelle [Mon, 11 Aug 2025 16:04:59 +0000 (18:04 +0200)]
BUG/MINOR: quic: don't coalesce probing and ACK packet of same type
Haproxy QUIC stack suffers from a limitation : it's not possible to emit
a packet which contains probing data and a ACK frame in it. Thus, in
case qc_do_build_pkt() is invoked which both values as true, probing has
the priority and ACK is ignored.
However, this has the undesired side-effect of possibly generating two
coalesced packets of the same type in the same datagram : the first one
with the probing data and the second with an ACK frame. This is caused
by qc_prep_pkts() loop which may call qc_do_build_pkt() multiple times
with the same QEL instance. This case is normally use when a full
datagram has been built but there is still content to emit on the
current encryption level.
To fix this, alter qc_prep_pkts() loop : if both probing and ACK is
requested, force the datagram to be written after packet encoding. This
will result in a datagram containing the packet with probing data as
final entry. A new datagram is started for the next packet which will
can contain the ACK frame.
This also has some impact on INITIAL padding. Indeed, if packet must be
the last due to probing emission, qc_prep_pkts() will also activate
padding to ensure final datagram is at least 1.200 bytes long.
Note that coalescing two packets of the same type is not invalid
according to QUIC RFC. However it could cause issue with some shaky
implementations, so it is considered as a bug.
Amaury Denoyelle [Mon, 11 Aug 2025 15:54:39 +0000 (17:54 +0200)]
BUG/MAJOR: quic: fix INITIAL padding with probing packet only
A QUIC datagram that contains an INITIAL packet must be padded to 1.200
bytes to prevent any deadlock due to anti-amplification protection. This
is implemented by encoding a PADDING frame on the last packet of the
datagram if necessary.
Previously, qc_prep_pkts() was responsible to activate padding when
calling qc_do_build_pkt(), as it knows which packet is the last to
encode. However, this has the side-effect of preventing PING emission
for probing with no data as this case was handled in an else-if branch
after padding. This was fixed by the below commit
Above logic was altered to fix the PING case : padding was set to false
explicitely in qc_prep_pkts(). Padding was then added in a specific
block dedicated to the PING case in qc_do_build_pkt() itself for INITIAL
packets.
However, the fix is incorrect if the last QEL used to built a packet is
not the initial one and probing is used with PING frame only. In this
case, specific block in qc_do_build_pkt() does not add padding. This
causes a BUG_ON() crash in qc_txb_store() which catches these packets as
irregularly formed.
To fix this while also properly handling PING emission, revert to the
original padding logic : qc_prep_pkts() is responsible to activate
INITIAL padding. To not interfere with PING emission, qc_do_build_pkt()
body is adjusted so that PING block is moved up in the function and
detached from the padding condition.
The main benefit from this patch is that INITIAL padding decision in
qc_prep_pkts() is clearer now.
Note that padding can also be activated by qc_do_build_pkt(), as packets
should be big enough for header protection decipher. However, this case
is different from INITIAL padding, so it is not covered by this patch.
Amaury Denoyelle [Tue, 12 Aug 2025 09:30:03 +0000 (11:30 +0200)]
BUG/MINOR: quic: do not emit probe data if CONNECTION_CLOSE requested
If connection closing is activated, qc_prep_pkts() can only built a
datagram with a single packet. This is because we consider that only a
single CONNECTION_CLOSE frame is relevant at this stage.
This is handled both by qc_prep_pkts() which ensure that only a single
packet datagram is built and also qc_do_build_pkt() which prevents the
invokation of qc_build_frms() if <cc> is set.
However, there is an incoherency for probing. First, qc_prep_pkts()
deactivates it if connection closing is requested. But qc_do_build_pkt()
may still emit probing frame as it does not check its <probe> argument
but rather <pto_probe> QEL field directly. This can results in a packet
mixing a PING and a CONNECTION close frames, which is useless.
Fix this by adjusting qc_do_build_pkt() : closing argument is also
checked on PING probing emission. Note that there is still shaky code
here as qc_do_build_pkt() should rely only on <probe> argument to ensure
this.
Amaury Denoyelle [Tue, 12 Aug 2025 15:27:03 +0000 (17:27 +0200)]
BUG/MEDIUM: quic: reset padding when building GSO datagrams
qc_prep_pkts() encodes input data into QUIC packets in a loop into one
or several datagrams. It supports GSO which requires to built a serie of
multiple datagrams of the same length.
Each packet encoding is performed via a call to qc_do_build_pkt(). This
function has an argument to specify if output packet must be completed
with a PADDING frame. This option is activated when qc_prep_pkts()
encodes the last packet of a datagram with at least one INITIAL packet
in it.
Padding is resetted each time a new datagram is started. However, this
was not performed if GSO is used to built the next datagram. This patch
fixes it by properly resetting padding in this case also.
The impact of this bug is unknown. It may have several effectfs, one of
the most obvious being the insertion of unnecessary padding in packets.
It could also potentially trigger an infinite loop in qc_prep_pkts(),
although this has never been encountered so far.
MINOR: dns: dns_connect_nameserver: fix fd leak at error path
This fixes the commit 2c7e05f80e3b
("MEDIUM: dns: don't call connect to dest socket for AF_INET*"). If we fail to
bind AF_INET sockets or the address family of the nameserver protocol isn't
something, what we expect, we need to close the fd, obtained by
connect.
This fixes the issue GitHub #3085
This must be backported along with the commit 2c7e05f80e3b.
BUG/MAJOR: stream: Remove READ/WRITE events on channels after analysers eval
It is possible to miss a synchronous write event in process_stream() if the
stream was woken up on a write event. In that case, it is possible to freeze
the stream until the next I/O event or timeout.
Concretely, the stream is woken up with CF_WRITE_EVENT on a channel. this
flag is removed from the channel when we leave proces_stream(). But before
leaving process_stream(), when a synchronous send is tried on this channel,
the flag is removed and eventually set again on success. But this event is
masked by the previous one, and the channel is not resync as it should be.
To fix the bug, CF_READ_EVENT and CF_WRITE_EVENT flags are removed from a
channel after the corresponding analysers evaluation. This way, we will be
able to detect a successful synchronous send to restart analysers evaluation
based on the new channel state. It is safe (or it should be) to do so
becaues these flags are only used by analysers and tested to resync the
stream inside process_stream().
It is a very old bug and I guess all versions are affected. It was observed
on 2.9 and higher, and with the master/worker only. But it could affect any
stream. It is tagged a MAJOR because this area is really sensitive to any
change.
This patch should fix the issue #3070. It should probably be backported to
all stable versions, but only after a period of observation and with a
special care because this area is really sensitive to changes. It is
probably reasonnable to backport it as far as 3.0 and wait for older
versions.
BUG/MEDIUM: ssl: apply ssl-f-use on every "ssl" bind
This patch introduces a change of behavior in the configuration parsing.
Previously the "ssl-f-use" lines were only applied on "ssl" bind lines
that does not have any "crt" configured.
Since there is no warning and you could mix bind lines with and without
crt, this is really confusing.
This patch applies the "ssl-f-use" lines on every "ssl" bind lines.
BUG/MEDIUM: quic-be: crash after backend CID allocation failures
This bug impacts only the QUIC backends. It arrived with this commit:
MINOR: quic-be: QUIC connection allocation adaptation (qc_new_conn())
which was supposed to be fixed by:
BUG/MEDIUM: quic: crash after quic_conn allocation failures
but this commit was not sufficient.
Such a crashe could be reproduced with -dMfail option. To reach it, the
<conn_id> object allocation must fail (from qc_new_conn()). So, this is
relatively rare, except on systems with limited memory.
Amaury Denoyelle [Thu, 21 Aug 2025 12:16:58 +0000 (14:16 +0200)]
BUG/MEDIUM: mux-h2: fix crash on idle-ping due to unwanted ABORT_NOW
An ABORT_NOW() was used during debugging idle-ping but was not removed
from the final code. This may cause crash, in particular when mixing
idle-ping with shorter http-request/http-keep-alive values.
Fix this situation by removing ABORT_NOW() statement.
Willy Tarreau [Wed, 20 Aug 2025 19:52:39 +0000 (21:52 +0200)]
[RELEASE] Released version 3.3-dev7
Released version 3.3-dev7 with the following main changes :
- MINOR: quic: duplicate GSO unsupp status from listener to conn
- MINOR: quic: define QUIC_FL_CONN_IS_BACK flag
- MINOR: quic: prefer qc_is_back() usage over qc->target
- BUG/MINOR: cfgparse: immediately stop after hard error in srv_init()
- BUG/MINOR: cfgparse-listen: update err_code for fatal error on proxy directive
- BUG/MINOR: proxy: avoid NULL-deref in post_section_px_cleanup()
- MINOR: guid: add guid_get() helper
- MINOR: guid: add guid_count() function
- MINOR: clock: add clock_set_now_offset() helper
- MINOR: clock: add clock_get_now_offset() helper
- MINOR: init: add REGISTER_POST_DEINIT_MASTER() hook
- BUILD: restore USE_SHM_OPEN build option
- BUG/MINOR: stick-table: cap sticky counter idx with tune.nb_stk_ctr instead of MAX_SESS_STKCTR
- MINOR: sock: update broken accept4 detection for older hardwares.
- CI: vtest: add os name to OT cache key
- CI: vtest: add Ubuntu arm64 builds
- BUG/MEDIUM: ssl: Fix 0rtt to the server
- BUG/MEDIUM: ssl: fix build with AWS-LC
- MEDIUM: acme: use lowercase for challenge names in configuration
- BUG/MINOR: init: Initialize random seed earlier in the init process
- DOC: management: clarify usage of -V with -c
- MEDIUM: ssl/cli: relax crt insertion in crt-list of type directory
- MINOR: tools: implement ha_aligned_zalloc()
- CLEANUP: fd: make use of ha_aligned_alloc() for the fdtab
- MINOR: pools: distinguish the requested alignment from the type-specific one
- MINOR: pools: permit to optionally specify extra size and alignment
- MINOR: pools: always check that requested alignment matches the type's
- DOC: api: update the pools API with the alignment and typed declarations
- MEDIUM: tree-wide: replace most DECLARE_POOL with DECLARE_TYPED_POOL
- OPTIM: tasks: align task and tasklet pools to 64
- OPTIM: buffers: align the buffer pool to 64
- OPTIM: queue: align the pendconn pools to 64
- OPTIM: connection: align connection pools to 64
- OPTIM: server: start to use aligned allocs in server
- DOC: management: fix typo in commit f4f93c56
- DOC: config: recommend single quoting passwords
- MINOR: tools: also implement ha_aligned_alloc_typed()
- MEDIUM: server: introduce srv_alloc()/srv_free() to alloc/free a server
- MINOR: server: align server struct to 64 bytes
- MEDIUM: ring: always allocate properly aligned ring structures
- CI: Update to actions/checkout@v5
- MINOR: quic: implement qc_ssl_do_hanshake()
- BUG/MEDIUM: quic: listener connection stuck during handshakes (OpenSSL 3.5)
- BUG/MINOR: mux-h1: fix wrong lock label
- MEDIUM: dns: don't call connect to dest socket for AF_INET*
- BUG/MINOR: spoe: Properly detect and skip empty NOTIFY frames
- BUG/MEDIUM: cli: Report inbuf is no longer full when a line is consumed
- BUG/MEDIUM: quic: crash after quic_conn allocation failures
- BUG/MEDIUM: quic-be: do not initialize ->conn too early
- BUG/MEDIUM: mworker: more verbose error upon loading failure
- MINOR: xprt: Add recvmsg() and sendmsg() parameters to rcv_buf() and snd_buf().
- MINOR: ssl: Add a "flags" field to ssl_sock_ctx.
- MEDIUM: xprt: Add a "get_capability" method.
- MEDIUM: mux_h1/mux_pt: Use XPRT_CAN_SPLICE to decide if we should splice
- MINOR: cfgparse: Add a new "ktls" option to bind and server.
- MINOR: ssl: Define HAVE_VANILLA_OPENSSL if openssl is used.
- MINOR: build: Add a new option, USE_KTLS.
- MEDIUM: ssl: Add kTLS support for OpenSSL.
- MEDIUM: splice: Don't consider EINVAL to be a fatal error
- MEDIUM: ssl: Add splicing with SSL.
- MEDIUM: ssl: Add ktls support for AWS-LC.
- MEDIUM: ssl: Add support for ktls on TLS 1.3 with AWS-LC
- MEDIUM: ssl: Handle non-Application data record with AWS-LC
- MINOR: ssl: Add a way to globally disable ktls.
Olivier Houchard [Wed, 13 Aug 2025 16:34:10 +0000 (16:34 +0000)]
MINOR: ssl: Add a way to globally disable ktls.
Add a new global option, "noktls", as well as a command line option,
"-dT", to totally disable ktls usage, even if it is activated on servers
or binds in the configuration.
That makes it easier to quickly figure out if a problem is related to
ktls or not.
MEDIUM: ssl: Handle non-Application data record with AWS-LC
Handle receiving and sending TLS records that are not application data
records.
When receiving, we ignore new session tickets records, we handle close
notify as a read0, and we consider any other records as a connection
error.
For sending, we're just sending close notify, so that the TLS connection
is properly closed.
Olivier Houchard [Thu, 26 Jun 2025 11:06:18 +0000 (13:06 +0200)]
MEDIUM: ssl: Add support for ktls on TLS 1.3 with AWS-LC
AWS-LC added a new API in AWS-LC 1.54 that allows the user to retrieve
the keys for TLS 1.3 connections with SSL_get_read_traffic_secret(), so
use it to be able to use ktls with TLS 1.3 too.
Olivier Houchard [Thu, 19 Jun 2025 16:44:22 +0000 (18:44 +0200)]
MEDIUM: ssl: Add ktls support for AWS-LC.
Add ktls support for AWS-LC. As it does not know anything
about ktls, it means extracting keys from the ssl lib, and provide them
to the kernel. At which point we can use regular recvmsg()/sendmsg()
calls.
This patch only provides support for TLS 1.2, AWS-LC provides a
different way to extract keys for TLS 1.3.
Note that this may work with BoringSSL too, but it has not been tested.
Olivier Houchard [Wed, 13 Aug 2025 16:52:13 +0000 (16:52 +0000)]
MEDIUM: ssl: Add splicing with SSL.
Implement the splicing methods to the SSL xprt (which will just call the
raw_sock methods if kTLS is enabled on the socket), and properly report
that a connection supports splicing if kTLS is configured on that
connection.
For OpenSSL, if the upper layer indicated that it wanted to start using
splicing by adding the CO_FL_WANT_SPLICING flag, make sure we don't read
any more data from the socket, and just drain what may be in the
internal OpenSSL buffers, before allowing splicing
MEDIUM: splice: Don't consider EINVAL to be a fatal error
Don't consider that EINVAL is a fatal error, when calling splice().
When doing splicing from a kTLS socket, splice() will set errno to
EINVAL if the next record to be read is not an application data record.
This is not a fatal error, it just means we have to use recvmsg() to
read it, and potentially we can then resume using splicing.
It is unfortunate that EINVAL was used for that case, but we should
never get any other case of receiving EINVAL from splice(), so it should
be safe to treat it as non-fatal.
Modify the SSL code to enable kTLS with OpenSSL.
It mostly requires our internal BIO to be able to handle the various
kTLS-specific controls in ha_ssl_ctrl(), as well as being able to use
recvmsg() and sendmsg() from ha_ssl_read() and ha_ssl_write().
MINOR: ssl: Define HAVE_VANILLA_OPENSSL if openssl is used.
If we're using OpenSSL as our crypto library, so add a define,
HAVE_VANILLA_OPENSSL, to make it easier to differentiate between the
various crypto libs.
MINOR: cfgparse: Add a new "ktls" option to bind and server.
Add a new "ktls" option to bind and server. Valid values are "on" and
"off".
It currently does nothing, but when kTLS will be implemented, it will
enable or disable kTLS for the corresponding sockets.
It is marked as experimental for now.
Olivier Houchard [Wed, 13 Aug 2025 13:39:46 +0000 (13:39 +0000)]
MEDIUM: mux_h1/mux_pt: Use XPRT_CAN_SPLICE to decide if we should splice
In both mux_h1 and mux_pt, use the new XPRT_CAN_SPLICE capability to
decide if we should attempt to use splicing or not.
If we receive XPRT_CONN_CAN_MAYBE_SPLICE, add a new flag on the
connection, CO_FL_WANT_SPLICING, to let the xprt know that we'd love to
be able to do splicing, so that it may get ready for that.
This should have no effect right now, and is required work for adding
kTLS support.
Olivier Houchard [Wed, 13 Aug 2025 13:39:26 +0000 (13:39 +0000)]
MEDIUM: xprt: Add a "get_capability" method.
Add a new method to xprts, get_capability, that can be used to query if
an xprt supports something or not.
The first capability implemented is XPRT_CAN_SPLICE, to know if the xprt
will be able to use splicing for the provided connection.
The possible answers are XPRT_CONN_CAN_NOT_SPLICE, which indicates
splicing will never be possible for that connection,
XPRT_CONN_COULD_SPLICE, which indicates that splicing is not usable
right now, but may be in the future, and XPRT_CONN_CAN_SPLICE, that
means we can splice right away.
Olivier Houchard [Wed, 20 Aug 2025 13:30:48 +0000 (13:30 +0000)]
MINOR: ssl: Add a "flags" field to ssl_sock_ctx.
Instead of adding more separate fields in ssl_sock_ctx, add a "flags"
one.
Convert the "can_send_early_data" to the flag SSL_SOCK_F_EARLY_ENABLED.
More flags will be added for kTLS support.
Olivier Houchard [Tue, 17 Jun 2025 14:43:20 +0000 (16:43 +0200)]
MINOR: xprt: Add recvmsg() and sendmsg() parameters to rcv_buf() and snd_buf().
In rcv_buf() and snd_buf(), use sendmsg/recvmsg instead of send and
recv, and add two new optional parameters to provide msg_control and
msg_controllen.
Those are unused for now, but will be used later for kTLS.
BUG/MEDIUM: mworker: more verbose error upon loading failure
When a worker crashes during its configuration parsing and without
emitting any messages, the master will emit the message "Failed to load
worker!". However that doesn't give us neither the PID of the worker,
nor the status code.
This patch fixes the problem by emitting a more verbose error.
BUG/MEDIUM: quic-be: do not initialize ->conn too early
This bug arrived with this commit:
BUG/MEDIUM: quic: do not release BE quic-conn prior to upper conn
which added a BUG_ON(qc->conn) statement at the beginning of quic_conn_release().
It is triggered if the connection is not released before releasing the quic_conn.
But this is always the case for a backend quic_conn when its allocation from
qc_new_conn() fails.
Such crashes could be reproduced with -dMfail option. To reach them, the
memory allocations must fail. So, this is relatively rare, except on systems
with limited memory.
To fix this, simply set ->conn quic_conn struct member to a not null value
(the one passed as parameter) after the quic_conn allocation has succeeded.
where qc_new_conn() was modified. The ->cids allocation was moved without
checking if a quic_conn_release() call could lead to crashes due to uninitialized
quic_conn members. Indeed, if qc_new_conn() fails, then quic_conn_release() is
called. This bug could impact both QUIC servers and clients.
Such crashes could be reproduced with -dMfail option. To reach them, the
memory allocations must fail. So, this is relatively rare, except on systems
with limited memory.
This patch ensures all the quic_conn members which could lead to crash
from quic_conn_release() are initialized before any remaining memory allocations
required for the quic_conn.
The <conn_id> variable allocated by the client is no more attached to
the connection during its allocation, but after the ->cids trees is allocated.
BUG/MEDIUM: cli: Report inbuf is no longer full when a line is consumed
When the command line parsing was refactored (20ec1de21 "MAJOR: cli: Refacor
parsing and execution of pipelined commands"), a regression was introduced.
When input data are consumed, information about the applet's input buffer
are no longer updated accordingly to state it is no longer full. So it is
possible to freeze the CLI applet. And a spinning loop may be encountered if
a client shutdown is detected in this state.
The fix is obivous. When data are consumed from the applet's input buffer,
APPCTX_FL_INBLK_FULL flag is removed to notify the input buffer is no longer
full and more data can be sent to the CLI applet.
This patch should fix the issue #3064. It must be backported to 3.2.
BUG/MINOR: spoe: Properly detect and skip empty NOTIFY frames
Since the SPOE was refactored, the detection of empty NOTIFY frames is
broken. So it is possible to send a NOTIFY frames to an agent with no
message at all. The bug happens because the frame type is now added to the
buffer before the messages encoding. So the buffer is never really empty.
To fix the issue, the condition to detect empty frame was adapted.
MEDIUM: dns: don't call connect to dest socket for AF_INET*
When we perform connect call for a datagram socket, used to send DNS requests,
we set for it the default destination address to some given nameserver. Then we
simply use send(), as the destination address is already set. In some usecases
described in GitHub issues #3001 and #2654, this approach becames inefficient,
nameservers change its IP addresses dynamically, this triggers DNS resolution
errors.
To fix this, let's perform the bind() on the wildcard address for the datagram
AF_INET* client socket. Like this we will allocate a port for it. Then let's
use sendto() instead of send().
If the nameserver is local and is listening on the UNIX domain socket, we
continue to use the existed approach (connect() and then send()).
This fixes issues #3001 and #2654.
This may be backported in all stable versions.
BUG/MEDIUM: quic: listener connection stuck during handshakes (OpenSSL 3.5)
This issue was reported in GH #3071 by @famfo where a wireshark capture
reveals that some handshake could not complete after having received
two Initial packets. This could happen when the packets were parsed
in two times, calling qc_ssl_provide_all_quic_data() two times.
This is due to crypto data stream counter which was incremented two times
from qc_ssl_provide_all_quic_data() (see cstream->rx.offset += data
statement around line 1223 in quic_ssl.c). One time by the callback
which "receives" the crypto data, and on time by qc_ssl_provide_all_quic_data().
Then when parsing the second crypto data frame, the parser detected
that the crypto were already provided.
To fix this, one could comment the code which increment the crypto data
stream counter by <data>. That said, when using the OpenSSL 3.5 QUIC API
one should not modified the crypto data stream outside of the OpenSSL 3.5
QUIC API.
So, this patch stop calling qc_ssl_provide_all_quic_data() and
qc_ssl_provide_quic_data() and only calls qc_ssl_do_hanshake() after
having received some crypto data. In addition to this, as these functions
are no more called when building haproxy against OpenSSL 3.5, this patch
disable their compilations (with #ifndef HAVE_OPENSSL_QUIC).
Willy Tarreau [Wed, 13 Aug 2025 15:36:18 +0000 (17:36 +0200)]
MEDIUM: ring: always allocate properly aligned ring structures
The rings were manually padded to place the various areas that compose
them into different cache lines, provided that the allocator returned
a cache-aligned address, which until now was not granted. By now
switching to the aligned API we can finally have this guarantee and
hope for more consistent ring performance between tests. Like previously
the few carefully crafted THREAD_PAD() could simply be replaced by
generic THREAD_ALIGN() that dictate the type's alignment.
This was the last user of THREAD_PAD() by the way.
Willy Tarreau [Wed, 13 Aug 2025 13:07:55 +0000 (15:07 +0200)]
MINOR: server: align server struct to 64 bytes
Several times recently, it was noticed that some benchmarks would
highly vary depending on the position of certain fields in the server
struct, and this could even vary between runs.
The server struct does have separate areas depending on the user cases
and hot/cold aspect of the members stored there, but the areas are
artificially kept apart using fixed padding instead of real alignment,
which has the first sad effect of artificially inflating the struct,
and the second one of misaligning it.
Now that we have all the necessary tools to keep them aligned, let's
just do it. The struct has shrunk from 4160 to 4032 bytes on 64-bit
systems, 152 of which are still holes or padding.
Willy Tarreau [Wed, 13 Aug 2025 09:52:59 +0000 (11:52 +0200)]
MEDIUM: server: introduce srv_alloc()/srv_free() to alloc/free a server
It happens that we free servers at various places in the code, both
on error paths and at runtime thanks to the "server delete" feature. In
order to switch to an aligned struct, we'll need to change the calloc()
and free() calls. Let's first spot them and switch them to srv_alloc()
and srv_free() instead of using calloc() and either free() or ha_free().
An easy trap to fall into is that some of them are default-server
entries. The new srv_free() function also resets the pointer like
ha_free() does.
This was done by running the following coccinelle script all over the
code:
This is marked medium because despite spotting all call places, we can
never rule out the possibility that some out-of-tree patches would
allocate their own servers and continue to use the old API... at their
own risk.
Willy Tarreau [Wed, 13 Aug 2025 15:11:32 +0000 (17:11 +0200)]
MINOR: tools: also implement ha_aligned_alloc_typed()
This one is a macro and will allocate a properly aligned and sized
object. This will help make sure that the alignment promised to the
compiler is respected.
When memstats is used, the type name is passed as a string into the
.extra field so that it can be displayed in "debug dev memstats". Two
tiny mistakes related to memstats macros were also fixed (calloc
instead of malloc for zalloc), and the doc was also added to document
how to use these calls.
Willy Tarreau [Mon, 11 Aug 2025 15:43:51 +0000 (17:43 +0200)]
OPTIM: server: start to use aligned allocs in server
This is currently for per-thread arrays like idle conns etc. We're
now cache-aligning the per-thread arrays so as to put an end to false
sharing. A comparative test between no alignment and alignment on a
simple config with round robin between 4 servers showed an average
rate of 1.75M/s vs 1.72M/s before for 100M requests. The gain seems
to be more commonly less than 1% however. This should mostly help
make measurements more reproducible across multiple runs.
Willy Tarreau [Mon, 11 Aug 2025 15:36:39 +0000 (17:36 +0200)]
OPTIM: connection: align connection pools to 64
The struct connection is used a lot by the muxes during many operations,
particularly at the beginning of the struct (flags, ctrl, xprt and mux).
We definitely want this one not to be falsely shared with another thread,
so let's align the pools to a cache line.
Willy Tarreau [Mon, 11 Aug 2025 15:31:08 +0000 (17:31 +0200)]
OPTIM: buffers: align the buffer pool to 64
This struct is used by memcpy() and friends, particularly during the
early recv() and send(). By keeping it 64-byte aligned, we let the
underlying libs/kernel use optimal operations (e.g. AVX512) for memory
copies while right now it's just random (buffers are found to be equally
aligned to 32 and 64 in practice).
Willy Tarreau [Mon, 11 Aug 2025 15:29:04 +0000 (17:29 +0200)]
OPTIM: tasks: align task and tasklet pools to 64
These structs are intensively used and really must not experience false
sharing, so let's declare them aligned to 64. We don't try to align the
struct themselves, as we don't want the compiler to expand them either.
Willy Tarreau [Wed, 6 Aug 2025 14:43:27 +0000 (16:43 +0200)]
MEDIUM: tree-wide: replace most DECLARE_POOL with DECLARE_TYPED_POOL
This will make the pools size and alignment automatically inherit
the type declaration. It was done like this:
sed -i -e 's:DECLARE_POOL(\([^,]*,[^,]*,\s*\)sizeof(\([^)]*\))):DECLARE_TYPED_POOL(\1\2):g' $(git grep -lw DECLARE_POOL src addons)
sed -i -e 's:DECLARE_STATIC_POOL(\([^,]*,[^,]*,\s*\)sizeof(\([^)]*\))):DECLARE_STATIC_TYPED_POOL(\1\2):g' $(git grep -lw DECLARE_STATIC_POOL src addons)
81 replacements were made. The only remaining ones are those which set
their own size without depending on a structure. The few ones with an
extra size were manually handled.
It also means that the requested alignments are now checked against the
type's. Given that none is specified for now, no issue is reported.
It was verified with "show pools detailed" that the definitions are
exactly the same, and that the binaries are similar.
Willy Tarreau [Mon, 11 Aug 2025 14:47:17 +0000 (16:47 +0200)]
MINOR: pools: always check that requested alignment matches the type's
For pool registrations that are created from the type declaration, we
now have the ability to verify that the requested alignment matches
the type's one. Let's not miss this opportunity, as we've met bugs in
the past that were caused by such mismatches. The principle is simple:
if the type alignment is known, we check that the configured alignment
is at least as large as that one otherwise we refuse to start (since
the code may crash at any moment). Obviously it doesn't crash for now!
Willy Tarreau [Mon, 11 Aug 2025 13:51:32 +0000 (15:51 +0200)]
MINOR: pools: permit to optionally specify extra size and alignment
The common macros REGISTER_TYPED_POOL(), DECLARE_TYPED_POOL() and
DECLARE_STATIC_TYPED_POOL() will now take two optional arguments,
one being the extra size to be added to the structure, and a second
one being the desired alignment to enforce. This will permit to
specify alignments larger than the default ones promised to the
compiler.
Willy Tarreau [Mon, 11 Aug 2025 13:24:33 +0000 (15:24 +0200)]
MINOR: pools: distinguish the requested alignment from the type-specific one
We're letting users request an alignment but that can violate one imposed
by a type, especially if we start seeing REGISTER_TYPED_POOL() grow in
adoption, encouraging users to specify alignment on their types. On the
other hand, if we ask the user to always specify the alignment, no control
is possible and the error is easy. Let's have a second field in the pool
registration, for the type-specific one. We'll set it to zero when unknown,
and to the types's alignment when known. This way it will become possible
to compare them at startup time to detect conflicts. For now no macro
permits to set both separately so this is not visible.
Willy Tarreau [Mon, 11 Aug 2025 16:55:09 +0000 (18:55 +0200)]
CLEANUP: fd: make use of ha_aligned_alloc() for the fdtab
We've forcefully aligned the fdtab in commit 97ea9c49f1 ("BUG/MEDIUM:
fd: always align fdtab[] to 64 bytes"), but now we don't need such hacks
anymore thanks to ha_aligned_alloc(). Let's use it and get rid of
fdtab_addr.
Willy Tarreau [Mon, 11 Aug 2025 16:46:28 +0000 (18:46 +0200)]
MINOR: tools: implement ha_aligned_zalloc()
This one is exactly ha_aligned_alloc() followed by a memset(0), as
it will be convenient for a number of call places as a replacement
for calloc().
Note that ideally we should also have a calloc version that performs
basic multiply overflow checks, but these are essentially used with
numbers of threads times small structs so that's fine, and we already
do the same everywhere in malloc() calls.
MEDIUM: ssl/cli: relax crt insertion in crt-list of type directory
In previous versions of haproxy, insertions of certificates in a
crt-list from the CLI would require to have the path of the directory,
in the path of the certificate. This would help avoiding that the
certificate wasn't loaded upon a reload because it is not at the right
place.
However, since version 3.0 and crt-store, the name stored in the tree
could be an alias and not a path, so that does not make sense anymore.
Even though path would be right, the check is not right anymore in this
case.
The tool or user inserting the certificate must now check itself that
the certificate was placed at the right spot on the filesystem.
In ticket #3065 an user complained that no success message is printed
anymore when using -c. The message does not appear by default since
version 2.9. This patch clarify the documentation.
BUG/MINOR: init: Initialize random seed earlier in the init process
The random seed used in ha_random functions needs to be first
initialized by calling ha_random_boot. This function was called rather
late in the init process, after the init functions (INITCALLS) are
called and after the configuration parsing for instance which means that
any ha_random call in an init function would return 0. This was the case
in 'vars_init' and 'cache_init' which tried to build seeds for specific
hash calculations but ended up not being seeded.
This patch can be backported on all stable branches.
MEDIUM: acme: use lowercase for challenge names in configuration
Both the RFC and the IANA registry refers to challenge names in
lowercase. If we need to implement more challenges, it's better to
use the correct naming.
In order to keep the compatibility with the previous configurations, the
parsing does a strcasecmp() instead of a strcmp().
Also rename every occurence in the code and doc in lowercase.
AWS-LC doesn't provide SSL_in_before(), and doesn't provide an easy way
to know if we already started the handshake or not. So instead, just add
a new field in ssl_sock_ctx, "can_write_early_data", that will be
initialized to 1, and will be set to 0 as soon as we start the
handshake.
In order to send early data, we have to make sure no handshake has been
initiated at all. To do that, we remove the CO_FL_SSL_WAIT_HS flag, so
that we won't attempt to start a handshake. However, by removing those
flags, we allow ssl_sock_to_buf() to call SSL_read(), as it's no longer
aware that no handshake has been done, and SSL_read() will begin the
handshake, thus preventing us from sending early data.
The fix is to just call SSL_in_before() to check if no handshake has
been done yet, in addition to checking CO_FL_SSL_WAIT_HS (both are
needed, as CO_FL_SSL_WAIT_HS may come back in case of renegociation).
In ssl_sock_from_buf(), fix the check to see if we may attempt to send
early data. Use SSL_in_before() instead of SSL_is_init_finished(), as
SSL_is_init_finished() will return 1 if the handshake has been started,
but not terminated, and if the handshake has been started, we can no
longer send early data.
This fixes errors when attempting to send early data (as well as
actually sending early data).
BUG/MINOR: stick-table: cap sticky counter idx with tune.nb_stk_ctr instead of MAX_SESS_STKCTR
Cap sticky counter index with tune.nb_stk_ctr instead of MAX_SESS_STKCTR for
sc-add-gpc. Same logic is already implemented for sc-inc-gpc and sc-set-gpt
keywords. So, it seems missed for sc-add-gpc.
This fixes the issue #3061 reported at GitHub. Thanks to @ma311 for
reporting their analysis of the issue.
This should be backported in all versions until 2.8, included 2.8.
Some optional features may still require the use of shm_open() in the
future. In this patch we restore the USE_SHM_OPEN build option that
was removed in 143be1b59 ("MEDIUM: errors: get rid of shm_open()") and
should guard the use of shm_open() in the code.
Similar to REGISTER_POST_DEINIT() hook (which is invoked during deinit)
but for master process only, when haproxy was started in master-worker
mode. The goal is to be able to register cleanup functions that will
only run for the master process right before exiting.
Since now_offset is a static variable and is not exposed outside from
clock.c, let's add an helper so that it becomes possible to set its
value from another source file.
BUG/MINOR: proxy: avoid NULL-deref in post_section_px_cleanup()
post_section_px_cleanup(), which was implemented in abcc73830
("MEDIUM: proxy: register a post-section cleanup function"), is called
for the current section no matter if the parsing was aborted due to
a fatal error. In this case, the curproxy pointer may point to NULL,
yet post_section_px_cleanup() assumes curproxy pointer is always valid,
which could lead to NULL-deref.
For instance, the config below will cause SEGFAULT:
listen toto titi
To fix the issue, let's simply consider that the curproxy pointer may
be NULL in post_section_px_cleanup(), in which case we skip the cleanup
for the curproxy since there is nothing we can do.
BUG/MINOR: cfgparse-listen: update err_code for fatal error on proxy directive
When improper arguments are provided on proxy directive (listen,
frontend or backend), such alert may be emitted:
"please use the 'bind' keyword for listening addresses"
This was introduced in 6e62fb6405 ("MEDIUM: cfgparse: check section
maximum number of arguments"). However, despite the error being reported
as alert, the err_code isn't updated accordingly, which could make the
upper parser think there was no error, while it isn't the case.
In practise since the proxy directive is ignored following proxy related
directives should raise errors, so this didn't cause much harm, yet
better fix that.
BUG/MINOR: cfgparse: immediately stop after hard error in srv_init()
Since 368d01361 (" MEDIUM: server: add and use srv_init() function"), in
case of srv_init() error, we simply increment cfgerr variable and keep
going.
It isn't enough, some treatment occuring later in check_config_validity()
assume that srv_init() succeeded for servers, and may cause undefined
behavior. To fix the issue, let's consider that if (srv_init() & ERR_CODE)
returns true, then we must stop checking the config immediately.
MINOR: quic: prefer qc_is_back() usage over qc->target
Previously quic_conn <target> member was used to determine if quic_conn
was used on the frontend (as server) or backend side (as client). A new
helper function can now be used to directly check flag
QUIC_FL_CONN_IS_BACK.
This reduces the dependency between quic_conn and their relative
listener/server instances.
Define a new quic_conn flag assign if the connection is used on the
backend side. This is similar to other haproxy components such as struct
connection and muxes element.
This flag is positionned via qc_new_conn(). Also update quic traces to
mark proxy side as 'F' or 'B' suffix.
MINOR: quic: duplicate GSO unsupp status from listener to conn
QUIC emission can use GSO to emit multiple datagrams with a single
syscall invokation. However, this feature relies on several kernel
parameters which are checked on haproxy process startup.
Even if these checks report no issue, GSO may still be unable due to the
underlying network adapter underneath. Thus, if a EIO occured on
sendmsg() with GSO, listener is flagged to mark GSO as unsupported. This
allows every other QUIC connections to share the status and avoid using
GSO when using this listener.
Previously, listener flag was checked for every QUIC emission. This was
done using an atomic operation to prevent races. Improve this by
duplicating GSO unsupported status as the connection level. This is done
on qc_new_conn() and also on thread rebinding if a new listener instance
is used.
The main benefit from this patch is to reduce the dependency between
quic_conn and listener instances.
projects / thirdparty / haproxy.git / log
