BUG/MINOR: config: don't count trailing spaces as empty arg (v2)
In parse_line(), spaces increment the arg count and it is incremented
again on '#' or end of line, resulting in an extra empty arg at the
end of arg's list. The visible effect is that the reported arg count
is in excess of 1. It doesn't seem to affect regular function but
specialized ones like anonymisation depends on this count.
This is the second attempt for this problem, here the explanation :
When called for the first line, no <out> was allocated yet so it's NULL,
letting the caller realloc a larger line if needed. However the words are
parsed and their respective args[arg] are filled with out+position, which
means that while the first arg is NULL, the other ones are no and fail the
test that was meant to avoid dereferencing a NULL. Let's simply check <out>
instead of <args> since the latter is always derived from the former and
cannot be NULL without the former also being.
This may need to be backported to stable versions.
BUG/MINOR: hlua: prevent crash when loading numerous arguments using lua-load(per-thread)
When providing multiple optional arguments with lua-load or
lua-load-per-thread directives, arguments where pushed 1 by 1
to the stack using lua_pushstring() without checking if the stack
could handle it.
This could easily lead to program crash when providing too much
arguments. I can easily reproduce the crash starting from ~50 arguments.
Calling lua_checkstack() before pushing to the stack fixes the crash:
According to lua.org, lua_checkstack() does some housekeeping and
allow the stack to be expanded as long as some memory is available
and the hard limit isn't reached.
When no memory is available to expand the stack or the limit is reached,
lua_checkstacks returns an error: in this case we force hlua_load_state()
to return a meaningfull error instead of crashing.
In practice though, cfgparse complains about too many words
way before such event may occur on a normal system.
TLDR: the ~50 arguments limitation is not an issue anymore.
No backport needed, except if 'MINOR: hlua: Allow argument on
lua-lod(-per-thread) directives' (ae6b568) is backported.
BUG/MINOR: hlua: _hlua_http_msg_delete incorrect behavior when offset is used
Calling the function with an offset when "offset + len" was superior or equal
to the targeted blk length caused 'v' value to be improperly set.
And because 'v' is directly provided to htx_replace_blk_value(), blk consistency was compromised.
(It seems that blk was overrunning in htx_replace_blk_value() due to
this and header data was overwritten in this case).
This patch adds the missing checks to make the function behave as
expected when offset is set and offset+len is greater or equals to the targeted blk length.
Some comments were added to the function as well.
MINOR: tools: Impprove hash_ipanon to not hash FD-based addresses
"stdout" and "stderr" are not hashed. In the same spirit, "fd@" and
"sockpair@" prefixes are not hashed too. There is no reason to hash such
address and it may be useful to diagnose bugs.
No backport needed, except if anonymization mechanism is backported.
MINOR: cli: correct commentary and replace 'set global-key' name
Correct a commentary in in include/haproxy/global-t.h and include/haproxy/tools.h
Replace the CLI command 'set global-key <key>' by 'set anon global-key <key>' in
order to find it easily when you don't remember it, the recommandation can guide
you when you just tap 'set anon'.
No backport needed, except if anonymization mechanism is backported.
MINOR: config: Add option line when the configuration file is dumped
Add an option to dump the number lines of the configuration file when
it's dumped. Other options can be easily added. Options are separated
by ',' when tapping the command line:
'./haproxy -dC[key],line -f [file]'
No backport needed, except if anonymization mechanism is backported.
MINOR: config: Add other keywords when dump the anonymized configuration file
Add keywords recognized during the dump of the configuration file,
these keywords are followed by sensitive information.
Remove the condition 'localhost' for the second argument of keyword
'server', consider as not essential and can disturb when comparing
it in cli section (there is no exception 'localhost').
No backport needed, except if anonymization mechanism is backported.
MINOR: tools: modify hash_ipanon in order to use it in cli
Add a parameter hasport to return a simple hash or ipstring when
ipstring has no port. Doesn't hash if scramble is null. Add
option PA_O_PORT_RESOLVE to str2sa_range. Add a case UNIX.
Those modification permit to use hash_ipanon in cli section
in order to dump the same anonymization of address in the
configuration file and with CLI.
No backport needed, except if anonymization mechanism is backported.
MINOR: cli: remove error message with 'set anon on|off'
Removed the error message in 'set anon on|off', it's more user
friendly: users use 'set anon on' even if the mode is already
activated, and the same for 'set anon off'. That allows users
to write the command line in the anonymized mode they want
without errors.
No backport needed, except if anonymization mechanism is backported.
hlua_http_msg_insert_data() function is called upon
HTTPMessage.insert() method from lua script.
This function did not work properly for multiple reasons:
- An incorrect argument check was performed and prevented the user
from providing optional offset argument.
- Input and output variables were inverted
and offset was not handled properly. The same bug
was also fixed in hlua_http_msg_del_data(), see:
'BUG/MINOR: hlua: fixing hlua_http_msg_del_data behavior'
The function now behaves as described in the documentation.
GH issue #1885 reported that HTTPMessage.remove() did not
work as expected.
It turns out that underlying hlua_http_msg_del_data() function
was not working properly due to input / output inversion as well
as incorrect user offset handling.
This patch fixes it so that the behavior is the one described in
the documentation.
Since this patch, HAProxy crashes when the first line of the configuration
file contains more than one parameter because, on the first call of
parse_line(), the output line is not allocated. Thus elements in the
arguments array may point on invalid memory area.
It may be considered as a bug to reference invalid memory area and should be
fixed. But for now, it is safer to revert this patch
If the reverted commit is backported, this one must be backported too.
BUG/MINOR: config: don't count trailing spaces as empty arg
In parse_line(), spaces increment the arg count and it is incremented
again on '#' or end of line, resulting in an extra empty arg at the
end of arg's list. The visible effect is that the reported arg count
is in excess of 1. It doesn't seem to affect regular function but
specialized ones like anonymisation depends on this count.
This may need to be backported to stable versions.
BUG/MEDIUM: resolvers: Remove aborted resolutions from query_ids tree
To avoid any UAF when a resolution is released, a mechanism was added to
abort a resolution and delayed the released at the end of the current
execution path. This mechanism depends on an hard assumption: Any reference
on an aborted resolution must be removed. So, when a resolution is aborted,
it is removed from the resolver lists and inserted into a death row list.
However, a resolution may still be referenced in the query_ids tree. It is
the tree containing all resolutions with a pending request. Because aborted
resolutions are released outside the resolvers lock, it is possible to
release a resolution on a side while a query ansswer is received and
processed on another one. Thus, it is still possible to have a UAF because
of this bug.
To fix the issue, when a resolution is aborted, it is removed from any list,
but it is also removed from the query_ids tree.
This patch should solve the issue #1862 and may be related to #1875. It must
be backported as far as 2.2.
BUG/MEDIUM: stconn: Reset SE descriptor when we fail to create a stream
If stream_new() fails after the frontend SC is attached, the underlying SE
descriptor is not properly reset. Among other things, SE_FL_ORPHAN flag is
not set again. Because of this error, a BUG_ON() is triggered when the mux
stream on the frontend side is destroyed.
Thus, now, when stream_new() fails, SE_FL_ORPHAN flag is set on the SE
descriptor and its stream-connector is set to NULL.
This patch should solve the issue #1880. It must be backported to 2.6.
BUG/MINOR: stream: Perform errors handling in right order in stream_new()
The frontend SC is attached before the backend one is allocated. Thus an
allocation error on backend SC must be handled before an error on the
frontend SC.
CLEANUP: list: fix again some style issues in the recent comments
While reading the recent changes around mt_list_for_each_entry_safe() I
noticed a spurious "q" at the beginning of a line introduced by commit 455843721 ("CLEANUP: list: Fix mt_list_for_each_entry_safe indentation")
and that visually confusing multi-line comments missing the trailing '\'
character were introduced by previous commit 60cffbaca ("MINOR: list:
documenting mt_list_for_each_entry_safe() macro"), which at first glance
made the macro look broken. In addition, multi-line comments must end
with a "*/" on its own line to instantly spot where it ends without
having to read the whole line, like this:
/* we know from the above that foo is always valid
* here so it's safe to end the string:
*/
*(unsigned char *)foo = 0;
Not like this:
/* we know from the above that foo is always valid
* here so it's safe to end the string: */
*(unsigned char *)foo = 0;
Finally, macro's main comment mentionned the wrong macro name and types,
and was randomly indented.
BUG/MINOR: hlua: fixing ambiguous sizeof in hlua_load_per_thread
As pointed out by chipitsine in GH #1879, coverity complains
about a sizeof with char ** type where it should be char *.
This was introduced in 'MINOR: hlua: Allow argument on
lua-lod(-per-thread) directives' (ae6b568)
Luckily this had no effect so far because on most platforms
sizeof(char **) == sizeof(char *), but this can not be safely
assumed for portability reasons.
The fix simply changes the argument to sizeof so that it refers to
'*per_thread_load[len]' instead of 'per_thread_load[len]'.
MEDIUM: mworker/cli: keep the connection of the FD that ask for a reload
When using the "reload" command over the master CLI, all connections to
the master CLI were cut, this was unfortunate because it could have been
used to implement a synchronous reload command.
This patch implements an architecture to keep the connection alive after
the reload.
The master CLI is now equipped with a listener which uses a socketpair,
the 2 FDs of this socketpair are stored in the mworker_proc of the
master, which the master keeps via the environment variable.
ipc_fd[1] is used as a listener for the master CLI. During the "reload"
command, the CLI will send the FD of the current session over ipc_fd[0],
then the reload is achieved, so the master won't handle the recv of the
FD. Once reloaded, ipc_fd[1] receives the FD of the session, so the
connection is preserved. Of course it is a new context, so everything
like the "prompt mode" are lost.
BUG/MINOR: anon: memory illegal accesses in tools.c with hash_anon and hash_ipanon
chipitsine reported in github issue #1872 that in function hash_anon and
hash_ipanon, index_hash can be equal to NB_L_HASH_WORD and can reach an
inexisting line table, the table is initialized hash_word[NB_L_HASH_WORD][20];
so hash_word[NB_L_HASH_WORD] doesn't exist.
No backport needed, except if anonymization mechanism is backported.
BUG/MINOR: hlua: Remove \n in Lua error message built with memprintf
Because memprintf return an error to the caller and not on screen.
the function which perform display of message on the right output
is in charge of adding \n if it is necessary.
- Adding some comments in mt_list_for_each_entry_safe() macro to make it
somehow understandable.
The macro is performing critical stuff but was not documented at all.
Moreover, nested loops with conditional tricks are used,
making it even harder to understand the steps performed in it.
- Added a "FIXME:" comment in a specific condition that seems to
never be reached even when deeply stress-testing mt_lists
(using test_list binary provided in the repository).
BUG/MINOR: smtpchk: SMTP Service check should gracefully close SMTP transaction
At present option smtpchk closes the TCP connection abruptly on completion of service checking,
even if successful. This can result in a very high volume of errors in backend SMTP server logs.
This patch ensures an SMTP QUIT is sent and a positive 2xx response is received from the SMTP
server prior to disconnection.
This patch depends on the following one:
* MINOR: smtpchk: Update expect rule to fully match replies to EHLO commands
This patch should fix the issue #1812. It may be backported as far as 2.2
with the commit above On the 2.2, proxy_parse_smtpchk_opt() function is
located in src/check.c
MINOR: smtpchk: Update expect rule to fully match replies to EHLO commands
The response to EHLO command is a multiline reply. However the corresponding
expect rule only match on the first line. For now, it is not an issue. But
to be able to send the QUIT command and gracefully close the connection, we
must be sure to consume the full EHLO reply first.
To do so, the regex has been updated to match all 2xx lines at a time.
The s1 server is acting like a SMTP server. But it sends two CRLF at the end of
each line, while only one CRLF must be returned. It only works becaue both CRLF
are received at the same time.
MINOR: clock: do not update the global date too often
Tests with forced wakeups on a 24c/48t machine showed that we're caping
at 7.3M loops/s, which means 6.6 microseconds of loop delay without
having anything to do.
This is caused by two factors:
- the load and update of the now_offset variable
- the update of the global_now variable
What is happening is that threads are not running within the one-
microsecond time precision provided by gettimeofday(), so each thread
waking up sees a slightly different date and causes undesired updates
to global_now. But worse, these undesired updates mean that we then
have to adjust the now_offset to match that, and adds significant noise
to this variable, which then needs to be updated upon each call.
By only allowing sightly less precision we can completely eliminate
that contention. Here we're ignoring the 5 lowest bits of the usec
part, meaning that the global_now variable may be off by up to 31 us
(16 on avg). The variable is only used to correct the time drift some
threads might be observing in environments where CPU clocks are not
synchronized, and it's used by freq counters. In both cases we don't
need that level of precision and even one millisecond would be pretty
fine. We're just 30 times better at almost no cost since the global_now
and now_offset variables now only need to be updated 30000 times a
second in the worst case, which is unnoticeable.
After this change, the wakeup rate jumped from 7.3M/s to 66M/s, meaning
that the loop delay went from 6.6us to 0.73us, that's a 9x improvement
when under load! With real tasks we're seeing a boost from 28M to 52M
wakeups/s. The clock_update_global_date() function now only takes
1.6%, it's good enough so that we don't need to go further.
MINOR: pollers: only update the local date during busy polling
This patch modifies epoll, kqueue and evports (the 3 pollers that support
busy polling) to only update the local date in the inner polling loop,
the global one being done when leaving the loop. Testing with epoll on
a 24c/48t machine showed a boost from 53M to 352M loops/s, indicating
that the loop was spending 85% of its time updating the global date or
causing side effects (which was confirmed with perf top showing 67% in
clock_update_global_date() alone).
Pollers that support busy polling spend a lot of time (and cause
contention) updating the global date when they're looping over themselves
while it serves no purpose: what's needed is only an update on the local
date to know when to stop looping.
This patch splits clock_pudate_date() into a pair of local and global
update functions, so that pollers can be easily improved.
BUG/MINOR: log: improper behavior when escaping log data
Patrick Hemmer reported an improper log behavior when using
log-format to escape log data (+E option):
Some bytes were truncated from the output:
- escape_string() function now takes an extra parameter that
allow the caller to specify input string stop pointer in
case the input string is not guaranteed to be zero-terminated.
- Minors checks were added into lf_text_len() to make sure dst
string will not overflow.
- lf_text_len() now makes proper use of escape_string() function.
BUG/MINOR: mux-h1: Account consumed output data on synchronous connection error
The commit 372b38f935 ("BUG/MEDIUM: mux-h1: Handle connection error after a
synchronous send") introduced a bug. In h1_snd_buf(), consumed data are not
properly accounted if a connection error is detected. Indeed, data are
consumed when the output buffer is filled. But, on connection error, we exit
from the loop without incremented total variable accordingly.
When this happens, this leaves the channel buffer in an inconsistent
state. The buffer may be empty with some output at the channel level.
Because an error is reported, it is harmless. But it is safer to fix this
bug now to avoid any regression in future.
BUG/MEDIUM: mux-quic: properly trim HTX buffer on snd_buf reset
MUX QUIC snd_buf operation whill return early if a qcs instance is
resetted. In this case, HTX is left untouched and the callback returns
the whole bufer size. This lead to an undefined behavior as the stream
layer is notified about a transfer but does not see its HTX buffer
emptied. In the end, the transfer may stall which will lead to a leak on
session.
To fix this, HTX buffer is now resetted when snd_buf is short-circuited.
This should fix the issue as now the stream layer can continue the
transfer until its completion.
This patch has already been tested by Tristan and is reported to solve
the github issue #1801.
Factorize common code between h3 and hq-interop snd_buf operation. This
is inserted in MUX QUIC snd_buf own callback.
The h3/hq-interop API has been adjusted to directly receive a HTX
message instead of a plain buf. This led to extracting part of MUX QUIC
snd_buf in qmux_http module.
REORG: mux-quic: extract traces in a dedicated source file
QUIC MUX implements several APIs to interface with stream, quic-conn and
app-ops layers. It is planified to better separate this roles, possibly
by using several files.
The first step is to extract QUIC MUX traces in a dedicated source
files. This will allow to reuse traces in multiple files.
The main objective is to be
able to support both TCP and HTTP proxy mode with a common base and add
specialized modules on top of it.
BUG/MINOR: mux-quic: do not keep detached qcs with empty Tx buffers
A qcs instance free may be postponed in stream detach operation if the
stream is not locally closed. This condition is there to achieve
transfering data still present in Tx buffer. Once all data have been
emitted to quic-conn layer, qcs instance can be released.
However, the stream is only closed locally if HTX EOM has been seen or
it has been resetted. In case the transfer finished without EOM, a
detached qcs won't be freed even if there is no more activity on it.
This bug was not reproduced but was found on code analysis. Its precise
impact is unknown but it should not cause any leak as all qcs instances
are freed with its parent qcc connection : this should eventually happen
on MUX timeout or QUIC idle timeout.
To adjust this, condition to mark a stream as locally closed has been
extended. On qcc_streams_sent_done() notification, if its Tx buffer has
been fully transmitted, it will be closed if either FIN STREAM was set
or the stream is detached.
OPTIM: hpack-huff: reduce the cache footprint of the huffman decoder
Some tables are currently used to decode bit blocks and lengths. We do
see such lookups in perf top. We have 4 512-byte tables and one 64-byte
one. Looking closer, the second half of the table (length) has so few
variations that most of the time it will be computed in a single "if",
and never more than 3. This alone allows to cut the tables in half. In
addition, one table (bits 15-11) is only 32-element long, while another
one (bits 11-4) starts at 0x60, so we can merge the two as they do not
overlap, and further save size. We're now down to 4 256-entries tables.
This is visible in h3 and h2 where the max request rate is slightly higher
(e.g. +1.6% for h2). The huff_dec() function got slightly larger but the
overall code size shrunk:
$ nm --size haproxy-before | grep huff_dec 000000000000029e T huff_dec
$ nm --size haproxy-after | grep huff_dec 0000000000000345 T huff_dec
$ size haproxy-before haproxy-after
text data bss dec hex filename 7591126 569268 276134810921742 a6a70e haproxy-before 7591082 568180 276134810920610 a6a2a2 haproxy-after
Miroslav Zagorac [Mon, 19 Sep 2022 10:18:53 +0000 (12:18 +0200)]
CLEANUP: httpclient: deleted unused variables
The locally defined static variables 'httpclient_srv_raw' and
'httpclient_srv_ssl' are not used anywhere in the source code,
except that they are set in the httpclient_precheck() function.
nb_hreq is a counter on qcc for active HTTP requests. It is incremented
for each qcs where a full HTTP request was received. It is decremented
when the stream is closed locally :
- on HTTP response fully transmitted
- on stream reset
A bug will occur if a stream is resetted without having processed a full
HTTP request. nb_hreq will be decremented whereas it was not
incremented. This will lead to a crash when building with
DEBUG_STRICT=2. If BUG_ON_HOT are not active, nb_hreq counter will wrap
which may break the timeout logic for the connection.
This bug was triggered on haproxy.org. It can be reproduced by
simulating the reception of a STOP_SENDING frame instead of a STREAM one
by patching qc_handle_strm_frm() :
To fix this bug, a qcs is now flagged with a new QC_SF_HREQ_RECV. This
is set when the full HTTP request is received. When the stream is closed
locally, nb_hreq will be decremented only if this flag was set.
Released version 2.7-dev6 with the following main changes :
- MINOR: Revert part of clarifying samples support per os commit
- BUILD: makefile: enable crypt(3) for NetBSD
- BUG/MINOR: quic: Retransmitted frames marked as acknowledged
- BUG/MINOR: quic: Possible crash with "tls-ticket-keys" on QUIC bind lines
- MINOR: http-check: Remove support for headers/body in "option httpchk" version
- BUG/MINOR: h1: Support headers case adjustment for TCP proxies
- BUG/MINOR: quic: Possible crash when verifying certificates
- BUILD: quic: add some ifdef around the SSL_ERROR_* for libressl
- BUILD: ssl: fix ssl_sock_switchtx_cbk when no client_hello_cb
- BUILD: quic: temporarly ignore chacha20_poly1305 for libressl
- BUILD: quic: enable early data only with >= openssl 1.1.1
- BUILD: ssl: fix the ifdef mess in ssl_sock_initial_ctx
- BUILD: quic: fix the #ifdef in ssl_quic_initial_ctx()
- MINOR: quic: add QUIC support when no client_hello_cb
- MINOR: quic: Add traces about sent or resent TX frames
- MINOR: quic: No TRACE_LEAVE() in retrieve_qc_conn_from_cid()
- BUG/MINOR: quic: Wrong connection ID to thread ID association
- BUG/MINOR: task: always reset a new tasklet's call date
- BUG/MINOR: task: make task_instant_wakeup() work on a task not a tasklet
- MINOR: task: permanently enable latency measurement on tasklets
- CLEANUP: task: rename ->call_date to ->wake_date
- BUG/MINOR: sched: properly account for the CPU time of dying tasks
- MINOR: sched: store the current profile entry in the thread context
- BUG/MINOR: stream/sched: take into account CPU profiling for the last call
- MINOR: tasks: do not keep cpu and latency times in struct task
- MINOR: tools: add generic pointer hashing functions
- CLEANUP: activity: make memprof use the generic ptr_hash() function
- CLEANUP: activity: make taskprof use ptr_hash()
- MINOR: debug: add struct ha_caller to describe a calling location
- CLEANUP: debug: use struct ha_caller for memstat
- DEBUG: task: define a series of wakeup types for tasks and tasklets
- DEBUG: task: use struct ha_caller instead of arrays of file:line
- DEBUG: applet: instrument appctx_wakeup() to log the caller's location
- DEBUG: task: simplify the caller recording in DEBUG_TASK
- CLEANUP: task: move tid and wake_date into the common part
- CLEANUP: sched: remove duplicate code in run_tasks_from_list()
- CLEANUP: activity: make the number of sched activity entries more configurable
- DEBUG: resolvers: unstatify process_resolvers() to make it appear in profiling
- DEBUG: quic: export the few task handlers that often appear in task dumps
- MEDIUM: tasks/activity: combine the called function with the caller
- MINOR: tasks/activity: improve the caller-callee activity hash
- MINOR: activity/cli: support aggregating task profiling outputs
- MINOR: activity/cli: support sorting task profiling by total CPU time
- BUG/MINOR: signals/poller: set the poller timeout to 0 when there are signals
- BUG/MINOR: quic: Speed up the handshake completion only one time
- BUG/MINOR: quic: Trace fix about packet number space information.
- BUG/MINOR: h3: Crash when h3 trace verbosity is "minimal"
- MINOR: h3: Add the quic_conn object to h3 traces
- MINOR: h3: Missing connection argument for a TRACE_LEAVE() argument
- MINOR: h3: Send the h3 settings with others streams (requests)
- MINOR: dev/udp: Apply the corruption to both directions
- BUILD: udp-perturb: Add a make target for udp-perturb tool
- BUG/MINOR: signals/poller: ensure wakeup from signals
- CI: cirrus-ci: bump FreeBSD image to 13-1
- DEV: flags: fix usage message to reflect available options
- DEV: flags: add missing CO_FL_FDLESS connection flag
- MINOR: flags: add a new file to host flag dumping macros
- MINOR: flags: implement a macro used to dump enums inside masks
- MINOR: flags/channel: use flag dumping for channel flags and analysers
- MINOR: flags/connection: use flag dumping for connection flags
- MINOR: flags/stconn: use flag dumping for stconn and sedesc flags
- MINOR: flags/stream: use flag dumping for stream error type
- MINOR: flags/stream: use flag dumping for stream flags
- MINOR: flags/task: use flag dumping for task state
- MINOR: flags/http_ana: use flag dumping for txn flags
- DEV: flags: remove the now unused SHOW_FLAG() definition
- DEV: flags: remove the now useless intermediary functions
- MINOR: flags/htx: use flag dumping to show htx and start-line flags
- MINOR: flags/http_ana: use flag dumping to show http msg states
- BUG/MEDIUM: proxy: ensure pause_proxy() and resume_proxy() own PROXY_LOCK
- MINOR: listener: small API change
- MINOR: proxy/listener: support for additional PAUSED state
- BUG/MINOR: stats: fixing stat shows disabled frontend status as 'OPEN'
- BUILD: flags: fix build warning in some macros used by show_flags
- BUILD: flags: fix the fallback macros for missing stdio
- CLEANUP: pollers: remove dead code in the polling loop
- BUG/MINOR: mux-h1: Increment open_streams counter when H1 stream is created
- REGTESTS: healthcheckmail: Relax matching on the healthcheck log message
- CLEANUP: listener: function comment typo in stop_listener()
- BUG/MINOR: listener: null pointer dereference suspected by coverity
- MINOR: flags/fd: decode FD flags states
- REORG: mux-h2: extract flags and enums into mux_h2-t.h
- MINOR: flags/mux-h2: decode H2C and H2S flags
- REGTESTS: log: test the log-forward feature
- BUG/MEDIUM: sink: bad init sequence on tcp sink from a ring.
- REGTESTS: ssl/log: test the log-forward with SSL
- MEDIUM: httpclient: httpclient_create_proxy() creates a proxy for httpclient
- MEDIUM: httpclient: allow to use another proxy
- DOC: fix TOC in starter guide for subsection 3.3.8. Statistics
- MINOR: httpclient: export httpclient_create_proxy()
- MEDIUM: quic: separate path for rx and tx with set_encryption_secrets
- BUG/MEDIUM: mux-quic: fix crash on early app-ops release
- REORG: mux-h1: extract flags and enums into mux_h1-t.h
- MINOR: flags/mux-h1: decode H1C and H1S flags
- CLEANUP: mux-quic: remove stconn usage in h3/hq
- BUG/MINOR: mux-quic: do not remotely close stream too early
- CLEANUP: exclude udp-perturb with .gitignore
- BUG/MEDIUM: server: segv when adding server with hostname from CLI
- CLEANUP: quic,ssl: fix tiny typos in C comments
- BUG/MEDIUM: captures: free() an error capture out of the proxy lock
- BUILD: fd: fix a build warning on the DWCAS
- MINOR: anon: add new macros and functions to anonymize contents
- MINOR: anon: store the anonymizing key in the global structure
- MINOR: anon: store the anonymizing key in the CLI's appctx
- MINOR: cli: anonymize commands 'show sess' and 'show sess all'
- MINOR: cli: anonymize 'show servers state' and 'show servers conn'
- MINOR: config: add command-line -dC to dump the configuration file
- SCRIPTS: announce-release: update some URLs to https
SCRIPTS: announce-release: update some URLs to https
Some components like Discourse were already redirecting to https. Other
ones like docs and git are covered by the certificate, and finally
switching the advertised scheme for www should increase the ratio of
H2 and H3 in the stats (resp 8.9 and 1.9%) and possibly help spot new
issues.
MINOR: config: add command-line -dC to dump the configuration file
This commit adds a new command line option -dC to dump the configuration
file. An optional key may be appended to -dC in order to produce an
anonymized dump using this key. The anonymizing process uses the same
algorithm as the CLI so that the same key will produce the same hashes
for the same identifiers. This way an admin may share an anonymized
extract of a configuration to match against live dumps. Note that key 0
will not anonymize the output. However, in any case, the configuration
is dumped after tokenizing, thus comments are lost.
MINOR: cli: anonymize 'show servers state' and 'show servers conn'
Modify proxy.c in order to anonymize the following confidential data on
commands 'show servers state' and 'show servers conn':
- proxy name
- server name
- server address
MINOR: cli: anonymize commands 'show sess' and 'show sess all'
Modify stream.c in order to hash the following confidential data if the
anonymized mode is enabled:
- configuration elements such as frontend/backend/server names
- IP addresses
MINOR: anon: store the anonymizing key in the CLI's appctx
In order to allow users to dump internal states using a specific key
without changing the global one, we're introducing a key in the CLI's
appctx. This key is preloaded from the global one when "set anon on"
is used (and if none exists, a random one is assigned). And the key
can optionally be assigned manually for the whole CLI session.
A "show anon" command was also added to show the anon state, and the
current key if the users has sufficient permissions. In addition, a
"debug dev hash" command was added to test the feature.
MINOR: anon: store the anonymizing key in the global structure
Add a uint32_t key in global to hash words with it. A new CLI command
'set global-key <key>' was added to change the global anonymizing key.
The global may also be set in the configuration using the global
"anonkey" directive. For now this key is not used.
MINOR: anon: add new macros and functions to anonymize contents
These macros and functions will be used to anonymize strings by producing
a short hash. This will allow to match config elements against dump elements
without revealing the original data. This will later be used to anonymize
configuration parts and CLI commands output. For now only string, identifiers
and addresses are supported, but the model is easily extensible.
Ilya reported in issue #1816 a build warning on armhf (promoted to error
here since -Werror):
src/fd.c: In function fd_rm_from_fd_list:
src/fd.c:209:87: error: passing argument 3 of __ha_cas_dw discards volatile qualifier from pointer target type [-Werror=discarded-array-qualifiers]
209 | unlikely(!_HA_ATOMIC_DWCAS(((long *)&fdtab[fd].update), (uint32_t *)&cur_list.u32, &next_list.u32))
| ^~~~~~~~~~~~~~
This happens only on such an architecture because the DWCAS requires the
pointer not the value, and gcc seems to be needlessly picky about reading
a const from a volatile! This may safely be backported to older versions.
BUG/MEDIUM: captures: free() an error capture out of the proxy lock
Ed Hein reported in github issue #1856 some occasional watchdog panics
in 2.4.18 showing extreme contention on the proxy's lock while the libc
was in malloc()/free(). One cause of this problem is that we call free()
under the proxy's lock in proxy_capture_error(), which makes no sense
since if we can free the object under the lock after it's been detached,
we can also free it after releasing the lock (since it's not referenced
anymore).
This should be backported to all relevant versions, likely all
supported ones.
BUG/MEDIUM: server: segv when adding server with hostname from CLI
When calling 'add server' with a hostname from the cli (runtime),
str2sa_range() does not resolve hostname because it is purposely
called without PA_O_RESOLVE flag.
This leads to 'srv->addr_node.key' being NULL. According to Willy it
is fine behavior, as long as we handle it properly, and is already
handled like this in srv_set_addr_desc().
This patch fixes GH #1865 by adding an extra check before inserting
'srv->addr_node' into 'be->used_server_addr'. Insertion and removal
will be skipped if 'addr_node.key' is NULL.
udp-perturb is a tool which can be used as a UDP gateway. It can be used
to reorder, remove or corrupt datagrams. It is compiled in dev/
directory and added to .gitignore to not clutter git status output.
BUG/MINOR: mux-quic: do not remotely close stream too early
A stream is considered as remotely closed once we have received all the
data with the FIN bit set.
The condition to close the stream was wrong. In particular, if we
receive an empty STREAM frame with FIN bit set, this would have close
the stream even if we do not have yet received all the data. The
condition is now adjusted to ensure that Rx buffer contains all the data
up to the stream final size.
In most cases, this bug is harmless. However, if compiled with
DEBUG_STRICT=2, a BUG_ON_HOT crash would have been triggered if close is
done too early. This was most notably the case sometimes on interop test
suite with quinn or kwik clients. This can also be artificially
reproduced by simulating reception of an empty STREAM frame with FIN bit
set in qc_handle_strm_frm() :
Small cleanup on snd_buf for application protocol layer.
* do not export h3_snd_buf
* replace stconn by a qcs argument. This is better as h3/hq-interop only
uses the qcs instance.
REORG: mux-h1: extract flags and enums into mux_h1-t.h
The same was performed for the H2 multiplexer. H1C and H1S flags are moved
in a dedicated header file. It will be mainly used to be able to decode
mux-h1 flags from the flags utility.
In this patch, we only move the flags to mux_h1-t.h.
BUG/MEDIUM: mux-quic: fix crash on early app-ops release
H3 SETTINGS emission has recently been delayed. The idea is to send it
with the first STREAM to reduce sendto syscall invocation. This was
implemented in the following patch : 3dd79d378c86b3ebf60e029f518add5f1ed54815
MINOR: h3: Send the h3 settings with others streams (requests)
This patch works fine under nominal conditions. However, it will cause a
crash if a HTTP/3 connection is released before having sent any data,
for example when receiving an invalid first request. In this case,
qc_release will first free qcc.app_ops HTTP/3 application protocol layer
via release callback. Then qc_send is called to emit any closing frames
built by app_ops release invocation. However, in qc_send, as no data has
been sent, it will try to complete application layer protocol
intialization, with a SETTINGS emission for HTTP/3. Thus, qcc.app_ops is
reused, which is invalid as it has been just freed. This will cause a
crash with h3_finalize in the call stack.
This bug can be reproduced artificially by generating incomplete HTTP/3
requests. This will in time trigger http-request timeout without any
data send. This is done by editing qc_handle_strm_frm function.
- ret = qcc_recv(qc->qcc, strm_frm->id, strm_frm->len,
+ ret = qcc_recv(qc->qcc, strm_frm->id, strm_frm->len - 1,
strm_frm->offset.key, strm_frm->fin,
(char *)strm_frm->data);
To fix this, application layer closing API has been adjusted to be done
in two-steps. A new shutdown callback is implemented : it is used by the
HTTP/3 layer to generate GOAWAY frame in qc_release prologue.
Application layer context qcc.app_ops is then freed later in qc_release
via the release operation which is now only used to liberate app layer
ressources. This fixes the problem as the intermediary qc_send
invocation will be able to reuse app_ops before it is freed.
This patch fixes the crash, but it would be better to adjust H3 SETTINGS
emission in case of early connection closing : in this case, there is no
need to send it. This should be implemented in a future patch.
This should fix the crash recently experienced by Tristan in github
issue #1801.
MEDIUM: quic: separate path for rx and tx with set_encryption_secrets
With quicTLS the set_encruption_secrets callback is always called with
the read_secret and the write_secret.
However this is not the case with libreSSL, which uses the
set_read_secret()/set_write_secret() mecanism. It still provides the
set_encryption_secrets() callback, which is called with a NULL
parameter for the write_secret during the read, and for the read_secret
during the write.
The exchange key was not designed in haproxy to be called separately for
read and write, so this patch allow calls with read or write key to
NULL.
DOC: fix TOC in starter guide for subsection 3.3.8. Statistics
This subsection has been moved from 3.4.9 to 3.3.8 somewhere along
2.4, but the TOC has not been updated - resulting in a invalid
anchor in the HTML version.
MEDIUM: httpclient: httpclient_create_proxy() creates a proxy for httpclient
httpclient_create_proxy() is a function which creates a proxy that could
be used for the httpclient. It will allocate a proxy, a raw server and
an ssl server.
This patch moves most of the code from httpclient_precheck() into a
generic function httpclient_create_proxy().
The proxy will have the PR_CAP_HTTPCLIENT capability.
This could be used for specifics httpclient instances that needs
different proxy settings.
BUG/MEDIUM: sink: bad init sequence on tcp sink from a ring.
The init of tcp sink, particularly for SSL, was done
too early in the code, during parsing, and this can cause
a crash specially if nbthread was not configured.
This was detected by William using ASAN on a new regtest
on log forward.
This patch adds the 'struct proxy' created for a sink
to a list and this list is now submitted to the same init
code than the main proxies list or the log_forward's proxies
list. Doing this, we are assured to use the right init sequence.
It also removes the ini code for ssl from post section parsing.
This patch should be backported as far as v2.2
Note: this fix uses 'goto' labels created by commit
'BUG/MAJOR: log-forward: Fix log-forward proxies not fully initialized'
but this code didn't exist before v2.3 so this patch needs to be
adapted for v2.2.
REORG: mux-h2: extract flags and enums into mux_h2-t.h
Originally in 1.8 we wanted to have an independent mux that could possibly
be disabled and would not impose dependencies on the outside. Everything
would fit into a single C file and that was fine.
Nowadays muxes are unavoidable, and not being able to easily inspect them
from outside is sometimes a bit of a pain. In particular, the flags utility
still cannot be used to decode their flags.
As a first step towards this, this patch moves the flags and enums to
mux_h2-t.h, as well as the two state decoding inline functions. It also
dropped the H2_SS_*_BIT defines that nobody uses. The mux_h2.c file remains
the only one to include that for now.
BUG/MINOR: listener: null pointer dereference suspected by coverity
Please refer to GH #1859 for more info.
Coverity suspected improper proxy pointer handling.
Without the fix it is considered safe for the moment, but it might not
be the case in the future as we want to keep the ability to have
isolated listeners.
Making sure stop_listener(), pause_listener(), resume_listener()
and listener_release() functions make proper use
of px pointer in that context.
No need for backport except if multi-connection protocols (ie:FTP)
were to be backported as well.
REGTESTS: healthcheckmail: Relax matching on the healthcheck log message
Depending on the timing, the conneciton on lisrv listener may be fully
accepted before any reject. Thus, instead of getting a socket error, an
invalid L7 response is reported. There is no reason to be strick on the
error type. Any failure is good here, because we just want to test the
email-alert feature.
This patch should fix issue #1857. It may be backported as far as 2.2.
BUG/MINOR: mux-h1: Increment open_streams counter when H1 stream is created
Since this counter was added, it was incremented at the wrong place for
client streams. It was incremented when the stream-connector (formely the
conn-stream) was created while it should be done when the H1 stream is
created. Thus, on parsing error, on H1>H2 upgrades or TCP>H1 upgrades, the
counter is not incremented. However, it is always decremented when the H1
stream is destroyed.
CLEANUP: pollers: remove dead code in the polling loop
As reported by Ilya and Coverity in issue #1858, since recent commit eea152ee6 ("BUG/MINOR: signals/poller: ensure wakeup from signals")
which removed the test for the global signal flag from the pollers'
loop, the remaining "wake" flag doesn't need to be tested since it
already participates to zeroing the wait_time and will be caught
on the previous line.
BUILD: flags: fix the fallback macros for missing stdio
The fallback macros for when stdio is not there didn't have the "..."
and were causing build issues on platforms with stricter dependencies
between includes.
BUG/MINOR: stats: fixing stat shows disabled frontend status as 'OPEN'
This patch adresses the issue #1626.
Adding support for PR_FL_PAUSED flag in the function stats_fill_fe_stats().
The command 'show stat' now properly reports a disabled frontend
using "PAUSED" state label.
This patch depends on the following commits:
- 7d00077fd5 "BUG/MEDIUM: proxy: ensure pause_proxy()
and resume_proxy() own PROXY_LOCK".
- 001328873c "MINOR: listener: small API change"
- d46f437de6 "MINOR: proxy/listener: support for additional PAUSED state"
MINOR: proxy/listener: support for additional PAUSED state
This patch is a prerequisite for #1626.
Adding PAUSED state to the list of available proxy states.
The flag is set when the proxy is paused at runtime (pause_listener()).
It is cleared when the proxy is resumed (resume_listener()).
BUG/MEDIUM: proxy: ensure pause_proxy() and resume_proxy() own PROXY_LOCK
There was a race involving hlua_proxy_* functions
and some proxy management functions.
pause_proxy() and resume_proxy() can be used directly from lua code,
but that could lead to some race as lua code didn't make sure PROXY_LOCK
was owned before calling the proxy functions.
This patch makes sure it won't happen again elsewhere in the code
by locking PROXY_LOCK directly in resume and pause proxy functions
so that it's not the caller's responsibility anymore.
(based on stop_proxy() behavior that was already safe prior to the patch)
This should be backported to stable series.
Note that the API will likely differ < 2.4
projects / thirdparty / haproxy.git / log
