Willy Tarreau [Wed, 3 May 2023 09:22:45 +0000 (11:22 +0200)]
DEBUG: cli: add "debug dev task" to show/wake/expire/kill tasks and tasklets
When analyzing certain types of bugs in field, sometimes it would be
nice to be able to wake up a task or tasklet to see how events progress
(e.g. to detect a missing wakeup condition), or expire or kill such a
task. This restricted command shows hte current state of a task or tasklet
and allows to manipulate it like this. However it must be used with extreme
care because while it does verify that the pointers are mapped, it cannot
know if they point to a real task, and performing such actions on something
not a task will easily lead to a crash. In addition, performing a "kill"
on a task has great chances of provoking a deferred crash due to a double
free and/or another kill that is not idempotent. Use with extreme care!
Willy Tarreau [Wed, 3 May 2023 09:29:54 +0000 (11:29 +0200)]
BUG/MINOR: stream/cli: fix stream age calculation in "show sess"
The "show sess" command displays the stream's age in synthetic form,
and also makes it appear in the long version (show sess all). But that
last one uses the wrong origin, it uses accept_date.tv_sec instead of
accept_ts (formerly known as tv_accept). This was introduced in 1.4.2
with the long format, with commit 66dc20a17 ("[MINOR] stats socket: add
show sess <id> to dump details about a session"), while the code that
split the two variables was introduced in 1.3.16 with commit b7f694f20
("[MEDIUM] implement a monotonic internal clock"). This problem was
revealed by recent change ad5a5f677 ("MEDIUM: tree-wide: replace timeval
with nanoseconds in tv_accept and tv_request") that made this value report
random garbage, and generally emphasized by the fact that in 2.8 the two
clocks have sufficiently large an offset for such mistakes to be noticeable
early.
Arguably a difference between date and accept_date could also make sense,
to indicate if the stream had been there for more than 49 days, but this
would introduce instabilities for most sockets (including negative times)
for extremely rare cases while the goal is essentially to see how much
longer than a configured timeout a stream has been there. And that's what
other locations (including the short form) provide.
This patch could be backported but most users will never notice. In case
of backport, tv_accept.tv_sec should be used instead of accept_date.tv_sec.
MINOR: ssl: disable CRL checks with WolfSSL when no CRL file
WolfSSL is enabling by default the CRL checks even if a CRL file wasn't
provided. This patch resets the default X509_STORE flags so this is
not checked by default.
As this feature has a dependency on resolvers being configured,
this test acts as good documentation as well.
This change also has a spelling fix for filename.
Released version 2.8-dev9 with the following main changes :
- MINOR: quic: Move traces at proto level
- BUG/MINOR: quic: Possible memory leak from TX packets
- BUG/MINOR: quic: Possible leak during probing retransmissions
- BUG/MINOR: quic: Useless probing retransmission in draining or killing state
- BUG/MINOR: quic: Useless I/O handler task wakeups (draining, killing state)
- CLEANUP: quic: rename frame types with an explicit prefix
- CLEANUP: quic: rename frame variables
- CLEANUP: quic: Remove useless parameters passes to qc_purge_tx_buf()
- CLEANUP: quic: Rename <buf> variable to <token> in quic_generate_retry_token()
- CLEANUP: quic: Rename <buf> variable into quic_padding_check()
- CLEANUP: quic: Rename <buf> variable into quic_rx_pkt_parse()
- CLEANUP: quic: Rename <buf> variable for several low level functions
- CLEANUP: quic: Make qc_build_pkt() be more readable
- CLEANUP: quic: Rename quic_get_dgram_dcid() <buf> variable
- CLEANUP: quic: Rename several <buf> variables at low level
- CLEANUP: quic: Rename <buf> variable into quic_packet_read_long_header()
- CLEANUP: quic: Rename <buf> variable into qc_parse_hd_form()
- CLEANUP: quic: Rename several <buf> variables into quic_sock.c
- DEBUG: crash using an invalid opcode on x86/x86_64 instead of an invalid access
- DEBUG: crash using an invalid opcode on aarch64 instead of an invalid access
- DEV: h2: add a script "mkhdr" to build h2 frames from scratch
- DEV: h2: support reading frame payload from a file
- MINOR: acme.sh: add the deploy script for acme.sh in admin directory
- BUG/MEDIUM: mux-quic: do not emit RESET_STREAM for unknown length
- BUG/MEDIUM: mux-quic: improve streams fairness to prevent early timeout
- BUG/MINOR: quic: prevent buggy memcpy for empty STREAM
- MINOR: mux-quic: do not set buffer for empty STREAM frame
- MINOR: mux-quic: do not allocate Tx buf for empty STREAM frame
- MINOR: quic: finalize affinity change as soon as possible
- BUG/MINOR: quic: fix race on quic_conns list during affinity rebind
- CI: switch to Fastly CDN to download LibreSSL
- BUILD: ssl: switch LibreSSL to Fastly CDN
- BUG/MINOR: clock: fix a few occurrences of 'now' being used in place of 'date'
- BUG/MINOR: spoe: use "date" not "now" in debug messages
- BUG/MINOR: activity: show wall-clock date, not internal date in show activity
- BUG/MINOR: opentracing: use 'date' instead of 'now' in debug output
- Revert "BUG/MINOR: clock: fix a few occurrences of 'now' being used in place of 'date'"
- BUG/MINOR: calltrace: fix 'now' being used in place of 'date'
- BUG/MINOR: trace: show wall-clock date, not internal date in show activity
- BUG/MINOR: hlua: return wall-clock date, not internal date in core.now()
- BUG/MEDIUM: spoe: Don't start new applet if there are enough idle ones
- BUG/MINOR: stconn: Fix SC flags with same value
- BUG/MINOR: resolvers: Use sc_need_room() to wait more room when dumping stats
- BUG/MEDIUM: tcpcheck: Don't eval custom expect rule on an empty buffer
- BUG/MINOR: stats: report the correct start date in "show info"
- MINOR: time: add conversions to/from nanosecond timestamps
- MINOR: time: replace calls to tv_ms_elapsed() with a linear subtract
- MINOR: spoe: switch the timeval-based timestamps to nanosecond timestamps
- MEDIUM: tree-wide: replace timeval with nanoseconds in tv_accept and tv_request
- MINOR: stats: use nanoseconds, not timeval to compute uptime
- MINOR: activity: use nanoseconds, not timeval to compute uptime
- MINOR: checks: use a nanosecond counters instead of timeval for checks->start
- MINOR: clock: do not use now.tv_sec anymore
- MEDIUM: clock: replace timeval "now" with integer "now_ns"
- MINOR: clock: replace the timeval start_time with start_time_ns
- MINOR: sample: Add bc_rtt and bc_rttvar
- MINOR: quic: use real sending rate measurement
- MINOR: proxy: factorize send rate measurement
Before this patch, global sending rate was measured on the QUIC lower
layer just after sendto(). This meant that all QUIC frames were
accounted for, including non STREAM frames and also retransmission.
To have a better reflection of the application data transferred, move
the incrementation into the MUX layer. This allows to account only for
STREAM frames payload on their first emission.
MINOR: clock: replace the timeval start_time with start_time_ns
Now that "now" is no more a timeval, there's no point keeping a copy
of it as a timeval, let's also switch start_time to nanoseconds, it
simplifies operations.
MEDIUM: clock: replace timeval "now" with integer "now_ns"
This puts an end to the occasional confusion between the "now" date
that is internal, monotonic and not synchronized with the system's
date, and "date" which is the system's date and not necessarily
monotonic. Variable "now" was removed and replaced with a 64-bit
integer "now_ns" which is a counter of nanoseconds. It wraps every
585 years, so if all goes well (i.e. if humanity does not need
haproxy anymore in 500 years), it will just never wrap. This implies
that now_ns is never nul and that the zero value can reliably be used
as "not set yet" for a timestamp if needed. This will also simplify
date checks where it becomes possible again to do "date1<date2".
All occurrences of "tv_to_ns(&now)" were simply replaced by "now_ns".
Due to the intricacies between now, global_now and now_offset, all 3
had to be turned to nanoseconds at once. It's not a problem since all
of them were solely used in 3 functions in clock.c, but they make the
patch look bigger than it really is.
The clock_update_local_date() and clock_update_global_date() functions
are now much simpler as there's no need anymore to perform conversions
nor to round the timeval up or down.
The wrapping continues to happen by presetting the internal offset in
the short future so that the 32-bit now_ms continues to wrap 20 seconds
after boot.
The start_time used to calculate uptime can still be turned to
nanoseconds now. One interrogation concerns global_now_ms which is used
only for the freq counters. It's unclear whether there's more value in
using two variables that need to be synchronized sequentially like today
or to just use global_now_ns divided by 1 million. Both approaches will
work equally well on modern systems, the difference might come from
smaller ones. Better not change anyhting for now.
One benefit of the new approach is that we now have an internal date
with a resolution of the nanosecond and the precision of the microsecond,
which can be useful to extend some measurements given that timestamps
also have this resolution.
Instead we're using ns_to_sec(tv_to_ns(&now)) which allows the tv_sec
part to disappear. At this point, "now" is only used as a timeval in
clock.c where it is updated.
MEDIUM: tree-wide: replace timeval with nanoseconds in tv_accept and tv_request
Let's get rid of timeval in storage of internal timestamps so that they
are no longer mistaken for wall clock time. These were exclusively used
subtracted from each other or to/from "now" after being converted to ns,
so this patch removes the tv_to_ns() conversion to use them natively. Two
occurrences of tv_isge() were turned to a regular wrapping subtract.
MINOR: time: replace calls to tv_ms_elapsed() with a linear subtract
Instead of operating on {sec, usec} now we convert both operands to
ns then subtract them and convert to ms. This is a first step towards
dropping timeval from these timestamps.
Interestingly, tv_ms_elapsed() and tv_ms_remain() are no longer used at
all and could be removed.
In order to ease the transition away from the timeval used in internal
timestamps, let's first create a few functions and macro to return a
counter from a timeval and conversely, as well as ease the conversions
to/from ns/us/ms/sec to save the user from having to count zeroes and
to think about appending ULL in conversions.
BUG/MINOR: stats: report the correct start date in "show info"
The "show info" help for "Start_time_sec" says "Start time in seconds"
so it's definitely the start date in human format, not the internal one
that is solely used to compute uptime. Since commit 28360dc ("MEDIUM:
clock: force internal time to wrap early after boot"), both are split
apart since the start time takes into account the offset needed to cause
the early wraparound, so we must only use start_date here.
BUG/MEDIUM: tcpcheck: Don't eval custom expect rule on an empty buffer
The commit a664aa6a6 ("BUG/MINOR: tcpcheck: Be able to expect an empty
response") instroduced a regression for expect rules relying on a custom
function. Indeed, there is no check on the buffer to be sure it is not empty
before calling the custom function. But some of these functions expect to
have data and don't perform any test on the buffer emptiness.
So instead of fixing all custom functions, we just don't eval them if the
buffer is empty.
This patch must be backported but only if the commit above was backported
first.
SC_FL_SND_NEVERWAIT and SC_FL_SND_EXP_MORE flags have the same value. It is
not critical because these flags are only used to know if MSG_MORE flag must
be set on a send().
BUG/MEDIUM: spoe: Don't start new applet if there are enough idle ones
It is possible to start too many applets on sporadic burst of events after
an inactivity period. It is due to the way we estimate if a new applet must
be created or not. It is based on a frequency counter. We compare the events
processing rate against the number of events currently processed (in
progress or waiting to be processed). But we should also take care of the
number of idle applets.
We already track the number of idle applets, but it is global and not
per-thread. Thus we now also track the number of idle applets per-thread. It
is not a big deal because this fills a hole in the spoe_agent structure.
Thanks to this counter, we can refrain applets creation if there is enough
idle applets to handle currently processed events.
This patch should be backported to every stable versions.
BUG/MINOR: hlua: return wall-clock date, not internal date in core.now()
That's hopefully the last one affected by this. It was a bit trickier
because there's the promise in the doc that the date is monotonous, so
we continue to use now-start_time as the uptime value and add it to
start_date to get the current date. It was also emphasized by commit 28360dc ("MEDIUM: clock: force internal time to wrap early after boot"),
causing core.now() to return a date of Mar 20 on Apr 27. No backport is
needed.
BUG/MINOR: trace: show wall-clock date, not internal date in show activity
Yet another case where "now" was used instead of "date" for a publicly
visible date that was already incorrect and became worse after commit 28360dc ("MEDIUM: clock: force internal time to wrap early after boot").
No backport is needed.
BUG/MINOR: calltrace: fix 'now' being used in place of 'date'
Since commit 28360dc ("MEDIUM: clock: force internal time to wrap early
after boot") we have a much clearer distinction between 'now' (the internal,
drifting clock) and 'date' (the wall clock time). The calltrace code was
using "now" instead of "date" since the value is displayed to humans.
The parts affecting the DeviceAtlas addon were wrong actually, the
"now" variable was a local time_t in a file that's not compiled with
the haproxy binary (dadwsch). Only the fix to the calltrace is correct,
so better revert and fix the only one in a separate commit. No backport
is needed.
BUG/MINOR: opentracing: use 'date' instead of 'now' in debug output
The filter was using "now" in visible output in debug mode, that's
not correct, we should rather use "date" since it's visible. No
backport is needed as it was mostly emphasized by commit 28360dc
("MEDIUM: clock: force internal time to wrap early after boot")
in 2.8..
BUG/MINOR: activity: show wall-clock date, not internal date in show activity
Another case where "now" was used instead of "date" for a publicly visible
date that was already incorrect and became worse after commit 28360dc
("MEDIUM: clock: force internal time to wrap early after boot"). No
backport is needed.
BUG/MINOR: spoe: use "date" not "now" in debug messages
The debug messages were still emitted with a date taken from "now" instead
of "date", which was not correct a long time ago but which became worse in
2.8 since commit 28360dc ("MEDIUM: clock: force internal time to wrap early
after boot"). Let's fix it. No backport is needed.
BUG/MINOR: clock: fix a few occurrences of 'now' being used in place of 'date'
Since commit 28360dc ("MEDIUM: clock: force internal time to wrap early
after boot") we have a much clearer distinction between 'now' (the internal,
drifting clock) and 'date' (the wall clock time). There were still a few
places where 'now' was being used for human consumption.
BUG/MINOR: quic: fix race on quic_conns list during affinity rebind
Each quic_conn are attached in a global thread-local quic_conns list
used for "show quic" command. During thread rebinding, a connection is
detached from its local list instance and moved to its new thread list.
However this operation is not thread-safe and may cause a race
condition.
To fix this, only remove the connection from its list inside
qc_set_tid_affinity(). The connection is inserted only after in
qc_finalize_affinity_rebind() on the new thread instance thus prevented
a race condition. One impact of this is that a connection will be
invisible during rebinding for "show quic".
A connection must not transition to closing state in between this two
steps or else cleanup via quic_handle_stopping() may not miss it. To
ensure this, this patch relies on the previous commit :
commit d6646dddccb1aae08f60717b5b6743c513c37299
MINOR: quic: finalize affinity change as soon as possible
MINOR: quic: finalize affinity change as soon as possible
During accept, a quic-conn is rebind to a new thread. This process is
done in two times :
* first on the original thread via qc_set_tid_affinity()
* then on the newly assigned thread via qc_finalize_affinity_rebind()
Most quic_conn operations (I/O tasklet, task and quic_conn FD socket
read) are reactivated ony after the second step. However, there is a
possibility that datagrams are handled before it via quic_dgram_parse()
when using listener sockets. This does not seem to cause any issue but
this may cause unexpected behavior in the future.
To simplify this, qc_finalize_affinity_rebind() will be called both by
qc_xprt_start() and quic_dgram_parse(). Only one invocation will be
performed thanks to the new flag QUIC_FL_CONN_AFFINITY_CHANGED.
MINOR: mux-quic: do not allocate Tx buf for empty STREAM frame
Sometimes it may be necessary to send an empty STREAM frame to signal
clean stream closure with FIN bit set. Prior to this change, a Tx buffer
was allocated unconditionnally even if no data is transferred.
Most of the times, allocation was not performed due to an older buffer
reused. But if data were already acknowledge, a new buffer is allocated.
No memory leak occurs as the buffer is properly released when the empty
frame acknowledge is received. But this allocation is unnecessary and it
consumes a connexion Tx buffer for nothing.
Improve this by skipping buffer allocation if no data to transfer.
qcs_build_stream_frm() is now able to deal with a NULL out argument.
MINOR: mux-quic: do not set buffer for empty STREAM frame
Previous patch fixes an issue occurring with empty STREAM frames without
payload. The crash was hidden in part because buf/data fields of
qf_stream were set even if no payload is referenced. This was not the
true cause of the crash but to ease future debugging, a STREAM frame
built with no payload now has its buf and data fields set to NULL.
BUG/MINOR: quic: prevent buggy memcpy for empty STREAM
Sometimes it may be necessary to send empty STREAM frames with only the
FIN bit set. For these frames, memcpy is thus unnecessary as their
payload is empty. However, we did not prevent its invocation inside
quic_build_stream_frame().
Normally, memcpy invocation with length==0 is safe. However, there is an
extra condition in our function to handle data wrapping. For an empty
STREAM frame in the context of MUX emission, this is safe as the frame
points to a valid buffer which causes the wrapping condition to be false
and resulting in a memcpy with 0 length.
However, in the context of retransmission, this may lead to a crash.
Consider the following scenario : two STREAM frames A and B are
produced, one with payload and one empty with FIN set, pointing to the
same stream_desc buffer. If A is acknowledged by the peer, its buffer is
released as no more data is left in it. If B needs to be resent, the
wrapping condition will be messed up to a reuse of a freed buffer. Most
of the times, <wrap> will be a negative number, which results in a
memcpy invocation causing a buffer overflow.
To fix this, simply add an extra condition to skip memcpy and wrapping
check if STREAM frame length is null inside quic_build_stream_frame().
This crash is pretty rare as it relies on a lot of conditions difficult
to reproduce. It seems to be the cause for the latest crashes reported
under github issue #2120. In all the inspected dumps, the segfault
occurred during retransmission with an empty STREAM frame being used as
input. Thanks again to Tristan from Mangadex for his help and
investigation on it.
BUG/MEDIUM: mux-quic: improve streams fairness to prevent early timeout
Since the following mentioned patch, a send-list mechanism was
implemented to improve streams priorization on sending.
commit 20f2a425ffeda2e623aac4c702f4e44b1e122d1d
MAJOR: mux-quic: rework stream sending priorization
This is done to prevent the same streams to always be used as first ones
on emission. However there is still a flaw on the algorithm. Once put in
the send-list, a streams is not removed until it has sent all of its
content. When a stream transfers a large object, it will remain in the
send-list during all the transfer and will soon monopolize the first
place. the stream does never leave its position until the transfer is
finished and will monopolize the first place. Other streams behind won't
have the opportunity to advance on their own transfers due to a Tx
buffer exhaustion.
This situation is especially problematic if a small timeout client is
used. As some streams won't advance on their transfer for a long period
of time, they will be aborted due to a stream layer timeout client
causing a RESET_STREAM emission.
To fix this, during sending each stream with at least some bytes
transferred from its tx.buf to qc_stream_desc out buffer is put at the
end of the send-list. This ensures that on the next iteration streams
that cannot transfer anything will be used in priority.
This patch improves significantly h2load benchmarks for large objects
with several streams opened in parallel on a single connection. Without
it, errors may be reported by h2load for aborted streams. For example,
this improved the following scenario on a 10mbit/s link with a 10s
timeout client :
$ ./build/bin/h2load --npn-list h3 -t 1 -c 1 -m 30 -n 30 https://198.18.10.11:20443/?s=500k
This fix may help with the github issue #2004 where chrome browser stop
to use QUIC after receiving RESET_STREAM frames.
BUG/MEDIUM: mux-quic: do not emit RESET_STREAM for unknown length
Some HTX responses may not always contain a EOM block. For example this
is the case if content-length header is missing from the HTTP server
response. Stream termination is thus signaled to QUIC mux via shutw
callback. However, this is interpreted inconditionnally as an early
close by the mux with a RESET_STREAM emission. Most of the times, QUIC
clients report this as an error.
To fix this, check if htx.extra is set to HTX_UNKOWN_PAYLOAD_LENGTH for
a qcs instance. If true, shutw will never be used to emit a
RESET_STREAM. Instead, the stream will be closed properly with a FIN
STREAM frame. If all data were already transfered, an empty STREAM frame
is sent.
This fix may help with the github issue #2004 where chrome browser stop
to use QUIC after receiving RESET_STREAM frames.
This issue was reported by Vladimir Zakharychev. Thanks to him for his
help and testing. It was also reproduced locally using httpterm with the
query string "/?s=1k&b=0&C=1".
DEV: h2: support reading frame payload from a file
Now we can build a series of data frames by reading from a file and
chunking it into frames of requested length. It's mostly useful for
data frames (e.g. post). One way to announce these upfront is to
capture the output of curl use without content-length:
Then just change the 5th byte from the end from 1 to 0 to remove the
end-of-stream bit, it will allow to chain a file, then to send an
empty DATA frame with ES set :
$ (dev/h2/mkhdr.sh -i 1 -t data -d CHANGELOG;
dev/h2/mkhdr.sh -i 1 -t data -l 0 -f es) > h2-data-changelog.bin
Then post that to the server:
$ cat post-h2-nocl.bin h2-data-changelog.bin | nc 0 4446
It's conveniently encoded as 4 zeroes (when the operand is zero). It's
unclear when support for it was added into GAS, if at all; even a
not-so-old 2.27 doesn't know about it. Let's byte-encode it.
DEBUG: crash using an invalid opcode on x86/x86_64 instead of an invalid access
BUG_ON() calls currently trigger a segfault. This is more convenient
than abort() as it doesn't rely on any function call nor signal handler
and never causes non-unwindable stacks when opening cores. But it adds
quite some confusion in bug reports which are rightfully tagged "segv"
and do not instantly allow to distinguish real segv (e.g. null derefs)
from code asserts.
Some CPU architectures offer various crashing methods. On x86 we have
INT3 (0xCC), which stops into the debugger, and UD0/UD1/UD2. INT3 looks
appealing but for whatever reason (maybe signal handling somewhere) it
loses the last call point in the stack, making backtraces unusable. UD2
has the merit of being only 2 bytes and causing an invalid instruction,
which almost never happens normally, so it's easily distinguishable.
Here it was defined as a macro so that the line number in the core
matches the one where the BUG_ON() macro is called, and the debugger
shows the last frame exactly at its calligg point.
E.g. when calling "debug dev bug":
Program terminated with signal SIGILL, Illegal instruction.
#0 debug_parse_cli_bug (args=<optimized out>, payload=<optimized out>, appctx=<optimized out>, private=<optimized out>) at src/debug.c:408
408 BUG_ON(one > zero);
[Current thread is 1 (Thread 0x7f7a660cc1c0 (LWP 14238))]
(gdb) bt
#0 debug_parse_cli_bug (args=<optimized out>, payload=<optimized out>, appctx=<optimized out>, private=<optimized out>) at src/debug.c:408
#1 debug_parse_cli_bug (args=<optimized out>, payload=<optimized out>, appctx=<optimized out>, private=<optimized out>) at src/debug.c:402
#2 0x000000000061a69f in cli_parse_request (appctx=appctx@entry=0x181c0160) at src/cli.c:832
#3 0x000000000061af86 in cli_io_handler (appctx=0x181c0160) at src/cli.c:1035
#4 0x00000000006ca2f2 in task_run_applet (t=0x181c0290, context=0x181c0160, state=<optimized out>) at src/applet.c:449
CLEANUP: quic: Rename several <buf> variables at low level
Make quic_stateless_reset_token_cpy(), quic_derive_cid() and quic_get_cid_tid()
be more readable: there is no struct buffer variable manipulated by these
functions.
CLEANUP: quic: Make qc_build_pkt() be more readable
There is no <buf> variable passed to this function.
Also rename <buf_end> to <end> to mimic others functions.
Rename <beg> to <first_byte> and <end> to <last_byte>.
CLEANUP: quic: Rename <buf> variable for several low level functions
Make quic_build_packet_long_header(), quic_build_packet_short_header() and
quic_apply_header_protection() be more readable: there is no struct buffer
variables used by these functions.
CLEANUP: quic: Remove useless parameters passes to qc_purge_tx_buf()
Remove the pointer to the connection passed as parameters to qc_purge_tx_buf()
and other similar function which came with qc_purge_tx_buf() implementation.
They were there do track the connection during tests.
CLEANUP: quic: rename frame types with an explicit prefix
Each frame type used in quic_frame union has been renamed with the
following prefix "qf_". This helps to differentiate frame instances from
other internal objects.
From the idle_timer_task(), the I/O handler must be woken up to send ack. But
there is no reason to do that in draining state or killing state. In draining
state this is even forbidden.
BUG/MINOR: quic: Useless probing retransmission in draining or killing state
The timer task responsible of triggering probing retransmission did not inspect
the state of the connection before doing its job. But there is no need to
probe the peer when the connection is in draining or killing state. About the
draining state, this is even forbidden.
BUG/MINOR: quic: Possible leak during probing retransmissions
qc_dgrams_retransmit() prepares two list of frames to be retransmitted into
two datagrams. If the first datagram could not be sent, the TX buffer will
be purged with the prepared packet and its frames, but this was not the case for
the second list of frames.
BUG/MINOR: quic: Possible memory leak from TX packets
This bug arrived with this commit which was not sufficient:
BUG/MEDIUM: quic: Missing TX buffer draining from qc_send_ppkts()
Indeed, there were also remaining allocated TX packets to be released and
their TX frames.
Implement qc_purge_tx_buf() to do so which depends on qc_free_tx_coalesced_pkts()
and qc_free_frm_list().
Released version 2.8-dev8 with the following main changes :
- BUG/MEDIUM: cli: Set SE_FL_EOI flag for '_getsocks' and 'quit' commands
- BUG/MEDIUM: cli: Eat output data when waiting for appctx shutdown
- BUG/MEDIUM: http-client: Eat output data when waiting for appctx shutdown
- BUG/MEDIUM: stats: Eat output data when waiting for appctx shutdown
- BUG/MEDIUM: log: Eat output data when waiting for appctx shutdown
- BUG/MEDIUM: dns: Kill idle DNS sessions during stopping stage
- BUG/MINOR: resolvers: Wakeup DNS idle task on stopping
- BUG/MEDIUM: resolvers: Force the connect timeout for DNS resolutions
- MINOR: hlua: Stop to check the SC state when executing a hlua cli command
- BUG/MEDIUM: mux-h1: Report EOI when a TCP connection is upgraded to H2
- BUG/MEDIUM: mux-h2: Never set SE_FL_EOS without SE_FL_EOI or SE_FL_ERROR
- MINOR: quic: Trace fix in quic_pto_pktns() (handshaske status)
- BUG/MINOR: quic: Wrong packet number space probing before confirmed handshake
- MINOR: quic: Modify qc_try_rm_hp() traces
- MINOR: quic: Dump more information at proto level when building packets
- MINOR: quic: Add a trace for packet with an ACK frame
- MINOR: activity: add a line reporting the average CPU usage to "show activity"
- BUG/MINOR: stick_table: alert when type len has incorrect characters
- MINOR: thread: keep a bitmask of enabled groups in thread_set
- MINOR: fd: optimize fd_claim_tgid() for use in fd_insert()
- MINOR: fd: add a lock bit with the tgid
- MINOR: fd: implement fd_migrate_on() to migrate on a non-local thread
- MINOR: receiver: reserve special values for "shards"
- MINOR: bind-conf: support a new shards value: "by-group"
- BUG/MEDIUM: fd: don't wait for tmask to stabilize if we're not in it.
- MINOR: quic: Add packet loss and maximum cc window to "show quic"
- BUG/MINOR: quic: Ignored less than 1ms RTTs
- MINOR: quic: Add connection flags to traces
- BUG/MEDIUM: quic: Code sanitization about acknowledgements requirements
- BUG/MINOR: quic: Possible wrapped values used as ACK tree purging limit.
- BUG/MINOR: quic: SIGFPE in quic_cubic_update()
- MINOR: quic: Display the packet number space flags in traces
- MINOR: quic: Remove a useless test about probing in qc_prep_pkts()
- BUG/MINOR: quic: Wrong Application encryption level selection when probing
- CI: bump "actions/checkout" to v3 for cross zoo matrix
- CI: enable monthly test on Fedora Rawhide
- BUG/MINOR: stream: Fix test on SE_FL_ERROR on the wrong entity
- BUG/MEDIUM: stream: Report write timeouts before testing the flags
- BUG/MEDIUM: stconn: Do nothing in sc_conn_recv() when the SC needs more room
- MINOR: stream: Uninline and export sess_set_term_flags() function
- MINOR: filters: Review and simplify errors handling
- REGTESTS: fix the race conditions in log_uri.vtc
- MINOR: channel: Forwad close to other side on abort
- MINOR: stream: Introduce stream_abort() to abort on both sides in same time
- MINOR: stconn: Rename SC_FL_SHUTR_NOW in SC_FL_ABRT_WANTED
- MINOR: channel/stconn: Replace channel_shutr_now() by sc_schedule_abort()
- MINOR: stconn: Rename SC_FL_SHUTW_NOW in SC_FL_SHUT_WANTED
- MINOR: channel/stconn: Replace channel_shutw_now() by sc_schedule_shutdown()
- MINOR: stconn: Rename SC_FL_SHUTR in SC_FL_ABRT_DONE
- MINOR: channel/stconn: Replace sc_shutr() by sc_abort()
- MINOR: stconn: Rename SC_FL_SHUTW in SC_FL_SHUT_DONE
- MINOR: channel/stconn: Replace sc_shutw() by sc_shutdown()
- MINOR: tree-wide: Replace several chn_cons() by the corresponding SC
- MINOR: tree-wide: Replace several chn_prod() by the corresponding SC
- BUG/MINOR: cli: Don't close when SE_FL_ERR_PENDING is set in cli analyzer
- MINOR: stconn: Stop to set SE_FL_ERROR on sending path
- MEDIUM: stconn: Forbid applets with more to deliver if EOI was reached
- MINOR: stconn: Don't clear SE_FL_ERROR when endpoint is reset
- MINOR: stconn: Add a flag to ack endpoint errors at SC level
- MINOR: backend: Set SC_FL_ERROR on connection error
- MINOR: stream: Set SC_FL_ERROR on channels' buffer allocation error
- MINOR: tree-wide: Test SC_FL_ERROR with SE_FL_ERROR from upper layer
- MEDIUM: tree-wide: Stop to set SE_FL_ERROR from upper layer
- MEDIUM: backend: Stop to use SE flags to detect connection errors
- MEDIUM: stream: Stop to use SE flags to detect read errors from analyzers
- MEDIUM: stream: Stop to use SE flags to detect endpoint errors
- MEDIUM: stconn: Rely on SC flags to handle errors instead of SE flags
- BUG/MINOR: stconn: Don't set SE_FL_ERROR at the end of sc_conn_send()
- BUG/MINOR: quic: Do not use ack delay during the handshakes
- CLEANUP: use "offsetof" where appropriate
- MINOR: ssl: remove OpenSSL 1.0.2 mention into certificate loading error
- BUG/MEDIUM: http-ana: Properly switch the request in tunnel mode on upgrade
- BUG/MEDIUM: log: Properly handle client aborts in syslog applet
- MINOR: stconn: Add a flag to report EOS at the stream-connector level
- MINOR: stconn: Propagate EOS from a mux to the attached stream-connector
- MINOR: stconn: Propagate EOS from an applet to the attached stream-connector
- MINOR: mux-h2: make the initial window size configurable per side
- MINOR: mux-h2: make the max number of concurrent streams configurable per side
- BUG/MINOR: task: allow to use tasklet_wakeup_after with tid -1
- CLEANUP: quic: remove unused QUIC_LOCK label
- CLEANUP: quic: remove unused scid_node
- CLEANUP: quic: remove unused qc param on stateless reset token
- CLEANUP: quic: rename quic_connection_id vars
- MINOR: quic: remove uneeded tasklet_wakeup after accept
- MINOR: quic: adjust Rx packet type parsing
- MINOR: quic: adjust quic CID derive API
- MINOR: quic: remove TID ref from quic_conn
- MEDIUM: quic: use a global CID trees list
- MINOR: quic: remove TID encoding in CID
- MEDIUM: quic: handle conn bootstrap/handshake on a random thread
- MINOR: quic: do not proceed to accept for closing conn
- MINOR: protocol: define new callback set_affinity
- MINOR: quic: delay post handshake frames after accept
- MEDIUM: quic: implement thread affinity rebinding
- BUG/MINOR: quic: transform qc_set_timer() as a reentrant function
- MINOR: quic: properly finalize thread rebinding
- MAJOR: quic: support thread balancing on accept
- MINOR: listener: remove unneeded local accept flag
- BUG/MINOR: http-ana: Update analyzers on both sides when switching in TUNNEL mode
- CLEANUP: backend: Remove useless debug message in assign_server()
- CLEANUP: cli: Remove useless debug message in cli_io_handler()
- BUG/MEDIUM: stconn: Propagate error on the SC on sending path
- MINOR: config: add "no-alpn" support for bind lines
- REGTESTS: add a new "ssl_alpn" test to test ALPN negotiation
- DOC: add missing documentation for "no-alpn" on bind lines
- MINOR: ssl: do not set ALPN callback with the empty string
- MINOR: ssl_crtlist: dump "no-alpn" on "show crtlist" when "no-alpn" was set
- MEDIUM: config: set useful ALPN defaults for HTTPS and QUIC
- BUG/MEDIUM: quic: prevent crash on Retry sending
- BUG/MINOR: cfgparse: make sure to include openssl-compat
- MINOR: clock: add now_mono_time_fast() function
- MINOR: clock: add now_cpu_time_fast() function
- MEDIUM: hlua: reliable timeout detection
- MEDIUM: hlua: introduce tune.lua.burst-timeout
- CLEANUP: hlua: avoid confusion between internal timers and tick based timers
- MINOR: hlua: hook yield on known lua state
- MINOR: hlua: safe coroutine.create()
- BUG/MINOR: quic: Stop removing ACK ranges when building packets
- MINOR: quic: Do not allocate too much ack ranges
- BUG/MINOR: quic: Unchecked buffer length when building the token
- BUG/MINOR: quic: Wrong Retry token generation timestamp computing
- BUG/MINOR: mux-quic: fix crash with app ops install failure
- BUG/MINOR: mux-quic: properly handle STREAM frame alloc failure
- BUG/MINOR: h3: fix crash on h3s alloc failure
- BUG/MINOR: quic: prevent crash on qc_new_conn() failure
- BUG/MINOR: quic: consume Rx datagram even on error
- CLEANUP: errors: fix obsolete function comments
- CLEANUP: server: fix update_status() function comment
- MINOR: server/event_hdl: add proxy_uuid to event_hdl_cb_data_server
- MINOR: hlua/event_hdl: rely on proxy_uuid instead of proxy_name for lookups
- MINOR: hlua/event_hdl: expose proxy_uuid variable in server events
- MINOR: hlua/event_hdl: fix return type for hlua_event_hdl_cb_data_push_args
- MINOR: server/event_hdl: prepare for upcoming refactors
- BUG/MINOR: event_hdl: don't waste 1 event subtype slot
- CLEANUP: event_hdl: updating obsolete comment for EVENT_HDL_CB_DATA
- CLEANUP: event_hdl: fix comment typo about _sync assertion
- MINOR: event_hdl: dynamically allocated event data members
- MINOR: event_hdl: provide event->when for advanced handlers
- MINOR: hlua/event_hdl: timestamp for events
- DOC: lua: restore 80 char limitation
- BUG/MINOR: server: incorrect report for tracking servers leaving drain
- MINOR: server: explicitly commit state change in srv_update_status()
- BUG/MINOR: server: don't miss proxy stats update on server state transitions
- BUG/MINOR: server: don't miss server stats update on server state transitions
- BUG/MINOR: server: don't use date when restoring last_change from state file
- MINOR: server: central update for server counters on state change
- MINOR: server: propagate server state change to lb through single function
- MINOR: server: propagate lb changes through srv_lb_propagate()
- MINOR: server: change adm_st_chg_cause storage type
- MINOR: server: srv_append_status refacto
- MINOR: server: change srv_op_st_chg_cause storage type
- CLEANUP: server: remove unused variables in srv_update_status()
- CLEANUP: server: fix srv_set_{running, stopping, stopped} function comment
- MINOR: server: pass adm and op cause to srv_update_status()
- MEDIUM: server: split srv_update_status() in two functions
- MINOR: server/event_hdl: prepare for server event data wrapper
- MINOR: quic: support migrating the listener as well
- MINOR: quic_sock: index li->per_thr[] on local thread id, not global one
- MINOR: listener: support another thread dispatch mode: "fair"
- MINOR: receiver: add a struct shard_info to store info about each shard
- MINOR: receiver: add RX_F_MUST_DUP to indicate that an rx must be duped
- MEDIUM: proto: duplicate receivers marked RX_F_MUST_DUP
- MINOR: proto: skip socket setup for duped FDs
- MEDIUM: config: permit to start a bind on multiple groups at once
- MINOR: listener: make accept_queue index atomic
- MEDIUM: listener: rework thread assignment to consider all groups
- MINOR: listener: use a common thr_idx from the reference listener
- MINOR: listener: resync with the thread index before heavy calculations
- MINOR: listener: make sure to avoid ABA updates in per-thread index
- MINOR: listener: always compare the local thread as well
- MINOR: Make `tasklet_free()` safe to be called with `NULL`
- CLEANUP: Stop checking the pointer before calling `tasklet_free()`
- CLEANUP: Stop checking the pointer before calling `pool_free()`
- CLEANUP: Stop checking the pointer before calling `task_free()`
- CLEANUP: Stop checking the pointer before calling `ring_free()`
- BUG/MINOR: cli: clarify error message about stats bind-process
- CI: cirrus-ci: bump FreeBSD image to 13-1
- REGTESTS: remove unsupported "stats bind-process" keyword
- CI: extend spellchecker whitelist, add "clen" as well
- CLEANUP: assorted typo fixes in the code and comments
- BUG/MINOR: sock_inet: use SO_REUSEPORT_LB where available
- BUG/MINOR: tools: check libssl and libcrypto separately
- BUG/MINOR: config: fix NUMA topology detection on FreeBSD
- BUILD: sock_inet: forward-declare struct receiver
- BUILD: proto_tcp: export the correct names for proto_tcpv[46]
- CLEANUP: protocol: move the l3_addrlen to plug a hole in proto_fam
- CLEANUP: protocol: move the nb_receivers to plug a hole in protocol
- REORG: listener: move the bind_conf's thread setup code to listener.c
- MINOR: proxy: make proxy_type_str() recognize peers sections
- MEDIUM: peers: call bind_complete_thread_setup() to finish the config
- MINOR: protocol: add a flags field to store info about protocols
- MINOR: protocol: move the global reuseport flag to the protocols
- MINOR: listener: automatically adjust shards based on support for SO_REUSEPORT
- MINOR: protocol: add a function to check if some features are supported
- MINOR: sock: add a function to check for SO_REUSEPORT support at runtime
- MINOR: protocol: perform a live check for SO_REUSEPORT support
- MINOR: listener: do not restrict CLI to first group anymore
- MINOR: listener: add a new global tune.listener.default-shards setting
- MEDIUM: listener: switch the default sharding to by-group
MEDIUM: listener: switch the default sharding to by-group
Sharding by-group is exactly identical to by-process for a single
group, and will use the same number of file descriptors for more than
one group, while significantly lowering the kernel's locking overhead.
Now that all special listeners (cli, peers) are properly handled, and
that support for SO_REUSEPORT is detected at runtime per protocol, there
should be no more reason for now switching to by-group by default.
That's what this patch does. It does only this and nothing else so that
it's easy to revert, should any issue be raised.
Testing on an AMD EPYC 74F3 featuring 24 cores and 48 threads distributed
into 8 core complexes of 3 cores each, shows that configuring 8 groups
(one per CCX) is sufficient to simply double the forwarded connection
rate from 112k to 214k/s, reducing kernel locking from 71 to 55%.
MINOR: listener: add a new global tune.listener.default-shards setting
This new setting accepts "by-process", "by-group" and "by-thread" and
will dictate how listeners will be sharded by default when nothing is
specified. While the default remains "by-process", "by-group" should be
much more efficient with many threads, while not changing anything for
single-group setups.
MINOR: listener: do not restrict CLI to first group anymore
Now that we're able to run listeners on any set of groups, we don't need
to maintain a special case about the stats socket anymore. It used to be
forced to group 1 only so as to avoid startup failures in case several
groups were configured, but if it's done now, it will automatically bind
the needed FDs to have one per group so this is no more an issue.
MINOR: protocol: perform a live check for SO_REUSEPORT support
When testing if a protocol supports SO_REUSEPORT, we're now able to
verify if the OS does really support it. While it may be supported at
build time, it may possibly have been blocked in a container for
example so we'd rather know what it's like.
MINOR: sock: add a function to check for SO_REUSEPORT support at runtime
The new function _sock_supports_reuseport() will be used to check if a
protocol type supports SO_REUSEPORT or not. This will be useful to verify
that shards can really work.
MINOR: protocol: add a function to check if some features are supported
The new function protocol_supports_flag() checks the protocol flags
to verify if some features are supported, but will support being
extended to refine the tests. Let's use it to check for REUSEPORT.
MINOR: listener: automatically adjust shards based on support for SO_REUSEPORT
Now if multiple shards are explicitly requested, and the listener's
protocol doesn't support SO_REUSEPORT, sharding is disabled, which will
result in the socket being automatically duped if needed. A warning is
emitted when this happens. If "shards by-group" or "shards by-thread"
are used, these will automatically be turned down to 1 since we want
this to be possible easily using -dR on the command line without having
to djust the config. For "by-thread", a diag warning will be emitted to
help troubleshoot possible performance issues.
MINOR: protocol: move the global reuseport flag to the protocols
Some protocol support SO_REUSEPORT and others not. Some have such a
limitation in the kernel, and others in haproxy itself (e.g. sock_unix
cannot support multiple bindings since each one will unbind the previous
one). Also it's really protocol-dependent and not just family-dependent
because on Linux for some time it was supported for TCP and not UDP.
Let's move the definition to the protocols instead. Now it's preset in
tcp/udp/quic when SO_REUSEPORT is defined, and is otherwise left unset.
The enabled() config condition test validates IPv4 (generally sufficient),
and -dR / noreuseport all protocols at once.
MINOR: protocol: add a flags field to store info about protocols
We'll use these flags to know if some protocols are supported, and if
so, with what options/extensions. Reuseport will move there for example.
Two functions were added to globally set/clear a flag.
MEDIUM: peers: call bind_complete_thread_setup() to finish the config
The listeners in peers sections were still not handing the thread
groups fine. Shards were silently ignored and if a listener was bound
to more than one group, it would simply fail. Now we can call the
dedicated function to resolve all this and possibly create the missing
extra listeners.
bind_complete_thread_setup() was adjusted to use the proxy_type_str()
instead of writing "proxy" at the only place where this word was still
hard-coded so that we continue to speak about peers sections when
relevant.
REORG: listener: move the bind_conf's thread setup code to listener.c
What used to be only two lines to apply a mask in a loop in
check_config_validity() grew into a 130-line block that performs deeply
listener-specific operations that do not have their place there anymore.
In addition it's worth noting that the peers code still doesn't support
shards nor being bound to more than one group, which is a second reason
for moving that code to its own function. Nothing was changed except
recreating the missing variables from the bind_conf itself (the fe only).
CLEANUP: protocol: move the nb_receivers to plug a hole in protocol
This field forces an unaligned hole between two list heads. Let's move
it up where it will be more easily combined with other fields. In
addition, turn it to unsigned while it's still not used.
CLEANUP: protocol: move the l3_addrlen to plug a hole in proto_fam
There's a two-byte hole in proto_fam after sock_family, let's move the
l3_addrlen there as a ushort. Note that contrary to what the comment
says, it's still not used by hash algorithms though it could.
BUG/MINOR: config: fix NUMA topology detection on FreeBSD
In 2.6-dev1, NUMA topology detection was enabled on FreeBSD with commit f5d48f8b3 ("MEDIUM: cfgparse: numa detect topology on FreeBSD."). But
it suffers from a minor bug which is that it forgets to check for the
number of domains and always emits a confusing warning indicating that
multiple sockets were found while it's not the case.
BUG/MINOR: tools: check libssl and libcrypto separately
The lib compatibility checks introduced in 2.8-dev6 with commit c3b297d5a
("MEDIUM: tools: further relax dlopen() checks too consider grouped
symbols") were partially incorrect in that they check at the same time
libcrypto and libssl. But if loading a library that only depends on
libcrypto, the ssl-only symbols will be missing and this might present
an inconsistency. This is what is observed on FreeBSD 13.1 when
libcrypto is being loaded, where it sees two symbols having disappeared.
The fix consists in splitting the checks for libcrypto and libssl.
No backport is needed, unless the patch above finally gets backported.
BUG/MINOR: sock_inet: use SO_REUSEPORT_LB where available
On FreeBSD 13.1 I noticed that thread balancing using shards was not
always working. Sometimes several threads would work, but most of the
time a single one was taking all the traffic. This is related to how
SO_REUSEPORT works on FreeBSD since version 12, as it seems there is
no guarantee that multiple sockets will receive the traffic. However
there is SO_REUSEPORT_LB that is designed exactly for this, so we'd
rather use it when available.
This patch may possibly be backported, but nobody complained and it's
not sure that many users rely on shards. So better wait for some feedback
before backporting this.
BUG/MINOR: cli: clarify error message about stats bind-process
In 2.7-dev2, "stats bind-process" was removed by commit 94f763b5e
("MEDIUM: config: remove deprecated "bind-process" directives from
frontends") and an error message indicates that it's no more supported.
However it says "stats" is not supported instead of "stats bind-process",
making it a bit confusing.
MINOR: listener: always compare the local thread as well
By comparing the local thread's load with the least loaded thread's
load, we can further improve the fairness and at the same time also
improve locality since it allows a small ratio of connections not to
be migrated. This is visible on CPU usage with long connections on
very large thread counts (224) and high bandwidth (200G). The cost
of checking the local thread's load remains fairly low so there's no
reason not to do this. We continue to update the index if we select
the local thread, because it means that the two other threads were
both more loaded so we'd rather find better ones.
MINOR: listener: make sure to avoid ABA updates in per-thread index
One limitation of the current thread index mechanism is that if the
values are assigned multiple times to the same thread and the index
loops, it can match again the old value, which will not prevent a
competing thread from finishing its CAS and assigning traffic to a
thread that's not the optimal one. The probability is low but the
solution is simple enough and consists in implementing an update
counter in the high bits of the index to force a mismatch in this
case (assuming we don't try to cover for extremely unlikely cases
where the update counter loops while the index remains equal). So
let's do that. In order to improve the situation a little bit, we
now set the index to a ulong so that in 32 bits we have 8 bits of
counter and in 64 bits we have 40 bits.
MINOR: listener: resync with the thread index before heavy calculations
During heavy accept competition, the CAS will occasionally fail and
we'll have to go through all the calculation again. While the first
two loops look heavy, they're almost never taken so they're quite
cheap. However the rest of the operation is heavy because we have to
consult connection counts and queue indexes for other threads, so
better double-check if the index is still valid before continuing.
Tests show that it's more efficient do retry half-way like this.
Willy Tarreau [Wed, 29 Mar 2023 15:02:17 +0000 (17:02 +0200)]
MINOR: listener: use a common thr_idx from the reference listener
Instead of seeing each listener use its own thr_idx, let's use the same
for all those from a shard. It should provide more accurate and smoother
thread allocation.
Willy Tarreau [Mon, 27 Mar 2023 08:38:51 +0000 (10:38 +0200)]
MEDIUM: listener: rework thread assignment to consider all groups
Till now threads were assigned in listener_accept() to other threads of
the same group only, using a single group mask. Now that we have all the
relevant info (array of listeners of the same shard), we can spread the
thr_idx to cover all assigned groups. The thread indexes now contain the
group number in their upper bits, and the indexes run over te whole list
of threads, all groups included.
One particular subtlety here is that switching to a thread from another
group also means switching the group, hence the listener. As such, when
changing the group we need to update the connection's owner to point to
the listener of the same shard that is bound to the target group.
projects / thirdparty / haproxy.git / log
