Willy Tarreau [Tue, 19 Nov 2024 06:43:04 +0000 (07:43 +0100)]
REGTESTS: add missing timeouts to 30 tests
No less than 30 tests were missing timeouts, preventing them from being
started with zero-warning. Since they were not supposed to trigger, they
have been set to 30s so as never to trigger, and now they do not produce
any warning anymore.
Willy Tarreau [Tue, 19 Nov 2024 07:27:47 +0000 (08:27 +0100)]
REGTESTS: silence warning "L6 sample fetches ignored" in cond_set_var
This reg-test uses req.len in an HTTP backend. It does work but emits a
warning suggesting that this is ignored, so most likely its days are
counted now. Let's just use req.hdrs,length instead.
Annoyingly, the content-type is mandatory when the file is not empty,
that might be something to revisit in the future to relax at least one
of the rules so that the config doesn't strictly require to know the
file contents upfront.
Willy Tarreau [Tue, 19 Nov 2024 06:50:54 +0000 (07:50 +0100)]
REGTESTS: make the unit explicit for very short timeouts
Two tests were using "timeout {client,server} 1" to forcefully trigger
them, but a forthcoming patch will emit a warning for such small unitless
values, so let's be explicit about the unit.
Willy Tarreau [Tue, 19 Nov 2024 06:48:01 +0000 (07:48 +0100)]
REGTESTS: silence warning "previous 'http-response' action is final"
The regtest "h1or2_to_h1c" contains both an allow and a deny at the end,
likely to help catch rare bugs. But this triggers a warning that we can
silence by placing a condition on the penultimate rule.
Willy Tarreau [Mon, 18 Nov 2024 18:07:05 +0000 (19:07 +0100)]
MINOR: cfgparse: parse tune.bufsize.small as a size
Till now this value was parsed as raw integer using atol() and would
silently ignore any trailing suffix, causing unexpected behaviors when
set, e.g. to "4k". Let's make use of parse_size_err() on it so that
units are supported. This requires to turn it to uint as well, which
was verified to be OK.
Willy Tarreau [Mon, 18 Nov 2024 18:04:57 +0000 (19:04 +0100)]
MINOR: cfgparse: parse tune.bufsize as a size
Till now this value was parsed as raw integer using atol() and would
silently ignore any trailing suffix, preventing from starting when set
e.g. to "64k". Let's make use of parse_size_err() on it so that units are
supported. This requires to turn it to uint as well, and to explicitly
limit its range to INT_MAX - 2*sizeof(void*), which was previously
partially handled as part of the sign check.
Willy Tarreau [Mon, 18 Nov 2024 18:01:28 +0000 (19:01 +0100)]
MINOR: cfgparse: parse tune.recv_enough as a size
Till now this value was parsed as raw integer using atol() and would
silently ignore any trailing suffix, causing unexpected behaviors when
set, e.g. to "512k". Let's make use of parse_size_err() on it so that
units are supported. This requires to turn it to uint as well, and
since it's sometimes compared to an int, we limit its range to
0..INT_MAX.
Willy Tarreau [Mon, 18 Nov 2024 17:51:31 +0000 (18:51 +0100)]
MINOR: cfgparse: parse tune.pipesize as a size
Till now this value was parsed as raw integer using atol() and would
silently ignore any trailing suffix, causing unexpected behaviors when
set, e.g. to "512k". Let's make use of parse_size_err() on it so that
units are supported. This requires to turn it to uint as well, which
was verified to be OK.
Willy Tarreau [Mon, 18 Nov 2024 17:50:02 +0000 (18:50 +0100)]
MINOR: cfgparse: parse tune.{rcvbuf,sndbuf}.{frontend,backend} as sizes
Till now these values were parsed as raw integer using atol() and would
silently ignore any trailing suffix, causing unexpected behaviors when
set, e.g. to "512k". Let's make use of parse_size_err() on them so that
units are supported. This requires to turn them to uint as well, which
is OK.
Willy Tarreau [Mon, 18 Nov 2024 17:45:45 +0000 (18:45 +0100)]
MINOR: cfgparse: parse tune.{rcvbuf,sndbuf}.{client,server} as sizes
Till now these values were parsed as raw integer using atol() and would
silently ignore any trailing suffix, causing unexpected behaviors when
set, e.g. to "512k". Let's make use of parse_size_err() on them so that
units are supported. This requires to turn them to uint as well, which
is OK.
Willy Tarreau [Mon, 18 Nov 2024 14:27:28 +0000 (15:27 +0100)]
MINOR: sample: extend the "when" converter to support an ACL
Sometimes conditions to decide of an anomaly are not as easy to define
as just an error or a success. One example use case would be to monitor
the transfer time and fix a threshold.
An idea suggested by Tristan would be to make permit the "when"
converter to refer to a more variable or dynamic condition.
Here we make this possible by making "when" rely on a named ACL. The
ACL then needs to be specified in either the proxy or the defaults
section. Since it is evaluated inline, it may even refer to information
available at the end (at log time) such as the data transfer time. If
the ACL evalutates to true, the converter passes the data.
Example: log "dbg={-}" when fine, or "dbg={... debug info ...}" on slow
transfers:
acl slow_xfer res.timer.data ge 10000 # more than 10s is slow
log-format "$HAPROXY_HTTP_LOG_FMT \
fsdbg={%[fs.debug_str,when(acl,slow_xfer)]} \
bsdbg={%[bs.debug_str,when(acl,slow_xfer)]}"
Willy Tarreau [Fri, 15 Nov 2024 17:42:29 +0000 (18:42 +0100)]
[RELEASE] Released version 3.1-dev13
Released version 3.1-dev13 with the following main changes :
- MEDIUM: mworker: depreciate the 'program' section
- BUILD: ot: use a cebtree instead of a list for variable names
- MINOR: startup: replace HAPROXY_LOAD_SUCCESS with global load_status
- BUG/MINOR: startup: set HAPROXY_CFGFILES in read_cfg
- BUG/MINOR: cli: don't show sockpairs in HAPROXY_CLI and HAPROXY_MASTER_CLI
- BUG/MEDIUM: stconn: Don't forward shut for SC in connecting state
- BUG/MEDIUM: resolvers: Insert a non-executed resulution in front of the wait list
- MINOR: debug: explicitly permit the counter condition to be empty
- MINOR: debug: add a new counter type for glitches
- MINOR: mux-h2: count glitches when they're reported
- BUG/MINOR: deinit: release uri_auth admin rules
- MINOR: uri_auth: add stats_uri_auth_free helper
- MEDIUM: uri_auth: implement clean uri_auth cleaning
- MINOR: mux-quic/h3: count glitches when they're reported
- BUG/MEDIUM: mux-h2: Don't send RST_STREAM frame for streams with no ID
- BUG/MINOR: Don't report early srv aborts on request forwarding in DONE state
- MINOR: promex: Expose the global node and description in process metrics
- MINOR: promex: Add global and proxies description as labels to all metrics
- OPTIM: pattern: only apply LRU cache for large enough lists
- BUG/MEDIUM: checks: make sure to always apply offsets to now_ms in expiration
- BUG/MINOR: debug: do not set task expiration to TICK_ETERNITY
- BUG/MEDIUM: mailers: make sure to always apply offsets to now_ms in expiration
- BUG/MINOR: mux_quic: make sure to always apply offsets to now_ms in expiration
- BUG/MINOR: peers: make sure to always apply offsets to now_ms in expiration
- BUG/MEDIUM: clock: make sure now_ms cannot be TICK_ETERNITY
- MINOR: debug/cli: replace "debug dev counters" with "debug counters"
- DOC: config: add tune.h2.{be,fe}.rxbuf to the global keywords index
- MINOR: chunk: add a BUG_ON upon the next init_trash_buffer()
MINOR: chunk: add a BUG_ON upon the next init_trash_buffer()
The trash pool is initialized twice in haproxy, first during STG_POOL,
and 2nd after configuration parsing.
Doing alloc_trash_chunk() between this 2 phases can lead to strange
things if we are using it after, indeed the pool is destroyed and
trying to do a free_trash_chunk() or accessing the pointer will lead to
crashes.
This patch checks that we don't have used buffers from the trash pool
before initializing the pool again.
Willy Tarreau [Fri, 15 Nov 2024 15:19:05 +0000 (16:19 +0100)]
MINOR: debug/cli: replace "debug dev counters" with "debug counters"
"debug dev" commands are not meant to be used by end-users, and are
purposely not documented. Yet due to their usefulness in troubleshooting
sessions, users are increasingly invited by developers to use some of
them.
"debug dev counters" is one of them. Better move it to "debug counters"
and document it so that users can check them even if the output can look
cryptic at times. This, combined with DEBUG_GLITCHES, can be convenient
to observe suspcious activity. The doc however precises that the format
may change between versions and that new entries/types might appear
within a stable branch.
Willy Tarreau [Fri, 15 Nov 2024 14:52:17 +0000 (15:52 +0100)]
BUG/MEDIUM: clock: make sure now_ms cannot be TICK_ETERNITY
In clock ticks, 0 is TICK_ETERNITY. Long ago we used to make sure now_ms
couldn't be zero so that it could be assigned to expiration timers, but
it has long changed after functions like tick_add() were instrumented to
make the check. The problem is that aside the rare few accidental direct
assignments to expiration dates, it's also used to mark the beginning of
an event that's later checked against TICK_ETERNITY to know if it has
already struck. The problem in this case is that certain events may just
be replaced or dropped just because they apparently never appeared. It's
probably the case for stconn's "lra" and "fsb" fields, just like it is
for all those involving tick_add_ifset(), like h2c->idle_start.
The right approach would be to change the type of now_ms to something
else that cannot take direct computations and that represents a timestamp,
forcing to always use the conversion functions. The variables holding such
timestamps would also be distinguished from intervals. At first glance we
could have for timestamps:
- 0 = never happened (for the past), eternity (for the future)
- X = date
and for intervals:
- 0 = not set
- X = interval
However this requires significant changes. Instead for now, let's just
make sure again that now_ms is never 0 by setting it to 1 when this
happens (1 / 4 billion times, or 1ms every 49.7 days).
This will need to be carefully backported to older versions. Note that
with this patch backported, the previous ones fixing the zero date are
not strictly needed.
Willy Tarreau [Fri, 15 Nov 2024 14:44:05 +0000 (15:44 +0100)]
BUG/MINOR: peers: make sure to always apply offsets to now_ms in expiration
Now_ms can be zero nowadays, so it's not suitable for direct assignment to
t->expire, as there's a risk that the timer never wakes up once assigned
(TICK_ETERNITY). Let's use tick_add(now_ms, 0) for an immediate wakeup
instead. The impact here might be a reconnect programmed upon signal
receipt at the wrapping date not having a working timeout.
Willy Tarreau [Fri, 15 Nov 2024 14:41:21 +0000 (15:41 +0100)]
BUG/MINOR: mux_quic: make sure to always apply offsets to now_ms in expiration
Now_ms can be zero nowadays, so it's not suitable for direct assignment to
t->expire, as there's a risk that the timer never wakes up once assigned
(TICK_ETERNITY). Let's use tick_add(now_ms, 0) for an immediate wakeup
instead. The impact looks nul since the task is also woken up, but better
not leave such tasks in the timer tree anyway.
Willy Tarreau [Fri, 15 Nov 2024 14:39:58 +0000 (15:39 +0100)]
BUG/MEDIUM: mailers: make sure to always apply offsets to now_ms in expiration
Now_ms can be zero nowadays, so it's not suitable for direct assignment to
t->expire, as there's a risk that the timer never wakes up once assigned
(TICK_ETERNITY). Let's use tick_add(now_ms, 0) for an immediate wakeup
instead. The impact here might be mailers suddenly stopping.
Willy Tarreau [Fri, 15 Nov 2024 14:37:38 +0000 (15:37 +0100)]
BUG/MINOR: debug: do not set task expiration to TICK_ETERNITY
Using "debug task", it's possible to change a task's expiration, but
we must be careful not to set it to TICK_ETERNITY. Let's use tick_add()
instead. The risk is basically nul since it's a debugging command, so
no backport is needed.
Willy Tarreau [Fri, 15 Nov 2024 14:34:46 +0000 (15:34 +0100)]
BUG/MEDIUM: checks: make sure to always apply offsets to now_ms in expiration
Now_ms can be zero nowadays, so it's not suitable for direct assignment to
t->expire, as there's a risk that the timer never wakes up once assigned
(TICK_ETERNITY). Let's use tick_add(now_ms, 0) for an immediate wakeup
instead. The impact here might be health checks suddenly stopping.
Willy Tarreau [Sun, 3 Nov 2024 17:42:26 +0000 (18:42 +0100)]
OPTIM: pattern: only apply LRU cache for large enough lists
As shown in issue #1518, the LRU cache has a non-null cost that can
sometimes be above the match cost it's trying to avoid. After a number
of tests, it appears that:
- "simple" match operations (sub, beg, end, int etc) reach a break-even
after ~20 patterns in list
- "heavy" match operations (reg) reach a break-even after ~5 patterns in
list
Let's only consult the LRU cache when the number of patterns in the
expression is at least as large as this limit. Of course there will
always be outliers but it already starts good.
Another improvement consists in reducing the cache size to further
speed up lookups, which makes sense if less expressions use the cache.
MINOR: promex: Add global and proxies description as labels to all metrics
While the global description is exposed, when defined, in a dedicated
metric, it is not possible to dump the description defined in a
frontend/listen/backend sections. So, thanks to this patch, it is now
possible to dump it as a label of all metrics of the corresponding
section. To do so, "desc-labels" parameter must be provided on the URL:
/metrics?desc-labels
When this parameter is set, if a description is provided in a section,
including the global one, the "desc" label will be added to all metrics of
this section. For instance:
MINOR: promex: Expose the global node and description in process metrics
The global node value is now exposed via "haproxy_process_node" metrics. The
metric value is always set to 1 and the node name itself is the "node"
label. The same is performed for the global description. But only if it is
defined. In that case "haproxy_process_description" metric is defined, with
1 as value and the description itself is set in the "desc" label.
BUG/MINOR: Don't report early srv aborts on request forwarding in DONE state
L7-retries may be ignored if server aborts are detected during the request
forwarding, when the request is already in DONE state.
When a request was fully processed (so in HTTP_MSG_DONE state) and is
waiting for be forwarded to the server, there is a test to detect server
aborts, to be able to report the error. However, this test must be skipped
if the response was not received yet, to let the reponse analyszers handle
the abort. It is important to properly handle the retries. This test must
only be performed if the response analysis was finished. It means the
response must be at least in HTTP_MSG_BODY state.
BUG/MEDIUM: mux-h2: Don't send RST_STREAM frame for streams with no ID
On server side, the H2 stream is first created with an unassigned ID (ID ==
0). Its ID is assigned when the request is emitted, before formatting the
HEADERS frame. However, the session may be aborted during that stage. We
must take care to not emit RST_STREAM frame for this stream, because it does
not exist yet for the server.
It is especially important to do so because, depending on the timing, it may
also happens before the H2 PREFACE was sent.
This patch must be backported to all stable versions. It is related to issue
Willy Tarreau [Thu, 14 Nov 2024 14:38:25 +0000 (15:38 +0100)]
MINOR: mux-quic/h3: count glitches when they're reported
The qcc_report_glitch() function is now replaced with a macro to support
enumerating counters for each individual glitch line. For now this adds
36 such counters. The macro supports an optional description, though that
is not being used for now.
As a reminder, this requires to build with -DDEBUG_GLITCHES=1.
proxy auth_uri struct was manually cleaned up during deinit, but the logic
behind was kind of akward because it was required to find out which ones
were shared or not. Instead, let's switch to a proper refcount mechanism
and free the auth_uri struct directly in proxy_free_common().
When uri_auth admin rules were implemented in 474be415
("[MEDIUM] stats: add an admin level") no attempt was made to free the
list of allocated rules, which makes valgrind unhappy upon deinit when
"stats admin" is used in the config.
To fix the issue, let's cleanup the admin rules list upon deinit where
uri_auth freeing is already handled.
While this could be backported to every stable versions, given how minor
this is and has no impact on the dying process, it is probably not worth
the effort.
Willy Tarreau [Thu, 14 Nov 2024 07:54:32 +0000 (08:54 +0100)]
MINOR: mux-h2: count glitches when they're reported
The h2c_report_glitch() function is now replaced with a macro to support
enumerating counters for each individual glitch line. For now this adds
43 such counters. The macro supports an optional description, though that
is not being used for now. It gives outputs like this (note that the last
one was purposely instrumented to pass a description):
> debug dev counters glt all
0 GLT mux_h2.c:5976 h2c_dec_hdrs()
0 GLT mux_h2.c:5960 h2c_dec_hdrs()
(...)
0 GLT mux_h2.c:2207 h2c_frt_recv_preface()
0 GLT mux_h2.c:1954 h2c_frt_stream_new(): new stream too early
As a reminder, this requires to build with -DDEBUG_GLITCHES=1.
Willy Tarreau [Thu, 14 Nov 2024 07:49:38 +0000 (08:49 +0100)]
MINOR: debug: add a new counter type for glitches
COUNT_GLITCH() will implement an unconditional counter on its declaration
line when DEBUG_GLITCHES is set, and do nothing otherwise. The output will
be reported as "GLT" and can be filtered as "glt" on the CLI. The purpose
is to help figure what's happening if some glitches counters start going
through the roof. The macro supports an optional string argument to
describe the cause of the glitch (e.g. "truncated header"), which is then
reported in the dump.
For now this is conditioned by DEBUG_GLITCHES but if it turns out to be
light enough, maybe we'll keep it enabled full time. In this case it
might have to be moved away from debug dev, or at least documented (or
done as debug counters maybe so that dev can remain undocumented and
updatable within a branch?).
Willy Tarreau [Thu, 14 Nov 2024 07:47:00 +0000 (08:47 +0100)]
MINOR: debug: explicitly permit the counter condition to be empty
In order to count new event types, we'll need to support empty conditions
so that we don't have to fake if (1) that would pollute the output. This
change checks if #cond is an empty string before concatenating it with
the optional var args, and avoids dumping the colon on the dump if the
whole description is empty.
BUG/MEDIUM: resolvers: Insert a non-executed resulution in front of the wait list
When a resolver is woken up to process DNS resolutions, it is possible to
trigger an infinite loop on the resolver's wait list because delayed
resolutions are always reinserted at the end of this list. This leads the
watchdog to kill the process. By re-inserting them in front of the list,
that fixes the bug.
When a resolver tries to send the queries for the resolutions in its wait
list, it may be unable to proceed for a resolution. This may happen because
the resolution must be skipped (no hostname to resolv, a resolution already
in-progress) or when an error occurred. In that case, the resolution is
re-inserted in the resolver's wait list to be retry later, on a next wakeup.
However, the resolution is inserted at the end of the wait list. So it is
immediately reevaluated, in the same execution loop, instead of to be
delayed. Most of time, it is not an issue because the resolution is
considered as not expired on the second run. But it is an problem when the
internal time wraps and is equal to 0. In that case, the resolution
expiration date is badly computed and it is always considered as expired. If
two or more resolutions are in that state, the resolver loops for ever on
its wait list, until the process is killed by the watchdog.
So we can argue that the way the resolution expiration date is computed must
be fixed. And it would be true in a perfect world. However, the resolvers
code is so crapy that it is hard to be sure to not introduce regressions. It
is farly easier to re-insert delayed resolutions in front of the wait
list. This fixes the issue and at worst, these resolutions will be evaluated
one time too many on the next wakeup and only if now_ms was equal to 0 on
the prior wakeup.
This patch should be backported to all stable versions. On 2.2, LIST_ADD()
must be used instead of LIST_INSERT()
BUG/MEDIUM: stconn: Don't forward shut for SC in connecting state
In connecting state, shutdown must not be forwarded or scheduled because
otherwise this will prevent any connection retries. Indeed, if a EOS is
reported by the mux during the connection establishment, this should be
handled by the stream to eventually retries. If the write side is closed
first, this will not be possible because the stconn will be switched in DIS
state. If the shut is scheduled because pending data are blocked, the same
may happen, depending on the abort-on-close option.
This patch should be slowly be backported as far as 2.4. But an observation
period is mandatory. On 2.4, the patch must be adapted to use the
stream-interface API.
BUG/MINOR: cli: don't show sockpairs in HAPROXY_CLI and HAPROXY_MASTER_CLI
Before this fix, HAPROXY_CLI and HAPROXY_MASTER_CLI have contained along with
CLI sockets addresses internal sockpairs, which are used only for master CLI
(reload sockpair and sockpair shared with a worker process). These internal
sockpairs are always need to be hidden.
At the moment there is no any client, who uses sockpair addresses for the
stats listener or in order to connect to master CLI. So, let's simply not copy
these internal sockpair addresses of MASTER and GLOBAL proxy listeners.
As listeners with sockpairs are skipped and they can be presented in the
listeners list in any order, let's add semicolon separator between addresses
only in the case, when there are already some string saved in the trash and we
are sure, that we are adding a new address to it. Otherwise, we could have such
weird output:
HAPROXY_MASTER_CLI=unix@/tmp/mcli.sock;;
This fix is need to be backported in all stable versions.
BUG/MINOR: startup: set HAPROXY_CFGFILES in read_cfg
load_cfg() is called only once before the first reading of the configuration
(we parse here only the global section). Then, before reading the rest of the
sections (second call of read_cfg()), we call clean_env(). As
HAPROXY_CFGFILES is set in load_cfg(), which is called only once, clean_env()
erases it. Thus, it's not longer shown in "show env" output.
To fix this, let's set HAPROXY_CFGFILES in read_cfg(). Like this in
master-worker mode it is set for master and for worker processes, as it was
before the refactoring.
This fix doesn't need to be backported as related to the latest master-worker
architecture change.
MINOR: startup: replace HAPROXY_LOAD_SUCCESS with global load_status
After master-worker refactoring, master performs re-exec only once up to
receiving "reload" command or USR2 signal. There is no more the second
master's re-exec to free unused memory. Thus, there is no longer need to export
environment variable HAPROXY_LOAD_SUCCESS with worker process load status. This
status can be simply saved in a global variable load_status.
Miroslav Zagorac [Mon, 28 Oct 2024 14:31:55 +0000 (15:31 +0100)]
BUILD: ot: use a cebtree instead of a list for variable names
In order for the function flt_ot_vars_scope_dump() to work, it is
necessary to take into account the changes made by the commits 47ec7c681
("OPTIM: vars: use a cebtree instead of a list for variable names") and 5d350d1e5 ("OPTIM: vars: use multiple name heads in the vars struct").
The function is only used if the OT_DEBUG=1 option is set when compiling
HAProxy.
The program section is unreliable and should not be used, more reliable
alternatives exist outside HAProxy. Let's depreciate the section so we
could remove it completely in 3.3.
Willy Tarreau [Fri, 8 Nov 2024 14:46:54 +0000 (15:46 +0100)]
[RELEASE] Released version 3.1-dev12
Released version 3.1-dev12 with the following main changes :
- MINOR: startup: tune.renice.{startup,runtime} allow to change priorities
- BUG/MEDIUM: promex: Fix dump of extra counters
- BUILD: import/mt_list: support building with TCC
- BUILD: compiler: define __builtin_prefetch() for tcc
- CLEANUP: quic: Remove the useless directive "tune.quic.backend.max-idle-timeou"
- DOC: config: document connection error 44 (reverse connect failure)
- CLEANUP: connection: properly name the CO_ER_SSL_FATAL enum entry
- DEBUG: cli: support closing "hard" using close() in addition to fd_delete()
- MINOR: connection: add more connection error codes to cover common errno
- MINOR: rawsock: set connection error codes when returning from recv/send/splice
- MINOR: connection: add new sample fetch functions fc_err_name and bc_err_name
- MINOR: quic: Help diagnosing malformed probing packets
- BUG/MINOR: quic: fix malformed probing packet building
- MINOR: listener: Remove useless checks on the receiver protocol existence
- MINOR: http-conv: Remove unreachable goto statement in sample_conv_q_preferred
- MINOR: http: don't %-encode the payload when not relevant
- MINOR: quic: simplify qc_parse_pkt_frms() return path
- MINOR: quic: use dynamically allocated frame on parsing
- MINOR: quic: extend return value of CRYPTO parsing
- BUG/MINOR: quic: repeat packet parsing to deal with fragmented CRYPTO
- BUG/MINOR: mworker: do 'program' postparser checks in read_cfg_in_discovery_mode
- EXAMPLES: add "traces.cfg" with traces examples
- BUG/MEDIUM: quic: do not consider ACK on released stream as error
- CLEANUP: stats: fix misleading comment on top of stat_idx_info
- MINOR: wdt: move the local timers to a struct
- MINOR: debug: add a function to dump a stuck thread
- DEBUG: wdt: better detect apparently locked up threads and warn about them
- DEBUG: cli: make it possible for "debug dev loop" to trigger warnings
- DEBUG: wdt: make the blocked traffic warning delay configurable
- DEBUG: wdt: add a stats counter "BlockedTrafficWarnings" in show info
- DEBUG: wdt: set the default blocked task delay to 100 ms
- MINOR: debug: move the "recover now" warn message after the optional notes
- MINOR: event_hdl: add event_hdl_sub_list_empty() helper func
- MINOR: pattern: add _pat_ref_new() helper func
- OPTIM: pattern: use malloc() to initialize new pat_ref struct
- MINOR: pattern: add pat_ref_free() helper func
- CLEANUP: guid: remove global tree export
- BUG/MINOR: guid/server: ensure thread-safety on GUID insert/delete
- DOC: management: explain the change of behavior of the program section
- BUG/MEDIUM: mux-h2: try to wait for the peer to read the GOAWAY
- BUG/MEDIUM: quic: prevent crash due to CRYPTO parsing error
BUG/MEDIUM: quic: prevent crash due to CRYPTO parsing error
A packet which contains several splitted and out of order CRYPTO frames
may be parsed multiple times to ensure it can be handled via ncbuf. Only
3 iterations can be performed to prevent excessive CPU usage.
There is a risk of crash if packet parsing is interrupted after maximum
iterations is reached, or no progress can be made on the ncbuf. This is
because <frm> may be dangling after list_for_each_entry_safe()
The crash occurs on qc_frm_free() invokation, on error path of
qc_parse_pkt_frms(). To fix it, always reset frm to NULL after
list_for_each_entry_safe() to ensure it is not dangling.
This should fix new report on github isue #2776. This regression has
been triggered by the following patch : 1767196d5b2d8d1e557f7b3911a940000166ecda
BUG/MINOR: quic: repeat packet parsing to deal with fragmented CRYPTO
As such, it must be backported up to 2.6, after the above patch.
Willy Tarreau [Fri, 8 Nov 2024 09:02:47 +0000 (10:02 +0100)]
BUG/MEDIUM: mux-h2: try to wait for the peer to read the GOAWAY
When timeout http-keep-alive is very short (e.g. 10ms), it's possible
sometimes for a client to face truncated responses due to an early
close that happens while the system is still pushing the last data,
colliding with the client's WINDOW_UPDATEs that trigger RSTs.
Here we're trying to do better: first we send a GOAWAY on timeout, then
we wait up to clientfin/client timeout for the peer to react so that we
don't immediately close. This is sufficient to avoid truncation as soon
as the timeout is more than a few hundred ms.
It's not certain it should be backported, because it's a bit sensistive
and might possibly fall into certain edge cases.
BUG/MINOR: guid/server: ensure thread-safety on GUID insert/delete
Since 3.0, it is possible to assign a GUID to proxies, listeners and
servers. These objects are stored in a global tree guid_tree.
Proxies and listeners are static. However, servers may be added or
deleted at runtime, which imply that guid_tree must be protected. Fix
this by declaring a read-write lock to protect tree access.
For now, only guid_insert() and guid_remove() are protected using a
write lock. Outside of these, GUID tree is not accessed at runtime. If
server CLI commands are extended to support GUID as server identifier,
lookup operation should be extended with a read lock protection.
Note that during stat-file preloading, GUID tree is accessed for lookup.
However, as it is performed on startup which is single threaded, there
is no need for lock here. A BUG_ON() has been added to ensure this
precondition remains true.
This bug could caused a segfault when using dynamic servers with GUID.
However, it was never reproduced for now.
This must be backported up to 3.0. To avoid a conflict issue, the
previous cleanup patch can be merged before it.
For now, pat_ref struct are never freed, except during init in case of
error. The freeing is done directly in the init functions because we
don't have an helper for that.
No having an helper func to properly free pat_ref struct doesn't encourage
us to free unused pat_ref structs, plus it is error-prone if new dynamic
members are added to pat_ref struct in the future.
To fix that, let's add a pat_ref_free() helper func and use it where
relevant (which means only under pat_ref init function for now..)
OPTIM: pattern: use malloc() to initialize new pat_ref struct
As mentioned in the previous commit, in _pat_ref_new(), it was not
strictly needed to explicitly assign all struct members to 0 since
the struct was allocated with calloc() which does the zeroing for us.
However, it was verified that we already initialize all fields explictly,
thus there is no reason to keep using calloc() instead of malloc(). In
fact using malloc() is less expensive, so let's use that instead now.
pat_ref_newid() and pat_ref_new() are two functions to create and
initialize a pat_ref struct based on input parameters.
Both function perform the same generic allocation and initialization
for pat_ref struct, thus there is quite a lot of code redundancy.
This is error-prone if the pat_ref init sequence has to be updated at
some point.
To reduce maintenance costs, let's add a _pat_ref_new() helper func that
takes care of the generic allocation and base initialization for pat_ref
struct.
event_hdl_sub_list_empty() may be used to know if the subscription list
passed as argument is empty or not (ie: if there currently are any
subcribers or not). It can be useful to know if the subscription is empty
is order to avoid unecessary preparation work and skip event publishing to
save CPU time if we already know that no one is interested in tracking the
changes for a given subscription list.
Willy Tarreau [Thu, 7 Nov 2024 06:56:13 +0000 (07:56 +0100)]
MINOR: debug: move the "recover now" warn message after the optional notes
At the end of the too long processing warning added by commit 0950778b3a
("MINOR: debug: add a function to dump a stuck thread"), there can be some
optional notes about lua and memory trimming. However it's a bit awkward
that they appear after the "trying to recover now" message. Let's just move
that message after the notes.
Willy Tarreau [Wed, 6 Nov 2024 16:51:57 +0000 (17:51 +0100)]
DEBUG: wdt: set the default blocked task delay to 100 ms
The warn-blocked-traffic-after can be significantly lowered. In any
case, in order to be usable it must be well below the limit to have a
chance to emit exploitable traces before the watchdog finally fires.
Even configured at 1ms it looks very difficult to trigger it on a
laptop doing SSL and compression, so applying a 100-fold factor to
cover for large configs and small machines sounds sane for 3.1. In any
case, even at 100ms, the service degradation becomes quite visible.
Willy Tarreau [Wed, 6 Nov 2024 10:21:45 +0000 (11:21 +0100)]
DEBUG: wdt: better detect apparently locked up threads and warn about them
In order to help users detect when threads are behaving abnormally, let's
try to emit a warning when one is no longer making any progress. This will
allow to catch faulty situations more accurately, instead of occasionally
triggering just after the long task. It will also let users know that there
is something wrong with their configuration, and inspect the call trace to
figure whether they're using excessively long rules or Lua for example (the
usual warnings about lua-load vs lua-load-per-thread are still reported).
The warning will only be emitted for threads not yet marked as stuck so
as not to interfere with panic dumps and avoid sending a warning just
before a panic. A tainted flag is set when this happens however (0x2000).
Willy Tarreau [Wed, 6 Nov 2024 10:20:45 +0000 (11:20 +0100)]
MINOR: debug: add a function to dump a stuck thread
There's currently no way to just emit a warning informing that a thread
is stuck without crashing. This is a problem because sometimes users
would benefit from this info to clean up their configuration (e.g. abuse
of map_regm, lua-load etc).
This commit adds a new function ha_stuck_warning() that will emit a
warning indicating that the designated thread has been stuck for XX
milliseconds, with a number of streams blocked, and will make that
thread dump its own state. The warning will then be sent to stderr,
along with some reminders about the impacts of such situations to
encourage users to fix their configuration.
In order not to disrupt operations, a local 4kB buffer is allocated
in the stack. This should be quite sufficient.
Willy Tarreau [Wed, 6 Nov 2024 17:07:14 +0000 (18:07 +0100)]
CLEANUP: stats: fix misleading comment on top of stat_idx_info
The comment asks to update the "metrics_info" array, which does not
exist, instead it's called stat_cols_info[] and is in stats.c. Let's
mention all that to save time searching for the needed info.
While no version seems to have ever known that "metrics_info", it's not
needed to backport this as it's only a comment.
BUG/MEDIUM: quic: do not consider ACK on released stream as error
When an ACK is received by haproxy, a lookup is performed to retrieve
the related emitted frames. For STREAM type frames, a lookup is
performed under quic_conn stream_desc tree. Indeed, the corresponding
stream instance could be already released if multiple ACK were received
refering to the same stream offset, which can happen notably if
retransmission occured.
qc_handle_newly_acked_frm() implements this logic. If the case with an
already released stream is encounted, an error is returned. In the end,
this error is propagated via qc_parse_pkt_frms() into
qc_treat_rx_pkts(), despite being in fact a perfectly valid case. Fix
this by adjusting ACK handling function to return a success value for
the particular case of released stream instead.
The impact of this bug is unknown, but it can have several consequences.
* if the packet with the ACK contains other frames after it, their
content will be skipped
* the packet won't be acknowledged by haproxy, even if it contains other
frames and is ack-eliciting. This may cause unneeded retransmission by
the client.
* RTT sampling information related to this ACK is ignored by haproxy
Finally, it also caused the increment of the quic_conn counter
dropped_parsing (droppars in "show quic" output) which should be
reserved only for real error cases.
This regression is present since the following patch : e7578084b0536e3e5988be7f09091c85beb8fa9d
MINOR: quic: implement dedicated type for out-of-order stream ACK
Before, qc_handle_newly_acked_frm() return type was always ignored. As
such, no backport is needed.
BUG/MINOR: mworker: do 'program' postparser checks in read_cfg_in_discovery_mode
cfg_program_postparser() contains 2 parts:
- check the combination of MODE_MWORKER and "program" section. if
"program" section was parsed, MODE_MWORKER is mandatory;
- check "command" keyword, which is mandatory for this section as
well.
This is more appropriate now, after the master-worker refactoring, do the
first part in read_cfg_in_discovery_mode, where we already check the
combination of MODE_MWORKER and -S option.
We need to do the second part just below, in read_cfg_in_discovery_mode() as
well, because it's only the master process, who parses now program section and
programs are forked before running postparser functions in step_init_2.
Otherwise, mworker_ext_launch_all() will emit a log message, that program is
started, but actually nothing has been launched, if 'command' keyword is
absent.
This not needs to be backported, as related to the master-worker refactoring.
BUG/MINOR: quic: repeat packet parsing to deal with fragmented CRYPTO
A ClientHello may be splitted accross several different CRYPTO frames,
then mixed in a single QUIC packet. This is used notably by clients such
as chrome to render the first Initial packet opaque to middleboxes.
Each packet frame is handled sequentially. Out-of-order CRYPTO frames
are buffered in a ncbuf, until gaps are filled and data is transferred
to the SSL stack. If CRYPTO frames are heavily splitted with small
fragments, buffering may fail as ncbuf does not support small gaps. This
causes the whole packet to be rejected and unacknowledged. It could be
solved if the client reemits its ClientHello after remixing its CRYPTO
frames.
This patch is written to improve CRYPTO frame parsing. Each CRYPTO
frames which cannot be buffered due to ncbuf limitation are now stored
in a temporary list. Packet parsing is completed until all frames have
been handled. If temporary list is not empty, reparsing is done on the
stored frames. With the newly buffered CRYPTO frames, ncbuf insert
operation may this time succeeds if the frame now covers a whole gap.
Reparsing will loop until either no progress can be made or it has been
done at least 3 times, to prevent CPU utilization.
This patch should fix github issue #2776.
This should be backported up to 2.6, after a period of observation. Note
that it relies on the following refactor patches :
MINOR: quic: extend return value of CRYPTO parsing
MINOR: quic: use dynamically allocated frame on parsing
MINOR: quic: simplify qc_parse_pkt_frms() return path
MINOR: quic: extend return value of CRYPTO parsing
qc_handle_crypto_frm() is the function used to handled a newly received
CRYPTO frame. Change its API to use a newly dedicated return type. This
allows to report if the frame was properly handled, ignored if already
parsed previously or rejected after a fatal error.
This commit does not have any functional changes. However, it allows to
simplify qc_handle_crypto_frm() API by removing <fast_retrans> as output
parameter. Also, this patch will be necessary to support multiple
iteration of packet parsing for CRYPTO frames.
MINOR: quic: use dynamically allocated frame on parsing
qc_parse_pkt_frms() is the function responsible to parse a received QUIC
packet. Payload is decoded and splitted into individual frames which are
then handled individually. Previously, frame was used as locally stack
allocated. Change this to work on a dynamically allocated frame.
This commit does bring any functional changes. However, it will be
useful to extend packet parsing. In particular, it will be necessary to
save some frames during parsing to reparse them after the others.
Change qc_parse_pkt_frms() return path for normal and error cases. Most
notably, it allows to remove local variable ret as now return value is
hardcoded on normal and err label. This also allows to define a
different trace for error leaving code.
MINOR: http: don't %-encode the payload when not relevant
As reported by Pierre Maoui in GH #2477, it's not possible to render
control chars from variables or expressions verbatim in the payload part
of http-return statements. That's a problem because this part should not
require to be encoded at all (we could even imagine building favicons on
the fly for example).
In fact it is the LOG_OPT_HTTP option when passed as default options on
parse_logformat_string() which tells the log encoder that the payload
should be http-encoded using lf_chunk() instead of being printed using the
per-type encoder.
This option was set when parsing logformat expressions for lf-string
expression under http-return statements, as well as logformat expressions
for set-map action. While it is true that those actions may only be
used under http context, the LOG_OPT_HTTP logformat option is not relevant
there, because the payload is expected to be used without being encoded.
So let's simply get rid of this option when parsing logformat expressions
for set-map action key/value and lf-string from http-request return
action, and add a note next to LOG_OPT_HTTP option to indicate that it is
used to tell the log encoder that the payload should be HTTP-encoded.
Thanks to Pierre for having reported the issue and Willy for the
analysis and patch proposal.
MINOR: http-conv: Remove unreachable goto statement in sample_conv_q_preferred
This was reported by Coverity. In sample_conv_q_preferred() function, a goto
statement after a "while(1)" loop is unreachable. Instead of just removing
it, the same goto statement in the loop is replaced by a break. It is safer
this way, in case the loop change in future.
MINOR: listener: Remove useless checks on the receiver protocol existence
The receiver protocol is always set when a listener is created or cloned. At
least for now. And there is no check on it at many places, except in
listener_accept() function. So, let's remove remaining useless checks. That
will avoid false Coverity reports in future.
which prevents haproxy from sending PING only packets/datagrams (some
packets/datagrams with only PING frame as ack-eliciting frames inside).
Such packets/datagrams are useful in rare cases during retransmissions
when one wants to probe the peer without exceeding the anti-amplification
limit.
Modify the condition passed to qc_build_pkt() to add padding to the current
datagram. One does not want to do that when probing the peer without ack-eliciting
frames passed as <frms> parameter. Indeed qc_build_pkt() calls qc_do_build_pkt()
which supports this case: if <probe> is true (probing required), qc_do_build_pkt()
handles the case where some padding must be added to a PING only packet/datagram.
This is the case when probing with an empty <frms> frame list of ack-eliciting
frames without exceeding the anti-amplification limit from qc_dgrams_retransmit().
Add some comments to qc_build_pkt() and qc_do_build_pkt() to clarify this
as this code is easy to break!
Thank you for @Tristan971 for having reported this issue in GH #2709.
MINOR: quic: Help diagnosing malformed probing packets
Add a BUG_ON() to detect some malformed packets which are supposed to probe the
peer without being ack-eliciting: the peer would not acknowledged such packets.
Willy Tarreau [Tue, 5 Nov 2024 17:04:21 +0000 (18:04 +0100)]
MINOR: connection: add new sample fetch functions fc_err_name and bc_err_name
These functions return a symbolic error code such as ECONNRESET to keep
logs compact while making them human-readable. It's a good alternative
to the numeric code in that it's more expressive, and a good one to the
full message since it's shorter and more precise (some codes even match
errno names).
The doc was updated so that the symbolic names appear in the table. It
could be useful to backport this feature to help with troubleshooting
some issues, though backporting the doc might possibly be more annoying
in case users have local patches already, so maybe the table update does
not need to be backported in this case.
Willy Tarreau [Tue, 5 Nov 2024 16:57:43 +0000 (17:57 +0100)]
MINOR: rawsock: set connection error codes when returning from recv/send/splice
For a long time the errno values returned by recv/send/splice() were not
translated to connection error codes. There are not that many eligible
and having them would help a lot when debugging some complex issues where
logs disagree with network traces. Let's add them now.
Willy Tarreau [Tue, 5 Nov 2024 16:49:15 +0000 (17:49 +0100)]
MINOR: connection: add more connection error codes to cover common errno
While we get reports of connection setup errors in fc_err/bc_err, we
don't have the equivalent for the recv/send/splice syscalls. Let's
add provisions for new codes that cover the common errno values that
recv/send/splice can return, i.e. ECONNREFUSED, ENOMEM, EBADF, EFAULT,
EINVAL, ENOTCONN, ENOTSOCK, ENOBUFS, EPIPE. We also add a special case
for when the poller reported the error itself. It's worth noting that
EBADF/EFAULT/EINVAL will generally indicate serious bugs in the code
and should not be reported.
The only thing is that it's quite hard to forcefully (and reliably)
trigger these errors in automated tests as the timing is critical.
Using iptables to manually reset established connections in the
middle of large transfers at least permits to see some ECONNRESET
and/or EPIPE, but the other ones are harder to trigger.
Willy Tarreau [Tue, 5 Nov 2024 16:43:35 +0000 (17:43 +0100)]
DEBUG: cli: support closing "hard" using close() in addition to fd_delete()
"debug dev close <fd>" currently closes that FD using fd_delete() after
checking that it's known from the fdtab. Sometimes we also want to just
perform a pure close() of FDs not in the fdtab (pollers, etc) in order
to provoke certain error cases. The optional "hard" argument to the
command will make it use a plain close() instead of fd_delete() and skip
the fd owner check. The main visible effect when closing a traffic socket
with it is that instead of dying from a double fd_delete() by seeing that
fd.owner is already 0, it will die during the next fd_insert() seeing that
fd.owner was not 0.
Willy Tarreau [Tue, 5 Nov 2024 17:05:58 +0000 (18:05 +0100)]
CLEANUP: connection: properly name the CO_ER_SSL_FATAL enum entry
It was the only one prefixed with "CO_ERR_", making it harder to batch
process and to look up. It was added in 2.5 by commit 61944f7a73 ("MINOR:
ssl: Set connection error code in case of SSL read or write fatal failure")
so it can be backported as far as 2.6 if needed to help integrate other
patches.
CLEANUP: quic: Remove the useless directive "tune.quic.backend.max-idle-timeou"
First there is a typo in the directive name, then it is not documented and
finally, it is not used at all. The directive is only removed from the
keyword list. Parsing function is not updated.
Willy Tarreau [Tue, 5 Nov 2024 14:41:57 +0000 (15:41 +0100)]
BUILD: compiler: define __builtin_prefetch() for tcc
We're using a few occurrences of __builtin_prefetch() but tcc doesn't
know about it so let's give it a dummy definition. Now the code builds
and works again with tcc without thread support.
Willy Tarreau [Tue, 5 Nov 2024 14:25:31 +0000 (15:25 +0100)]
BUILD: import/mt_list: support building with TCC
TCC is often convenient to quickly test builds, run CI tests etc. It has
limited thread support (e.g. no thread-local stuff) but that is often
sufficient for testing. TCC lacks __atomic_exchange_n() but has the
exactly equivalent __atomic_exchange(), and doesn't have any barrier.
For this reason we force the atomic_exchange to use the stricter SEQ_CST
mem ordering that allows to ignore the barrier.
[wt: that's upstream commit ca8b865 ("BUILD: support building with TCC")]
When extra counters are dumped for an entity (frontend, backend, server or
listener), there is a filter on capabilities. Some extra counters are not
available for all entities and must be ignored. However, when this was
performed, the field number, used as an index to dump the metric value, was
still incremented while it should not and leads to an overflow or a stats
mix-up.
MINOR: startup: tune.renice.{startup,runtime} allow to change priorities
This commit introduces the tune.renice.startup and tune.renice.runtime
global keywords that allows to change the priority with setpriority().
tune.renice.startup is parsed and applied in the worker or the standalone
process for configuration parsing. If this keyword is used alone, the
nice value is changed to the previous one after configuration parsing.
tune.renice.runtime is applied after configuration parsing, so in the
worker or a standalone process. Combined with tune.renice.startup it
allows to have a different nice value during configuration parsing and
during runtime.
The feature was discussed in github issue #1919.
Example:
global
tune.renice.startup 15
tune.renice.runtime 0
Willy Tarreau [Fri, 1 Nov 2024 09:17:02 +0000 (10:17 +0100)]
[RELEASE] Released version 3.1-dev11
Released version 3.1-dev11 with the following main changes :
- BUG/MINOR: httpclient: return NULL when no proxy available during httpclient_new()
- BUG/MEDIUM: mworker/httpclient: initialization skipped by accident in mworker mode
- BUG/MINOR: resolvers/mworker: missing default resolvers in mworker mode
- MINOR: mworker/ocsp: skip ocsp-update proxy init in master
- BUG/MEDIUM: stconn: Wait iobuf is empty to shut SE down during a check send
- MINOR: mux-h1: Show the SD iobuf in trace messages on stream send events
- MINOR: mux-h1: Add a trace on shutdown when keep-alive is not possible
- BUG/MINOR: http-ana: Don't report a server abort if response payload is invalid
- BUG/MEDIUM: stconn: Check FF data of SC to perform a shutdown in sc_notify()
- BUG/MAJOR: filters/htx: Add a flag to state the payload is altered by a filter
- REGTESTS: Never reuse server connection in http-messaging/truncated.vtc
- BUG/MINOR: quic: avoid leaking post handshake frames
- MINOR: quic: send new tokens (NEW_TOKEN) even for 1RTT sessions
- BUG/MEDIUM: quic: avoid freezing 0RTT connections
- DOC: config: fix rfc7239 forwarded typo in desc
- MINOR: http_ext: implement rfc7239_{nn,np} converters
- CLEANUP: http_ext: remove useless BUG_ON() in http_handle_xot_header()
- BUG/MINOR: sample: free err2 in smp_resolve_args for type ARGT_REG
- MINOR: arg: add an argument type for identifier
- BUILD: buffers: keep b_getblk_nc() and b_peek_varint() in buf.h
- CLEANUP: buffers: simplify b_get_varint()
- OPTIM: buffers: avoid a useless wrapping check for ofs == 0
- MINOR: debug: make mark_tainted() return the previous value
- MINOR: chunk: drop the global thread_dump_buffer
- MINOR: debug: split ha_thread_dump() in two parts
- MINOR: debug: slightly change the thread_dump_pointer signification
- MINOR: debug: make ha_thread_dump_done() take the pointer to be used
- MINOR: debug: replace ha_thread_dump() with its two components
- MEDIUM: debug: on panic, make the target thread automatically allocate its buf
- BUILD: mux-h2/traces: fix build on 32-bit due to size of the DATA frame
- CI: prepare Coverity build for Ubuntu 24
- CI: bump development builds explicitely to Ubuntu 24.04
- CI: modernize macos builds to macos-15
- BUG/MINOR: mworker: fix mworker-max-reloads parser
- MINOR: mux-quic: simplify sending of empty STREAM FIN
- BUG/MINOR: mux-quic: do not close STREAM with empty FIN if no data sent
- CLEANUP: debug: make the BUG_ON() macros check the condition in the outer one
- MEDIUM: debug: add match counters for BUG_ON/WARN_ON/CHECK_IF
- MINOR: debug: add a new debug macro COUNT_IF()
- MINOR: debug: add "debug dev counters" to list code counters
- BUG/MEDIUM: stats-html: Never dump more data than expected during 0-copy FF
- BUG/MEDIUM: mux-h2: Remove H2S from send list if data are sent via 0-copy FF
- BUG/MINOR: stconn: Pretend the SE have more data to deliver on abortonclose
- CLEANUP: stream: remove outdated comments
- DEBUG: stream: Add debug counters to track some client/server aborts
- DEBUG: mux-h1: Add debug counters to track some errors
- MINOR: mux-h1: Add support of the debug string for logs
- MINOR: stream: maintain per-stream counters of the number of passes on code
- MINOR: filters: add per-filter call counters
- MINOR: sample: add the "when" converter to condition some expressions
- BUG/MEDIUM: connection/http-reuse: fix address collision on unhandled address families
- BUILD: spoe: fix build warning on older gcc around sub-struct initialization
- Revert "OPTIM: mux-h2: make h2_send() report more accurate wake up conditions"
- DEBUG: mux-h1: Add debug counters to track errors with in/out pending data
- BUG/MINOR: mux-h1: Fix conditions on pipe in some COUNT_IF()
- MINOR: activity/memprofile: show per-DSO stats
- BUG/MINOR: mworker/cli: show master startup logs in recovery mode
- MINOR: mworker: stop MASTER proxy listener on worker mcli sockpair
- MINOR: error: simplify startup_logs_init_shm
- BUG/MINOR: mworker: show worker warnings in startup logs
- CLEANUP: mworker: clean mworker_reexec
- MINOR: mworker/cli: split mworker_cli_proxy_create
- BUG/MINOR: server: fix dynamic server leak with check on failed init
- BUG/MEDIUM: server: fix race on servers_list during server deletion
- BUG/MEDIUM: stconn: Report blocked send if sends are blocked by an error
- BUG/MINOR: http-ana: Fix wrong client abort reports during responses forwarding
- BUG/MINOR: stconn: Don't disable 0-copy FF if EOS was reported on consumer side
- MINOR: mworker/cli: add 'debug' to 'show proc'
- MINOR: mworker/cli: remove comment line for program when useless
- MINOR: mworker/cli: 'show proc debug' for old workers
- BUILD: debug: silence a build warning with threads disabled
- CLEANUP: mux-h2: remove the unused "full" variable in h2_frt_transfer_data()
- MINOR: pools: export the pools variable
- MINOR: debug: place a magic pattern at the beginning of post_mortem
- MINOR: debug: place the post_mortem struct in its own section.
- MINOR: debug: store important pointers in post_mortem
- MINOR: debug: do not limit backtraces to stuck threads
- MINOR: cli: remove non-printable characters from 'debug dev fd'
- MINOR: cli: add an 'echo' command
- MINOR: debug: also add a pointer to struct global to post_mortem
- CLEANUP: mworker: make mworker_create_master_cli more readable
- BUG/MEIDUM: mworker: fix fd leak from master to worker
- BUG/MINOR: mworker/cli: fix mworker_cli_global_proxy_new_listener
- MINOR: tools: add strnlen2() helper
- CLEANUP: log: use strnlen2() in _lf_text_len() to compute string length
- DOC: design: add notes about more detailed error reporting for logs
- MINOR: debug: also add fdtab and acitvity to struct post_mortem
- MINOR: debug: remove the redundant process.thread_info array from post_mortem
- DEV: gdb: add a number of gdb scripts to navigate in core dumps
- BUG/MINOR: trace: stop rewriting argv with -dt
- MEDIUM: protocol: make abns a custom unix socket address family
- MEDIUM: protocol: rely on AF_CUST_ABNS family to recognize ABNS sockets
- CLEANUP: tools: rely on address family to detect ABNS sockets
- MINOR: protocol: create abnsz socket address family
- MINOR: sock: restore effective UNIX family in sock_get_old_sockets()
- MEDIUM: sock: also restore effective unix family in get_{src,dst}()
- MEDIUM: sock_unix: use per-family addrcmp function
- MEDIUM: socket: add zero-terminated ABNS alternative
- BUG/MINOR: ssl/cli: 'set ssl cert' does not check the transaction name correctly
- BUG/MINOR: mworker: mworker_reexec: unset MODE_STARTING before free startup logs ring
- BUG/MINOR: errors: startup_logs_free: set global startup_logs ptr to NULL
- BUG/MINOR: errors: print_message: don't allocate startup logs ring
- BUG/MINOR: startup: don't fork worker if started with -c -W
- BUG/MINOR: startup: dump libs only in worker if started with -W -dL
- BUG/MINOR: startup: dump keywords only in worker if started with -W -dKAll
- BUG/MINOR: startup: don't dump polling info for master in verbose mode
- CI: switch QUIC Interop on AWS-LC to common docker image
- CI: switch QUIC Interop on LibreSSL to common docker image
- CI: enable chacha20 test on LibreSSL QUIC Interop
- DOC: config: add missing glitch_{cnt,rate} data types
- DOC: config: add missing glitch_{cnt,rate} sample definitions
- CI: LibreSSL QUIC Interop: fix docker context
- DEBUG: mux-h1: Add H1C expiration dates in trace messages
- BUG/MEDIUM: mux-h1: Fix how timeouts are applied on H1 connections
- BUG/MINOR: http-ana: Report internal error if an action yields on a final eval
- MINOR: stream: Save last evaluated rule on invalid yield
- MINOR: quic: complete trace in qc_may_build_pkt()
- MINOR: quic: move qc_send_mux() prototype into quic_tx.h
- MINOR: stream: Replace last_rule_file/line fields by a more generic field
- MINOR: stream: Save the last filter evaluated interrupting the processing
- MINOR: stream: Save the entity waiting to continue its processing
- MINOR: stream: Use an enum to identify last and waiting entities for streams
- MINOR: stream: Add http-buffer-request option in the waiting entities
- DOC: config: Add documentation about last_entity sample fetch
- DOC: config: Add documentation about waiting_entity sample fetch
MINOR: stream: Add http-buffer-request option in the waiting entities
When http-buffer-request option is set on a proxy, the processing will be
paused to wait the full request payload or a full buffer. So it is an entity
that block the processing, just like a rule or a filter that yields. So now,
it is reported as a waiting entity if an error or a timeout occurred.
To do so, an stream entity type is added for this option. There is no
pointer. And "waiting_entity" sample fetch returns the option name.
MINOR: stream: Save the entity waiting to continue its processing
When a rule or a filter yields because it waits for something to be able to
continue its processing, this entity is saved in the stream. If an error or
a timeout occurred, info on this entity may be retrieved via the
"waiting_entity" sample fetch, for instance to dump it in the logs. This
info may be useful to found root cause of some bugs because it is a way to
know the processing was temporarily stopped. This may explain timeouts for
instance.
MINOR: stream: Save the last filter evaluated interrupting the processing
It is very similar to the last evaluated rule. When a filter returns an
error that interrupts the processing, it is saved in the stream, in the
last_entity field, with the type 2. The pointer on filter config is
saved. This pointer never changes during runtime and is part of the proxy's
structure. It is an element of the filter_configs list in the proxy
structure.
"last_entity" sample fetch was update accordingly. The filter identifier is
returned, if defined. Otherwise the save pointer.
MINOR: stream: Replace last_rule_file/line fields by a more generic field
The last evaluated rule is now saved in a generic structure, named
last_entity, with a type to identify it. The idea is to be able to store
other kind of entity that may interrupt a specific processing.
The type of the last evaluated rule is set to 1. It will be replace later by
an enum to be more explicit. In addition, the pointer to the rule itself is
saved instead of its location.
The sample fetch "last_entity" was added to retrieve the information about
it. In this case, it is the rule localtion, the config file containing the
rule followed by the line where the rule is defined, separated by a
colon. This sample fetch is not documented yet.
MINOR: stream: Save last evaluated rule on invalid yield
When an action yields while it is not allowed, an internal error is
reported. This interrupts the processing. So info about the last evaluated
rule must be filled.
This patch may be bakcported if needed. If so, the commit ("MINOR: stream:
Save last evaluated rule on invalid yield") must be backported first.
BUG/MEDIUM: mux-h1: Fix how timeouts are applied on H1 connections
There were several flaws in the way the different timeouts were applied on
H1 connections. First, the H1C task handling timeouts was not created if no
client/server timeout was specified. But there are other timeouts to
consider. First, the client-fin/server-fin timeouts. But for frontend
connections, http-keey-alive and http-request timeouts may also be used. And
finally, on soft-stop, the close-spread-time value must be considered too.
So at the end, it is probably easier to always create a task to manage H1C
timeouts. Especially since the client/server timeouts are most often set.
Then, when the expiration date of the H1C's task must only be updated if the
considered timeout is set. So tick_add_ifset() must be used instead of
tick_add(). Otherwise, if a timeout is undefined, the taks may expire
immediately while it should in fact never expire.
Finally, the idle expiration date must only be considered for idle
connections.
This patch should be backported in all stable versions, at least as far as
2.6. On the 2.4, it will have to be slightly adapted for the idle_exp
part. On 2.2 and 2.0, the patch will have to be rewrite because
h1_refresh_timeout() is quite different.