Willy Tarreau [Thu, 18 Aug 2022 14:12:15 +0000 (16:12 +0200)]
BUG/MINOR: mux-h2: send a CANCEL instead of ES on truncated writes
If a POST upload is cancelled after having advertised a content-length,
or a response body is truncated after a content-length, we're not allowed
to send ES because in this case the total body length must exactly match
the advertised value. Till now that's what we were doing, and that was
causing the other side (possibly haproxy) to respond with an RST_STREAM
PROTOCOL_ERROR due to "ES on DATA frame before content-length".
We can behave a bit cleaner here. Let's detect that we haven't sent
everything, and send an RST_STREAM(CANCEL) instead, which is designed
exactly for this purpose.
This patch could be backported to older versions but only a little bit
of exposure to make sure it doesn't wake up a bad behavior somewhere.
It relies on the following previous commit:
"MINOR: mux-h2: make streams know if they need to send more data"
Willy Tarreau [Thu, 18 Aug 2022 14:03:51 +0000 (16:03 +0200)]
MINOR: mux-h2: make streams know if they need to send more data
H2 streams do not even know if they are expected to send more data or
not, which is problematic when closing because we don't know if we're
closing too early or not. Let's start by adding a new stream flag
"H2_SF_MORE_HTX_DATA" to indicate this on the tx path.
Willy Tarreau [Thu, 18 Aug 2022 13:30:41 +0000 (15:30 +0200)]
MINOR: mux-h2/traces: report transition to SETTINGS1 before not after
Traces indicating "switching to XXX" generally apply before the transition
so that the current connection state is visible in the trace. SETTINGS1
was incorrect in this regard, with the trace being emitted after. Let's
fix this.
No need to backport this, as this is purely cosmetic.
Willy Tarreau [Thu, 18 Aug 2022 09:19:57 +0000 (11:19 +0200)]
BUG/MEDIUM: mux-h2: do not fiddle with ->dsi to indicate demux is idle
When switching to H2_CS_FRAME_H, we do not want to present the previous
frame's state, flags, length etc in traces, or we risk to confuse the
analysis, making the reader think that the header information presented
is related to the new frame header being analysed. A naive approach could
have consisted in simply relying on the current parser state (FRAME_H
being that state), but traces are emitted before switching the state,
so traces cannot rely on this.
This was initially addressed by commit 73db434f7 ("MINOR: h2/trace: report
the frame type when known") which used to set dsi to -1 when the connection
becomes idle again, but was accidentally broken by commit 5112a603d
("BUG/MAJOR: mux_h2: Don't consume more payload than received for skipped
frames") which moved dsi after calling the trace function.
But in both cases there's problem with this approach. If an RST or WU frame
cannot be uploaded due to a busy mux, and at the same time we complete
processing on a perfect end of frame with no single new frame header, we
can leave the demux loop with dsi=-1 and with RST or WU to be sent, and
these ones will be sent for stream ID -1. This is what was reported in
github issue #1830. This can be reproduced with a config chaining an h1->h2
proxy to an empty h2 frontend, and uploading a large body such as below:
This shows that we must never affect ->dsi which must always remain valid,
and instead we should set "something else". That something else could be
served by the demux frame type, but that one also needs to be preserved
for the RST_STREAM case. Instead, let's just add a connection flag to say
that the demuxing is in progress. This will be set once a new demux header
is set and reset after the end of a frame. This way the trace subsystem
can know that dft/dfl must not be displayed, without affecting the logic
relying on such values.
Given that the commits above are old and were backported to 1.8, this
new one also needs to be backported as far as 1.8.
Many thanks to David le Blanc (@systemmonkey42) for spotting, reporting,
capturing and analyzing this bug; his work permitted to quickly spot the
problem.
Willy Tarreau [Thu, 18 Aug 2022 16:04:37 +0000 (18:04 +0200)]
BUG/MEDIUM: cli: always reset the service context between commands
Erwan Le Goas reported that chaining certain commands on the CLI would
systematically crash the process; for example, "show version; show sess".
This happened since the conversion of cli context to appctx->svcctx,
because if applet_reserve_svcctx() is called a first time for a tiny
context, it's allocated in-situ, and later a keyword that wants a
larger one will see that it's not null and will reuse it and will
overwrite the end of the first one's context.
What is missing is a reset of the svcctx when looping back to
CLI_ST_GETREQ.
This needs to be backported to 2.6, and relies on previous commit
"MINOR: applet: add a function to reset the svcctx of an applet".
Willy Tarreau [Thu, 18 Aug 2022 16:13:34 +0000 (18:13 +0200)]
MINOR: applet: add a function to reset the svcctx of an applet
The CLI needs to reset the svcctx between commands, and there was nothing
done to handle this. Let's add appctx_reset_svcctx() to do that, it's the
closing equivalent of appctx_reserve_svcctx().
This will have to be backported to 2.6 as it will be used by a subsequent
patch to fix a bug.
Amaury Denoyelle [Wed, 17 Aug 2022 16:02:47 +0000 (18:02 +0200)]
MEDIUM: h3: concatenate multiple cookie headers
As specified by RFC 9114, multiple cookie headers must be concatenated
into a single entry before passing it to a HTTP/1.1 connection. To
implement this, reuse the same function as already used for HTTP/2
module.
This should answer to feature requested in github issue #1818.
Amaury Denoyelle [Wed, 17 Aug 2022 14:33:53 +0000 (16:33 +0200)]
REORG: h2: extract cookies concat function in http_htx
As specified by RFC 7540, multiple cookie headers are merged in a single
entry before passing it to a HTTP/1.1 connection. This step is
implemented during headers parsing in h2 module.
Extract this code in the generic http_htx module. This will allow to
reuse it quickly for HTTP/3 implementation which has the same
requirement for cookie headers.
Amaury Denoyelle [Wed, 17 Aug 2022 14:33:13 +0000 (16:33 +0200)]
BUG/MEDIUM: quic: fix crash on MUX send notification
MUX notification on TX has been edited recently : it will be notified
only when sending its own data, and not for example on retransmission by
the quic-conn layer. This is subject of the patch : b29a1dc2f4a334c1c7fea76c59abb4097422c05c
BUG/MINOR: quic: do not notify MUX on frame retransmit
A new flag QUIC_FL_CONN_RETRANS_LOST_DATA has been introduced to
differentiate qc_send_app_pkts invocation by MUX and directly by the
quic-conn layer in quic_conn_app_io_cb(). However, this is a first
problem as internal quic-conn layer usage is not limited to
retransmission. For example for NEW_CONNECTION_ID emission.
Another problem much important is that send functions are also called
through quic_conn_io_cb() which has not been protected from MUX
notification. This could probably result in crash when trying to notify
the MUX.
To fix both problems, quic-conn flagging has been inverted : when used
by the MUX, quic-conn is flagged with QUIC_FL_CONN_TX_MUX_CONTEXT. To
improve the API, MUX must now used qc_send_mux which ensure the flag is
set. qc_send_app_pkts is now static and can only be used by the
quic-conn layer.
This must be backported wherever the previously mentionned patch is.
BUG/MINOR: quic: Missing initializations for ducplicated frames.
When duplication frames in qc_dup_pkt_frms(), ->pkt member was not correctly
initialized (copied from the original frame). This could not have any impact
because this member is initialized whe the frame is added to a packet.
This was also the case for ->flags.
Also replace the pool_zalloc() call by a call to pool_alloc().
Mateusz Malek [Wed, 17 Aug 2022 12:22:09 +0000 (14:22 +0200)]
BUG/MEDIUM: http-ana: fix crash or wrong header deletion by http-restrict-req-hdr-names
When using `option http-restrict-req-hdr-names delete`, HAproxy may
crash or delete wrong header after receiving request containing multiple
forbidden characters in single header name; exact behavior depends on
number of request headers, number of forbidden characters and position
of header containing them.
This patch fixes GitHub issue #1822.
Must be backported as far as 2.2 (buggy feature got included in 2.2.25,
2.4.18 and 2.5.8).
Amaury Denoyelle [Fri, 12 Aug 2022 12:30:04 +0000 (14:30 +0200)]
BUG/MINOR: quic: do not notify MUX on frame retransmit
On STREAM emission, quic-conn notifies MUX through a callback named
qcc_streams_sent_done(). This also happens on retransmission : in this
case offset are examined and notification is ignored if already seen.
Indeed, if offset diff is NULL, frame is now not ignored. This is to
support FIN notification with a final empty STREAM frame. A side-effect
of this is that if the last stream frame is retransmitted, it won't be
ignored in qcc_streams_sent_done().
In most cases, this side-effect is harmless as qcs instance will soon be
freed after being closed. But if qcs is still alive, this will cause a
BUG_ON crash as it is considered as locally closed.
This bug depends on delay condition and seems to be extremely rare. But
it might be the reason for a crash seen on interop with s2n client on
http3 testcase :
To reproduce it locally, code was artificially patched to produce
retransmission and avoid qcs liberation.
In order to fix this and avoid future class of similar problem, the best
way is to not call qcc_streams_sent_done() to notify MUX for
retranmission. To implement this, we test if any of
QUIC_FL_CONN_RETRANS_OLD_DATA or the new flag
QUIC_FL_CONN_RETRANS_LOST_DATA is set. A new wrapper
qc_send_app_retransmit() has been added to set the new flag as a
complement to already existing qc_send_app_probing().
Amaury Denoyelle [Wed, 17 Aug 2022 08:08:16 +0000 (10:08 +0200)]
MINOR: quic: refactor application send
Adjust qc_send_app_pkts function : remove <old_data> arg and provide a
new wrapper function qc_send_app_probing() which should be used instead
when probing with old data.
This simplifies the interface of the default function, most notably for
the MUX which does not interfer with retransmission.
QUIC_FL_CONN_RETRANS_OLD_DATA flag is set/unset directly in the wrapper
qc_send_app_probing().
At the same time, function documentation has been updated to clarified
arguments and return values.
This commit will be useful for the next patch to differentiate MUX and
retransmission send context. As a consequence, the current patch should
be backported wherever the next one will be.
Amaury Denoyelle [Thu, 11 Aug 2022 16:35:55 +0000 (18:35 +0200)]
MINOR: mux-quic: add missing args on some traces
Complete some MUX traces by adding qcc or qcs instance as arguments when
this is possible. This will be useful when several connections are
interleaved.
Amaury Denoyelle [Tue, 16 Aug 2022 09:29:08 +0000 (11:29 +0200)]
BUG/MEDIUM: mux-quic: reject uni stream ID exceeding flow control
Emit STREAM_LIMIT_ERROR if a client tries to open an unidirectional
stream with an ID greater than the value specified by our flow-control
limit. The code is similar to the bidirectional stream opening.
MAX_STREAMS_UNI emission is not implement for the moment and is left as
a TODO. This should not be too urgent for the moment : in HTTP/3, a
client has only a limited use for unidirectional streams (H3 control
stream + 2 QPACK streams). This is covered by the value provided by
haproxy in transport parameters.
This patch has been tagged with BUG as it should have prevented last
crash reported on github issue #1808 when opening a new unidirectional
streams with an invalid ID. However, it is probably not the main cause
of the bug contrary to the patch
commit 11a6f4007b908b49ecd3abd5cd10fba177f07c11
BUG/MINOR: quic: Wrong status returned by qc_pkt_decrypt()
Amaury Denoyelle [Tue, 16 Aug 2022 15:42:47 +0000 (17:42 +0200)]
MINOR: qpack: report error on enc/dec stream close
As specified by RFC 9204, encoder and decoder streams must not be
closed. If the peer behaves incorrectly and closes one of them, emit a
H3_CLOSED_CRITICAL_STREAM connection error.
To implement this, QPACK stream decoding API has been slightly adjusted.
Firstly, fin parameter is passed to notify about FIN STREAM bit.
Secondly, qcs instance is passed via unused void* context. This allows
to use qcc_emit_cc_app() function to report a CONNECTION_CLOSE error.
Amaury Denoyelle [Tue, 16 Aug 2022 15:16:47 +0000 (17:16 +0200)]
MINOR: h3: report error on control stream close
As specified by RFC 9114 the control stream must not be closed. If the
peer behaves incorrectly and closes it, emit a H3_CLOSED_CRITICAL_STREAM
connection error.
Amaury Denoyelle [Tue, 16 Aug 2022 12:41:57 +0000 (14:41 +0200)]
MINOR: quic: adjust quic_frame flag manipulation
Replace a plain '=' operator by '|=' when setting quic_frame
QUIC_FL_TX_FRAME_LOST flag.
For the moment, this change has no impact as only two exclusive flags
are defined for quic_frame. On the edited code path we are certain that
QUIC_FL_TX_FRAME_ACKED is not set due to a previous if statement, so a
plain equal or a binary OR is strictly identical.
This change will be useful if new flags are defined for quic_frame in
the future. These new flags won't be resetted automatically thanks to
binary OR without explictly intended, which otherwise could easily lead
to new bugs.
Amaury Denoyelle [Tue, 16 Aug 2022 09:13:59 +0000 (11:13 +0200)]
CLEANUP: exclude haring with .gitignore
haring is a new utility to decode file-backed rings. It is compiled in
dev/ directory and so the binary should be specified in .gitignore to
not clutter git status output.
MINOR: stick-table: Add table_expire() and table_idle() new converters
table_expire() returns the expiration delay for a stick-table entry associated
to an input sample. Its counterpart table_idle() returns the time the entry
remained idle since the last time it was updated.
Both converters may take a default value as second argument which is returned
when the entry is not present.
Willy Tarreau [Wed, 17 Aug 2022 08:45:22 +0000 (10:45 +0200)]
MINOR: chunk: inline alloc_trash_chunk()
This function is responsible for all calls to pool_alloc(trash), whose
total size can be huge. As such it's quite a pain that it doesn't provide
more hints about its users. However, since the function is tiny, it fully
makes sense to inline it, the code is less than 0.1% larger with this.
This way we can now detect where the callers are via "show profiling",
e.g.:
Willy Tarreau [Wed, 17 Aug 2022 07:35:16 +0000 (09:35 +0200)]
MINOR: pools/memprof: store and report the pool's name in each bin
Storing the pointer to the pool along with the stats is quite useful as
it allows to report the name. That's what we're doing here. We could
store it in place of another field but that's not convenient as it would
require to change all functions that manipulate counters. Thus here we
store one extra field, as well as some padding because the struct turns
56 bytes long, thus better go to 64 directly. Example of output from
"show profiling memory":
Willy Tarreau [Wed, 17 Aug 2022 07:12:53 +0000 (09:12 +0200)]
MINOR: pool/memprof: report pool alloc/free in memory profiling
Pools are being used so well that it becomes difficult to profile their
usage via the regular memory profiling. Let's add new entries for pools
there, named "p_alloc" and "p_free" that correspond to pool_alloc() and
pool_free(). Ideally it would be nice to only report those that fail
cache lookups but that's complicated, particularly on the free() path
since free lists are released in clusters to the shared pools.
It's worth noting that the alloc_tot/free_tot fields can easily be
determined by multiplying alloc_calls/free_calls by the pool's size, and
could be better used to store a pointer to the pool itself. However it
would require significant changes down the code that sorts output.
If this were to cause a measurable slowdown, an alternate approach could
consist in using a different value of USE_MEMORY_PROFILING to enable pools
profiling. Also, this profiler doesn't depend on intercepting regular malloc
functions, so we could also imagine enabling it alone or the other one alone
or both.
Tests show that the CPU overhead on QUIC (which is already an extremely
intensive user of pools) jumps from ~7% to ~10%. This is quite acceptable
in most deployments.
Willy Tarreau [Wed, 17 Aug 2022 06:53:36 +0000 (08:53 +0200)]
MINOR: memprof: export the minimum definitions for memory profiling
Right now it's not possible to feed memory profiling info from outside
activity.c, so let's export the function and move the enum and struct
to the include file.
BUG/MINOR: quic: Wrong status returned by qc_pkt_decrypt()
This bug came with this big commit:
"MEDIUM: quic: xprt traces rework"
This is the <ret> variable value which must be returned by most of the xprt functions.
This leaded packets which could not be decrypted to be parsed, with weird frames
to be parsed as found by Tristan in GH #1808.
To be backported where the commit above was backported.
BUG/MINOR: quic: MIssing check when building TX packets
When building an ack-eliciting frame only packet, if we did not manage to add
at least one such a frame to the packet, we did not notify the caller about
the fact the packet is empty. This could lead the caller to believe
everything was ok and make it endlessly try to build packet again and again.
This issue was amplified by the recent changes where a while(1) loop has been
added to qc_send_app_pkt() which calls qc_do_build_pkt() through qc_prep_app_pkts()
until we could not prepare packets. Before this recent change, I guess only
one empty packet was sent.
This patch checks that non empty packets could be built by qc_do_build_pkt()
and makes this function return an error if this was the case. Also note that
such an issue could happened only when the packet building was limited by
the congestion control.
Thank you to Tristan for having reported this issue in GH #1808.
Amaury Denoyelle [Fri, 12 Aug 2022 13:56:21 +0000 (15:56 +0200)]
BUG/MINOR: mux-quic: fix crash with traces in qc_detach()
qc_detach() is used to free a qcs as notified by sedesc. If there is no
more stream active and the connection is considered as dead, it will
then be freed. This prevent to dereference qcc in TRACE macro. Else this
will cause a crash.
Use a different code-path on release for qc_detach() to fix this bug.
This will fix the last occurence of crash on github issue #1808.
This has been introduced by recent QUIC MUX traces rework. Thus, it does
not need to be backport.
Willy Tarreau [Fri, 12 Aug 2022 13:38:20 +0000 (15:38 +0200)]
MINOR: ring: archive a previous file-backed ring on startup
In order to ensure that an instant restart of the process will not wipe
precious debugging information, and to leave time for an admin to archive
a copy of a ring, now upon startup, any previously existing file will be
renamed with the extra suffix ".bak", and any previously existing file
with suffix ".bak" will be removed.
Willy Tarreau [Fri, 12 Aug 2022 13:03:12 +0000 (15:03 +0200)]
BUILD: sink: replace S_IRUSR, S_IWUSR with their octal value
The build broke on freebsd with S_IRUSR undefined after commit 0b8e9ceb1
("MINOR: ring: add support for a backing-file"). Maybe another include
is needed there, but the point is that we really don't care about these
symbolic names, file modes are more readable as 0600 than via these
cryptic names anyway, so let's go back to 0600. This will teach me not
to try to make things too clean.
Willy Tarreau [Fri, 12 Aug 2022 10:09:41 +0000 (12:09 +0200)]
DEV: haring: support remapping LF in contents with CR VT
Some traces may contain LF characters which are quite cumbersome to
deal with using the common tools. Given that the utility still has
access to the raw traces and knows where the delimiters are, let's
offer the possibility to remap LF characters to a different sequence.
Here we're using CR VT which will have the same visual appearance but
will remain on the same line for grep etc. This behavior is enabled by
the -l option. It's not enabled by default because it's 50% slower due
to content processing.
Willy Tarreau [Fri, 12 Aug 2022 09:23:59 +0000 (11:23 +0200)]
DEV: haring: add a simple utility to read file-backed rings
With the ability to back a memory ring into an mmapped file, it makes
sense to be able to dump these files. That's what this utility does.
The entire ring is dumped to stdout. It's well suited to large dumps,
it converts roughly 6 GB of logs per second.
The utility is really meant for developers at the moment. It might
evolve into a more general tool but at the moment it's still possible
that it might need to be run under gdb to process certain crash dumps.
Also at the moment it must not be used on a ring being actively written
to or it will dump garbage.
The code is made so that we can envision later to attach to a live
ring and dump live contents, but this requires that the utility is
built with the exact same options (threads etc), and that the file
is opened read-write. For now these parts have been commented out,
waiting for a reasonably balanced and non-intrusive solution to be
found (e.g. signals must be intercepted so that the tool cannot
leave the ring with a watcher present).
If it is detected that the memory layout of the ring struct differs,
a warning is emitted. At the end, if an error occurs, a warning is
printed as well (this does happen when the process is not cleanly
stopped, but it indicates the end was reached).
Willy Tarreau [Thu, 11 Aug 2022 14:38:20 +0000 (16:38 +0200)]
MINOR: ring: add support for a backing-file
This mmaps a file which will serve as the backing-store for the ring's
contents. The idea is to provide a way to retrieve sensitive information
(last logs, debugging traces) even after the process stops and even after
a possible crash. Right now this was possible by connecting to the CLI
and dumping the contents of the ring live, but this is not handy and
consumes quite a bit of resources before it is needed.
With a backing file, the ring is effectively RAM-mapped file, so that
contents stored there are the same as those found in the file (the OS
doesn't guarantee immediate sync but if the process dies it will be OK).
Note that doing that on a filesystem backed by a physical device is a
bad idea, as it will induce slowdowns at high loads. It's really
important that the device is RAM-based.
Also, this may have security implications: if the file is corrupted by
another process, the storage area could be corrupted, causing haproxy
to crash or to overwrite its own memory. As such this should only be
used for debugging.
Willy Tarreau [Fri, 12 Aug 2022 05:50:43 +0000 (07:50 +0200)]
MINOR: ring: support creating a ring from a linear area
Instead of allocating two parts, one for the ring struct itself and
one for the storage area, ring_make_from_area() will arrange the two
inside the same memory area, with the storage starting immediately
after the struct. This will allow to store a complete ring state in
shared memory areas for example.
BUG/MEDIUM: quic: Wrong use of <token_odcid> in qc_lsntr_pkt_rcv()
This commit was not complete:
"BUG/MEDIUM: quic: Possible use of uninitialized <odcid>
variable in qc_lstnr_params_init()"
<token_odcid> should have been directly passed to qc_lstnr_params_init()
without dereferencing it to prevent haproxy to have new chances to crash!
Willy Tarreau [Thu, 11 Aug 2022 14:12:11 +0000 (16:12 +0200)]
BUG/MEDIUM: ring: fix too lax 'size' parser
It took me a while to figure why a ring declared with "size 1M" was causing
strange effects in a ring, it's just because it's parsed as "1", which is
smaller than the default 16384 size and errors are silently ignored.
This commit tries to address this the best possible way without breaking
existing configs that would work by accident, by warning that the size is
ignored if it's smaller than the current one, and by printing the parsed
size instead of the input string in warnings and errors. This way if some
users have "size 10000" or "size 100k" it will continue to work as 16kB
like today but they will now be aware of it.
In addition the error messages were a bit poor in context in that they
only provided line numbers. The ring name was added to ease locating the
problem.
As the issue was present since day one and was introduced in 2.2 with
commit 99c453df9d ("MEDIUM: ring: new section ring to declare custom ring
buffers."), it could make sense to backport this as far as 2.2, but with
2.2 being quite old now it doesn't seem very reasonable to start emitting
new config warnings in config that apparently worked well.
Thus it looks more reasonable to backport this as far as 2.4.
BUG/MEDIUM: quic: Possible use of uninitialized <odcid> variable in qc_lstnr_params_init()
When receiving a token into a client Initial packet without a cluster secret defined
by configuration, the <odcid> variable used to parse the ODCID from the token
could be used without having been initialized. Such a packet must be dropped. So
the sufficient part of this patch is this check:
+ }
+ else if (!global.cluster_secret && token_len) {
+ /* Impossible case: a token was received without configured
+ * cluster secret.
+ */
+ TRACE_PROTO("Packet dropped", QUIC_EV_CONN_LPKT,
+ NULL, NULL, NULL, qv);
+ goto drop;
}
Take the opportunity of this patch to rework and make it more readable this part
of code where such a packet must be dropped removing the <check_token> variable.
When an ODCID is parsed from a token, new <token_odcid> new pointer variable
is set to the address of the parsed ODCID. This way, is not set but used it will
make crash haproxy. This was not always the case with an uninitialized local
variable.
Adapt the API to used such a pointer variable: <token> boolean variable is removed
from qc_lstnr_params_init() prototype.
Amaury Denoyelle [Thu, 11 Aug 2022 16:22:22 +0000 (18:22 +0200)]
BUG/MEDIUM: mux-quic: fix crash due to invalid trace arg
Traces argument were incorrectly used in qcs_free(). A qcs was specified
as first arg instead of a connection. This will lead to a crash if
developer qmux traces are activated. This is now fixed.
This bug has been introduced with QUIC MUX traces rework. No need to
backport.
Amaury Denoyelle [Wed, 10 Aug 2022 14:58:01 +0000 (16:58 +0200)]
MINOR: mux-quic: define new traces
Add new traces to help debugging on QUIC MUX. Most notable, the
following functions are now traced :
* qcc_emit_cc
* qcs_free
* qcs_consume
* qcc_decode_qcs
* qcc_emit_cc_app
* qcc_install_app_ops
* qcc_release_remote_stream
* qcc_streams_sent_done
* qc_init
Amaury Denoyelle [Wed, 10 Aug 2022 14:42:35 +0000 (16:42 +0200)]
CLEANUP: mux-quic: adjust traces level
Change default devel level for some traces in QUIC MUX:
* proto : used to notify about reception/emission of frames
* state : modification of internal state of connection or streams
* data : detailled information about transfer and flow-control
Amaury Denoyelle [Wed, 10 Aug 2022 14:39:54 +0000 (16:39 +0200)]
MINOR: mux-quic: define protocol error traces
Replace devel traces with error level on all errors situation. Also a
new event QMUX_EV_PROTO_ERR is used. This should help to detect invalid
situations quickly.
BUG/MINOR: quic: Possible infinite loop in quic_build_post_handshake_frames()
This loop is due to the fact that we do not select the next node before
the conditional "continue" statement. Furthermore the condition and the
"continue" statement may be removed after replacing eb64_first() call
by eb64_lookup_ge(): we are sure this condition may not be satisfied.
Add some comments: this function initializes connection IDs with sequence
number 1 upto <max> non included.
Take the opportunity of this patch to remove a "return" wich broke this
traces rule: for any function, do not call TRACE_ENTER() without TRACE_LEAVE()!
Add also TRACE_ERROR() for any encoutered errors.
This lock was there be able to handle the RX packets for a connetion
from several threads. This is no more needed since a QUIC connection
is always handled by the same thread.
Willy Tarreau [Thu, 11 Aug 2022 11:56:42 +0000 (13:56 +0200)]
BUILD: stconn: fix build warning at -O3 about possible null sc
gcc-6.x and 7.x emit build warnings about sc possibly being null upon
return from sc_detach_endp(). This actually is not the case and the
compiler is a little bit overzealous there, but there exists code
paths that can make this analysis non-trivial so let's at least add
a similar BUG_ON() to let both the compiler and the deverloper know
this doesn't happen.
Add a least as much as possible TRACE_ENTER() and TRACE_LEAVE() calls
to any function. Note that some functions do not have any access to the
a quic_conn argument when receiving or parsing datagram at very low level.
In 2.3-dev2, we've added more consistency checks for a number of bug-
inducing programming errors related to the tasks, via commit e5d79bccc
("MINOR: tasks/debug: add a few BUG_ON() to detect use of wrong timer
queue"), and this check comes from there.
The problem that happens here is that when hard-stop-after is set, we
can abort the current thread even if there are still ongoing checks
(or connections in fact). In this case some tasks are present in a
thread's wait queue and are thus bound exclusively to this thread.
During deinit(), the collect and cleanup of all memory areas also
stops servers and kills their check tasks. And calling task_destroy()
does in turn call task_unlink_wq()... except that it's called from
thread 0 which doesn't match the initially planned thread number.
Several approaches are possible. One of them would consist in letting
threads perform their own cleanup (tasks, pools, FDs, etc). This would
possibly be even faster since done in parallel, but some corner cases
might be way more complicated (e.g. who will kill a check's task, or
what to do with a task found in a local wait queue or run queue, and
what about other consistency checks this could violate?).
Thus for now this patches takes an easier and more conservative
approach consisting in admitting that when the process is stopping,
this rule is not necessarily valid, and to let thread 0 collect all
other threads' garbage.
Willy Tarreau [Wed, 10 Aug 2022 15:08:17 +0000 (17:08 +0200)]
BUG/MEDIUM: poller: use fd_delete() to release the poller pipes
The poller pipes needed to communicate between multiple threads are
allocated in init_pollers_per_thread() and released in
deinit_pollers_per_thread(). The former adds them via fd_insert()
so that they are known, but the former only closes them using a
regular close().
This asymmetry represents a problem, because we have in the fdtab[]
an entry for something that may disappear when one thread leaves, and
since these FD numbers are very low, there is a very high likelihood
that they are immediately reassigned to another thread trying to
connect() to a server or just sending a health check. In this case,
the other thread is going to fd_insert() the fd and the recently
added consistency checks will notive that ->owner is not NULL and
will crash. We just need to use fd_delete() here to match fd_insert().
Note that this test was added in 2.7-dev2 by commit 36d9097cf
("MINOR: fd: Add BUG_ON checks on fd_insert()") which was backported
to 2.4 as a safety measure (since it allowed to catch particularly
serious issues). The patch in itself isn't wrong, it just revealed
a long-dormant bug (been there since 1.9-dev1, 4 years ago). As such
the current patch needs to be backported wherever the commit above
is backported.
Many thanks to Christian Ruppert for providing detailed traces in
github issue #1807 and Cedric Paillet for bringing his complementary
analysis that helped to understand the required conditions for this
issue to happen (fast health checks @100ms + randomly long connections
~7s + fast reloads every second + hard-stop-after 5s were necessary
on the dev's machine to trigger it from time to time).
Willy Tarreau [Wed, 10 Aug 2022 05:26:27 +0000 (07:26 +0200)]
BUG/MEDIUM: quic: always remove the connection from the accept list on close
Fred managed to reproduce a crash showing a corrupted accept_list when
firing thousands of concurrent picoquicdemo clients to a same instance.
It may happen if the connection was placed into the accept_list and
immediately closed before being processed (e.g. on error or t/o ?).
In any case the quic_conn_release() function should always detach a
connection to be deleted from any list, like it does for other lists,
so let's add an MT_LIST_DELETE() here.
BUG/MINOR: quic: fix crash on handshake io-cb for null next enc level
When arriving at the handshake completion, next encryption level will be
null on quic_conn_io_cb(). Thus this must be check this before
dereferencing it via qc_need_sending() to prevent a crash.
This was reproduced quickly when browsing over a local nextcloud
instance through QUIC with firefox.
This has been introduced in the current dev with quic-conn Tx
refactoring. No need to backport it.
Considered a stream as opened when receiving a STOP_SENDING frame as the
first frame on the stream.
This patch is tagged as BUG because a BUG_ON may occur if only a
STOP_SENDING frame has been received for a frame. This will reset the
stream in respect with RFC9000 but internally it is considered invalid
transition to reset an idle stream.
To fix this, simply use qcs_idle_open() on STOP_SENDING parsing
function. This will mark the stream as OPEN before resetting it.
This was detected on haproxy.org with the following backtrace :
MINOR: quic: skip sending if no frame to send in io-cb
Check on quic_conn_io_cb() if sending is required. This allows to skip
over Tx buffer allocation if not needed.
To implement this, we check if frame lists on current and next
encryption level are empty. We also need to check if there is no need to
send ACK, PROBE or CONNECTION_CLOSE. This has been isolated in a new
function qc_need_sending() which may be reuse in some other functions in
the future.
MINOR: quic: refactor datagram commit in Tx buffer
This is the final patch on quic-conn Tx refactor. Extend the function
which is used to write a datagram header to save at the same time
written buffer data. This makes sense as the two operations are used at
the same occasion when a pre-written datagram is comitted.
Complete refactor of quic-conn Tx buffer. The buffer is now released
on every send operation completion. This should help to reduce memory
footprint as now Tx buffers are allocated and released on demand.
To simplify allocation/free of quic-conn Tx buffer, two static functions
are created named qc_txb_alloc() and qc_txb_release().
MINOR: quic: replace custom buf on Tx by default struct buffer
On first prototype version of QUIC, emission was multithreaded. To
support this, a custom thread-safe ring-buffer has been implemented with
qring/cbuf.
Now the thread model has been adjusted : a quic-conn is always used on
the same thread and emission is not multi-threaded. Thus, qring/cbuf
usage can be replace by a standard struct buffer.
The code has been simplified even more as for now buffer is always
drained after a prepare/send invocation. This is the case since a
datagram is always considered as sent even on sendto() error. BUG_ON
statements guard are here to ensure that this model is always valid.
Thus, code to handle data wrapping and consume too small contiguous
space with a 0-length datagram is removed.
qc_send_app_pkts() has now a while loop implemented which allows to send
all possible frames even if the send buffer is full between packet
prepare and send. This is present since commit : dc07751ed7ebad10f49081d28a9a5ae785f53d76
MINOR: quic: Send packets as much as possible from qc_send_app_pkts()
This means we can remove code from the MUX which implement this at the
upper layer. This is useful to simplify qc_send_frames() function.
As mentionned commit is subject to backport, this commit should be
backported as well to 2.6.
Willy Tarreau [Tue, 9 Aug 2022 07:08:18 +0000 (09:08 +0200)]
MINOR: debug/memstats: permit to pass the size to free()
Right now the free() call is not intercepted since all this is done
using macros and that would break a lot of stuff. Instead a __free()
macro was provided but never used. In addition it used to only report
a zero size, which is not very convenient.
With this patch comes a better solution. Instead it provides a new
will_free() macro that can be prepended before a call to free(). It
only keeps the counters up to date, and also supports being passed a
size. The pool_free_area() command now uses it, which finally allows
the stats to look correct:
Willy Tarreau [Tue, 9 Aug 2022 06:51:08 +0000 (08:51 +0200)]
MINOR: debug/memstats: automatically determine first column size
The first column's width may vary a lot depending on outputs, and it's
annoying to have large empty columns on small names and mangled large
columns that are not yet large enough. In order to overcome this, this
patch adds a width field to the memstats applet's context, and this
width is calculated the first time the function is entered, by estimating
the width of all lines that will be dumped. This is simple enough and
does the job well. If in the future some filtering criteria are added,
it will still be possible to perform a single pass on everything
depending on the desired output format.
Willy Tarreau [Tue, 9 Aug 2022 06:40:08 +0000 (08:40 +0200)]
MINOR: debug: also store the function name in struct mem_stats
The calling function name is now stored in the structure, and it's
reported when the "all" argument is passed. The first column is
significantly enlarged because some names are really wide :-(
Willy Tarreau [Tue, 9 Aug 2022 06:15:27 +0000 (08:15 +0200)]
MINOR: debug: store and report the pool's name in struct mem_stats
Let's add a generic "extra" pointer to the struct mem_stats to store
context-specific information. When tracing pool_alloc/pool_free, we
can now store a pointer to the pool, which allows to report the pool
name on an extra column. This significantly improves tracing
capabilities.
Willy Tarreau [Tue, 9 Aug 2022 06:09:24 +0000 (08:09 +0200)]
MINOR: debug: make the mem_stats section aligned to void*
Not specifying the alignment will let the linker choose it, and it turns
out that it will not necessarily be the same size as the one chosen for
struct mem_stats, as can be seen if any new fields are added there. Let's
enforce an alignment to void* both for the section and for the structure.
MINOR: quic: Replace pool_zalloc() by pool_malloc() for fake datagrams
These fake datagrams are only used by the low level I/O handler. They
are not provided to the "by connection" datagram handlers. This
is why they are not MT_LIST_APPEND()ed to the listner RX buffer list
(see &quic_dghdlrs[cid_tid].dgrams in quic_lstnr_dgram_dispatch().
Replace the call to pool_zalloc() to by the lighter call to pool_malloc()
and initialize only the ->buf and ->length members. This is safe because
only these fields are inspected by the low level I/O handler.
BUG/MEDIUM: quic: Missing AEAD TAG check after removing header protection
After removing the packet header protection, we can check the packet is long
enough to contain a 16 bytes length AEAD TAG (at this end of the packet).
This test was missing.
MINOR: quic: Too much useless traces in qc_build_frms()
These traces about the available room into the packet currently built and
its payload length could be displayed for each STREAM frame, even for
those which have no chance to be embedded into a packet leading to
very traces to be displayed from a connection with a lot of stream.
This was revealed by traces provide by Tristan in GH #1808
BUG/MEDIUM: quic: Wrong packet length check in qc_do_rm_hp()
When entering this function, we first check the packet length is not too short.
But this was done against the datagram lenght in place of the packet length.
This could lead to the header protection to be removed using data past
the end of the packet (without buffer overflow).
Use the packet length in place of the datagram length which is at <end>
address passed as parameter to this function. As the packet length
is already stored in ->len packet struct member, this <end> parameter is no
more useful.
Willy Tarreau [Sun, 7 Aug 2022 15:28:59 +0000 (17:28 +0200)]
[RELEASE] Released version 2.7-dev3
Released version 2.7-dev3 with the following main changes :
- BUILD: makefile: Fix install(1) handling for OpenBSD/NetBSD/Solaris/AIX
- BUG/MEDIUM: tools: avoid calling dlsym() in static builds (try 2)
- MINOR: resolvers: resolvers_destroy() deinit and free a resolver
- BUG/MINOR: resolvers: shut off the warning for the default resolvers
- BUG/MINOR: ssl: allow duplicate certificates in ca-file directories
- BUG/MINOR: tools: fix statistical_prng_range()'s output range
- BUG/MINOR: quic: do not send CONNECTION_CLOSE_APP in initial/handshake
- BUILD: debug: Add braces to if statement calling only CHECK_IF()
- BUG/MINOR: fd: Properly init the fd state in fd_insert()
- BUG/MEDIUM: fd/threads: fix incorrect thread selection in wakeup broadcast
- MINOR: init: load OpenSSL error strings
- MINOR: ssl: enhance ca-file error emitting
- BUG/MINOR: mworker/cli: relative pid prefix not validated anymore
- BUG/MAJOR: mux_quic: fix invalid PROTOCOL_VIOLATION on POST data overlap
- BUG/MEDIUM: mworker: proc_self incorrectly set crashes upon reload
- BUILD: add detection for unsupported compiler models
- BUG/MEDIUM: stconn: Only reset connect expiration when processing backend side
- BUG/MINOR: backend: Fallback on RR algo if balance on source is impossible
- BUG/MEDIUM: master: force the thread count earlier
- BUG/MAJOR: poller: drop FD's tgid when masks don't match
- DEBUG: fd: detect possibly invalid tgid in fd_insert()
- BUG/MINOR: sockpair: wrong return value for fd_send_uxst()
- MINOR: sockpair: move send_fd_uxst() error message in caller
- Revert "BUG/MINOR: peers: set the proxy's name to the peers section name"
- DEBUG: fd: split the fd check
- MEDIUM: resolvers: continue startup if network is unavailable
- BUG/MINOR: fd: always remove late updates when freeing fd_updt[]
- MINOR: cli: emit a warning when _getsocks was used more than once
- BUG/MINOR: mworker: PROC_O_LEAVING used but not updated
- Revert "MINOR: cli: emit a warning when _getsocks was used more than once"
- MINOR: cli: warning on _getsocks when socket were closed
- BUG/MEDIUM: mux-quic: fix missing EOI flag to prevent streams leaks
- MINOR: quic: Congestion control architecture refactoring
- MEDIUM: quic: Cubic congestion control algorithm implementation
- MINOR: quic: New "quic-cc-algo" bind keyword
- BUG/MINOR: quic: loss time limit variable computed but not used
- MINOR: quic: Stop looking for packet loss asap
- BUG/MAJOR: quic: Useless resource intensive loop qc_ackrng_pkts()
- MINOR: quic: Send packets as much as possible from qc_send_app_pkts()
- BUG/MEDIUM: queue/threads: limit the number of entries dequeued at once
- MAJOR: threads/plock: update the embedded library
- MINOR: thread: provide an alternative to pthread's rwlock
- DEBUG: tools: provide a tree dump function for ebmbtrees as well
- MINOR: ebtree: add ebmb_lookup_shorter() to pursue lookups
- BUG/MEDIUM: pattern: only visit equivalent nodes when skipping versions
- BUG/MINOR: mux-quic: prevent crash if conn released during IO callback
- CLEANUP: mux-quic: remove useless app_ops is_active callback
- BUG/MINOR: mux-quic: do not free conn if attached streams
- MINOR: mux-quic: save proxy instance into qcc
- MINOR: mux-quic: use timeout server for backend conns
- MEDIUM: mux-quic: adjust timeout refresh
- MINOR: mux-quic: count in-progress requests
- MEDIUM: mux-quic: implement http-keep-alive timeout
- MINOR: peers: Add a warning about incompatible SSL config for the local peer
- MINOR: peers: Use a dedicated reconnect timeout when stopping the local peer
- BUG/MEDIUM: peers: limit reconnect attempts of the old process on reload
- BUG/MINOR: peers: Use right channel flag to consider the peer as connected
- BUG/MEDIUM: dns: Properly initialize new DNS session
- BUG/MINOR: backend: Don't increment conn_retries counter too early
- MINOR: server: Constify source server to copy its settings
- REORG: server: Export srv_settings_cpy() function
- BUG/MEDIUM: proxy: Perform a custom copy for default server settings
- BUG/MINOR: quic: Missing in flight ack eliciting packet counter decrement
- BUG/MEDIUM: quic: Floating point exception in cubic_root()
- MINOR: h3: support HTTP request framing state
- MINOR: mux-quic: refresh timeout on frame decoding
- MINOR: mux-quic: refactor refresh timeout function
- MEDIUM: mux-quic: implement http-request timeout
- BUG/MINOR: quic: Avoid sending truncated datagrams
- BUG/MINOR: ring/cli: fix a race condition between the writer and the reader
- BUG/MEDIUM: sink: Set the sink ref for forwarders created during ring parsing
- BUG/MINOR: sink: fix a race condition between the writer and the reader
- BUG/MINOR: quic: do not reject datagrams matching minimum permitted size
- MINOR: quic: Add two new stats counters for sendto() errors
- BUG/MINOR: quic: Missing Initial packet dropping case
- MINOR: quic: explicitely ignore sendto error
- BUG/MINOR: quic: adjust errno handling on sendto
- BUG/MEDIUM: quic: break out of the loop in quic_lstnr_dghdlr
- MINOR: threads: report the number of thread groups in build options
- MINOR: config: automatically preset MAX_THREADS based on MAX_TGROUPS
- BUILD: SSL: allow to pass additional configure args to QUICTLS
- CI: enable weekly "m32" builds on x86_64
- CLEANUP: assorted typo fixes in the code and comments
- BUG/MEDIUM: fix DH length when EC key is used
- REGTESTS: ssl: adopt tests to OpenSSL-3.0.N
- REGTESTS: ssl: adopt tests to OpenSSL-3.0.N
- REGTESTS: ssl: fix grep invocation to use extended regex in ssl_generate_certificate.vtc
- BUILD: cfgparse: always defined _GNU_SOURCE for sched.h and crypt.h
Willy Tarreau [Sun, 7 Aug 2022 14:55:07 +0000 (16:55 +0200)]
BUILD: cfgparse: always defined _GNU_SOURCE for sched.h and crypt.h
_GNU_SOURCE used to be defined only when USE_LIBCRYPT was set. It's also
needed for sched_setaffinity() to be exported. As a side effect, when
USE_LIBCRYPT is not set, a warning is emitted, as Ilya found and reported
in issue #1815. Let's just define _GNU_SOURCE regardless of USE_LIBCRYPT,
and also explicitly add sched.h, as right now it appears to be inherited
from one of the other includes.
dh of length 1024 were chosen for EVP_PKEY_EC key type.
let us pick "default_dh_param" instead.
issue was found on Ubuntu 22.04 which is shipped with OpenSSL configured
with SECLEVEL=2 by default. such SECLEVEL value prohibits DH shorter than
2048:
OpenSSL error[0xa00018a] SSL_CTX_set0_tmp_dh_pkey: dh key too small
better strategy for chosing DH still may be considered though.
Willy Tarreau [Sat, 6 Aug 2022 14:37:27 +0000 (16:37 +0200)]
MINOR: config: automatically preset MAX_THREADS based on MAX_TGROUPS
MAX_THREADS was not changed when setting MAX_TGROUPS, which still limits
some possibilities. Let's preset it to 4 * LONGBITS when MAX_TGROUPS is
larger than 1, or LONGBITS when it's set to 1. This means that the new
default value is 256 threads.
The rationale behind this is that the main use of thread groups is
mostly to address NUMA issues and that we don't necessarily need large
thread counts when using many groups, and 256 threads is already plenty
even on quite large systems.
For now it's important not to go too far because some internal structs
are arrays of MAX_THREADS entries, for example accept_queue_ring, which
is around 8kB per thread. Such structures will need to become dynamic
before defaulting to large thread counts (at 4096 threads max the
accept queues would require 32 MB RAM alone).
Willy Tarreau [Fri, 5 Aug 2022 06:45:56 +0000 (08:45 +0200)]
BUG/MEDIUM: quic: break out of the loop in quic_lstnr_dghdlr
The function processes packets sent by other threads in the current
thread's queue. But if, for any reason, other threads write faster
than the current one processes, this can lead to a situation where
the function never returns.
It seems that it might be what's happening in issue #1808, though
unfortunately, this function is one of the rare without traces. But
the amount of calls to functions like qc_lstnr_pkt_rcv() on a single
thread seems to indicate this possibility.
Thanks to Tristan for his efforts in collecting extremely precious
traces!
qc_snd_buf returned a size_t which means that it was never negative
despite its documentation. Thus the caller who checked for this was
never informed of a sendto error.
Clean this by changing the return value of qc_snd_buf() to an integer.
A 0 is returned on success. Every other values are considered as an
error.
This commit should be backported up to 2.6. Note that to not cause
malfunctions, it must be backported after the previous patch : 906b0589546b700b532472ede019e5c5a8ac1f38
MINOR: quic: explicitely ignore sendto error
This is to ensure that a sendto error does not cause send to be
interrupted which may cause a stalled transfer without a proper retry
mechanism.
The impact of this bug seems null as caller explicitely ignores sendto
error. However this part of code seems to be subject to strange issues
and it may fix them in part. It may be of interest for github issue #1808.
qc_snd_buf() returns an error if sendto has failed. On standard
conditions, we should check for EAGAIN/EWOULDBLOCK errno and if so,
register the file-descriptor in the poller to retry the operation later.
However, quic_conn uses directly the listener fd which is shared for all
QUIC connections of this listener on several threads. Thus, it's
complicated to implement fd supversion via the poller : there is no
mechanism to easily wakeup quic_conn or MUX after a sendto failure.
A quick and simple solution for the moment is to considered a datagram
as properly emitted even on sendto error. In the end, this will trigger
the quic_conn retransmission timer as data will be considered lost on
the network and the send operation will be retried. This solution will
be replaced when fd management for quic_conn is reworked.
In fact, this quick hack was already in use in the current code, albeit
not voluntarily. This is due to a bug caused by an API mismatch on the
return type of qc_snd_buf() which never emits a negative error code
despite its documentation. Thus, all its invocation were considered as a
success. If this bug was fixed, the sending would would have been
interrupted by a break which could cause the transfer to freeze.
qc_snd_buf() invocation is clean up : the break statement is removed.
Send operation is now always explicitely conducted entirely even on
error and buffer data is purged.
A simple optimization has been added to skip over sendto when looping
over several datagrams at the first sendto error. However, to properly
function, it requires a fix on the return type of qc_snd_buf() which is
provided in another patch.
As the behavior before and after this patch seems identical, it is not
labelled as a BUG. However, it should be backported for cleaning
purpose. It may also have an impact on github issue #1808.
Willy Tarreau [Fri, 5 Aug 2022 08:09:32 +0000 (10:09 +0200)]
BUG/MINOR: quic: do not reject datagrams matching minimum permitted size
The dgram length check in quic_get_dgram_dcid() rejects datagrams
matching exactly the minimum allowed length, which doesn't seem
correct. I doubt any useful packet would be that small but better
fix this to avoid confusing debugging sessions in the future.
Willy Tarreau [Thu, 4 Aug 2022 15:18:54 +0000 (17:18 +0200)]
BUG/MINOR: sink: fix a race condition between the writer and the reader
This is the same issue as just fixed in b8e0fb97f ("BUG/MINOR: ring/cli:
fix a race condition between the writer and the reader") but this time
for sinks. They're also sucking the ring and present the same race at
high write loads.
This must be backported to 2.2 as well. See comments in the aforementioned
commit for backport hints if needed.
BUG/MEDIUM: sink: Set the sink ref for forwarders created during ring parsing
A reference to the sink was added in every forwarder by the commit 2ae25ea24
("MINOR: sink: Add a ref to sink in the sink_forward_target structure"). But
this commit is incomplete. It is not performed for the forwarders created
during a ring parsing.
Willy Tarreau [Thu, 4 Aug 2022 15:00:21 +0000 (17:00 +0200)]
BUG/MINOR: ring/cli: fix a race condition between the writer and the reader
The ring's CLI reader unlocks the read side of a ring and relocks it for
writing only if it needs to re-subscribe. But during this time, the writer
might have pushed data, see nobody subscribed hence woken nobody, while
the reader would have left marking that the applet had no more data. This
results in a dump that will not make any forward progress: the ring is
clogged by this reader which believes there's no data and the writer
will never wake it up.
The right approach consists in verifying after re-attaching if the writer
had made any progress in between, and to report that another call is
needed. Note that a jump back to the beginning would also work but here
we provide better fairness between readers this way.
This needs to be backported to 2.2. The applet API needed to signal the
availability of new data changed a few times since then.
There is a remaining loop in this ugly qc_snd_buf() function which could
lead haproxy to send truncated UDP datagrams. For now on, we send
a complete UDP datagram or nothing!
Implement http-request timeout for QUIC MUX. It is used when the
connection is opened and is triggered if no HTTP request is received in
time. By HTTP request we mean at least a QUIC stream with a full header
section. Then qcs instance is attached to a sedesc and upper layer is
then responsible to wait for the rest of the request.
This timeout is also used when new QUIC streams are opened during the
connection lifetime to wait for full HTTP request on them. As it's
possible to demux multiple streams in parallel with QUIC, each waiting
stream is registered in a list <opening_list> stored in qcc with <start>
as timestamp in qcs for the stream opening. Once a qcs is attached to a
sedesc, it is removed from <opening_list>. When refreshing MUX timeout,
if <opening_list> is not empty, the first waiting stream is used to set
MUX timeout.
This is efficient as streams are stored in the list in their creation
order so CPU usage is minimal. Also, the size of the list is
automatically restricted by flow control limitation so it should not
grow too much.
Streams are insert in <opening_list> by application protocol layer. This
is because only application protocol can differentiate streams for HTTP
messaging from internal usage. A function qcs_wait_http_req() has been
added to register a request stream by app layer. QUIC MUX can then
remove it from the list in qc_attach_sc().
As a side-note, it was necessary to implement attach qcc_app_ops
callback on hq-interop module to be able to insert a stream in waiting
list. Without this, a BUG_ON statement would be triggered when trying to
remove the stream on sedesc attach. This is to ensure that every
requests streams are registered for http-request timeout.
MUX timeout is explicitely refreshed on MAX_STREAM_DATA and STOP_SENDING
frame parsing to schedule http-request timeout if a new stream has been
instantiated. It was already done on STREAM parsing due to a previous
patch.
MINOR: mux-quic: refactor refresh timeout function
Try to reorganize qcc_refresh_timeout() to improve its readability. The
main objective is to reduce the indentation level and if sequences by
using goto statement to the end of the function. Also, backend and
frontend code path should be more explicit with this new version.
MINOR: mux-quic: refresh timeout on frame decoding
Refresh the MUX connection timeout in frame parsing functions. This is
necessary as these Rx operation are completed directly from the
quic-conn layer outside of MUX I/O callback. Thus, the timeout should be
refreshed on this occasion.
Note that however on STREAM parsing refresh is only conducted when
receiving the current consecutive data offset.
Timeouts related function have been moved up in the source file to be
able to use them in qcc_decode_qcs().
This commit will be useful for http-request timeout. Indeed, a new
stream may be opened during qcc_decode_qcs() which should trigger this
timeout until a full header section is received and qcs instance is
attached to sedesc.
Store the current step of HTTP message in h3s stream. This reports if we
are in the parsing of headers, content or trailers section. A new enum
h3s_st_req is defined for this.
This field is stored in h3s struct but only used for request stream. It
is left undefined for other streams (control or QPACK streams).
h3_is_frame_valid() has been extended to take into account this state
information. A connection error H3_FRAME_UNEXPECTED is reported if an
invalid frame according to the current state is received; for example a
DATA frame at the beginning of a stream.
BUG/MINOR: quic: Missing in flight ack eliciting packet counter decrement
The decrement was missing in quic_pktns_tx_pkts_release() called each time a
packet number space is discarded. This is not sure this bug could have an impact
during handshakes. This counter is used to cancel the timer used both for packet
detection and PTO, setting its value to null. So there could be retransmissions
or probing which could be triggered for nothing.
BUG/MEDIUM: proxy: Perform a custom copy for default server settings
When a proxy is initialized with the settings of the default proxy, instead
of doing a raw copy of the default server settings, a custom copy is now
performed by calling srv_settings_copy(). This way, all settings will be
really duplicated. Without this deep copy, some pointers are shared between
several servers, leading to UAF, double-free or such bugs.
This patch relies on following commits:
* b32cb9b51 REORG: server: Export srv_settings_cpy() function
* 0b365e3cb MINOR: server: Constify source server to copy its settings
This patch should fix the issue #1804. It must be backported as far as 2.0.
BUG/MINOR: backend: Don't increment conn_retries counter too early
The connection retry counter is incremented too early when a connection
fails. In SC_ST_CER state, errors handling must be performed before
incrementing the counter. Otherwise, we may consider the max connection
attempt is reached while a last one is in fact possible.
projects / thirdparty / haproxy.git / log
