Olivier Houchard [Fri, 11 Jan 2019 17:17:17 +0000 (18:17 +0100)]
MINOR: checks: Store the proxy in checks.
Instead of assuming we have a server, store the proxy directly in struct
check, and use it instead of s->server.
This should be a no-op for now, but will be useful later when we change
mail checks to avoid having a server.
Jarno Huuskonen [Wed, 9 Jan 2019 11:41:19 +0000 (13:41 +0200)]
REGTESTS: test case for map_regm commit 271022150d
Minimal test case for map_regm commit 271022150d7961b9aa39dbfd88e0c6a4bc48c3ee.
Config and test is adapted from: Daniel Schneller's example
(https://www.mail-archive.com/haproxy@formilux.org/msg30523.html).
Willy Tarreau [Fri, 11 Jan 2019 18:38:25 +0000 (19:38 +0100)]
BUG/MAJOR: cache: fix confusion between zero and uninitialized cache key
The cache uses the first 32 bits of the uri's hash as the key to reference
the object in the cache. It makes a special case of the value zero to mean
that the object is not in the cache anymore. The problem is that when an
object hashes as zero, it's still inserted but the eb32_delete() call is
skipped, resulting in the object still being chained in the memory area
while the block has been reclaimed and used for something else. Then when
objects which were chained below it (techically any object since zero is
at the root) are deleted, the walk through the upper object may encounter
corrupted values where valid pointers were expected.
But while this should only happen statically once on 4 billion, the problem
gets worse when the cache-use conditions don't match the cache-store ones,
because cache-store runs with an uninitialized key, which can create objects
that will never be found by the lookup code, or worse, entries with a zero
key preventing eviction of the tree node and resulting in a crash. It's easy
to accidently end up on such a config because the request rules generally
can't be used to decide on the response :
In this test, mixing traffic with /images/$RANDOM and /foo/$RANDOM will
result in random keys being inserted, some of them possibly being zero,
and crashes will quickly happen.
The fix consists in 1) always initializing the transaction's cache_hash
to zero, and 2) never storing a response for which the hash has not been
calculated, as indicated by the value zero.
It is worth noting that objects hashing as value zero will never be cached,
but given that there's only one chance among 4 billion that this happens,
this is totally harmless.
Willy Tarreau [Thu, 10 Jan 2019 09:33:32 +0000 (10:33 +0100)]
BUG/MEDIUM: connection: properly unregister the mux on failed initialization
When mux->init() fails, session_free() will call it again to unregister
it while it was already done, resulting in null derefs or use-after-free.
This typically happens on out-of-memory conditions during H1 or H2 connection
or stream allocation.
BUG/MEDIUM: ssl: Disable anti-replay protection and set max data with 0RTT.
When using early data, disable the OpenSSL anti-replay protection, and set
the max amount of early data we're ready to accept, based on the size of
buffers, or early data won't work with the released OpenSSL 1.1.1.
Daniel Corbett [Wed, 9 Jan 2019 13:13:29 +0000 (08:13 -0500)]
BUG/MEDIUM: init: Initialize idle_orphan_conns for first server in server-template
When initializing server-template all of the servers after the first
have srv->idle_orphan_conns initialized within server_template_init()
The first server does not have this initialized and when http-reuse
is active this causes a segmentation fault when accessed from
srv_add_to_idle_list(). This patch removes the check for
srv->tmpl_info.prefix within server_finalize_init() and allows
the first server within a server-template to have srv->idle_orphan_conns
properly initialized.
BUG/MINOR: lua/htx: Respect the reserve when data are send from an HTX applet
In the function hlua_applet_htx_send_yield(), there already was a test to
respect the reserve but the wrong function was used to get the available space
for data in the HTX buffer. Instead of calling htx_free_space(), the function
htx_free_data_space() must be used. But in fact, there is no reason to bother
with that anymore because the function channel_htx_recv_max() has been added for
this purpose.
The result of this bug is that the call to htx_add_data() failed unexpectedly
while the amount of written data was incremented, leading the applet to think
all data was sent. To prevent any futher bugs, a test has been added to yield if
we are not able to write data into the channel buffer.
Willy Tarreau [Mon, 31 Dec 2018 06:41:24 +0000 (07:41 +0100)]
BUG/CRITICAL: mux-h2: re-check the frame length when PRIORITY is used
Tim Düsterhus reported a possible crash in the H2 HEADERS frame decoder
when the PRIORITY flag is present. A check is missing to ensure the 5
extra bytes needed with this flag are actually part of the frame. As per
RFC7540#4.2, let's return a connection error with code FRAME_SIZE_ERROR.
Many thanks to Tim for responsibly reporting this issue with a working
config and reproducer. This issue was assigned CVE-2018-20615.
BUG/MINOR: proto_htx: Use HTX versions to truncate or erase a buffer
channel_truncate() is not aware of the underlying format of the messages. So if
there are some outgoing data in the channel when called, it does some unexpected
operations on the channel's buffer. So the HTX version, channel_htx_truncate(),
must be used. The same is true for channel_erase(). It resets the buffer but not
the HTX message. So channel_htx_erase() must be used instead. This patch is
flagged as a bug, but as far as we know, it was never hitted.
This patch should be backported to 1.9. If so, following patch must be
backported too:
* MINOR: channel/htx: Add the HTX version of channel_truncate/erase
MINOR: channel/htx: Add the HTX version of channel_truncate/erase
The function channel_htx_truncate() can now be used on HTX buffer to truncate
all incoming data, keeping outgoing one intact. This function relies on the
function channel_htx_erase() and htx_truncate().
This patch may be backported to 1.9. If so, the patch "MINOR: channel/htx: Add
the HTX version of channel_truncate()" must also be backported.
When the reg tests fail, it may be useful to display additional information
coming from varnishtest, especially when this latter aborts.
In such case, the test output may be made of lines prefixed by "* diag"
string.
BUG/MINOR: cache: Disable the cache if any compression filter precedes it
We need to check if any compression filter precedes the cache filter. This is
only possible when the compression is configured in the frontend while the cache
filter is configured on the backend (via a cache-store action or
explicitly). This case cannot be detected during HAProxy startup. So in such
cases, the cache is disabled.
BUG/MINOR: filters: Detect cache+compression config on legacy HTTP streams
On legacy HTTP streams, it is forbidden to use the compression with the
cache. When the compression filter is explicitly specified, the detection works
as expected and such configuration are rejected at startup. But it does not work
when the compression filter is implicitly defined. To fix the bug, the implicit
declaration of the compression filter is checked first, before calling .check()
callback of each filters.
BUG/MINOR: compression: Disable it if another one is already in progress
Since the commit 9666720c8 ("BUG/MEDIUM: compression: Use the right buffer
pointers to compress input data"), the compression can be done twice. The first
time on the frontend and the second time on the backend. This may happen by
configuring the compression in a default section.
To fix the bug, when the response is checked to know if it should be compressed
or not, if the flag HTTP_MSGF_COMPRESSING is set, the compression is not
performed. It means it is already handled by a previous compression filter.
MEDIUM: mux-h1: Clarify how shutr/shutw are handled
Now, h1_shutr() only do a shutdown read and try to set the flag
H1C_F_CS_SHUTDOWN if shutdown write was already performed. On its side,
h1_shutw(), if all conditions are met, do the same for the shutdown write. The
real connection close is done when the mux h1 is released, in h1_release().
The flag H1C_F_CS_SHUTW was renamed to H1C_F_CS_SHUTDOWN to be less ambiguous.
BUG/MINOR: mux-h1: Close connection on shutr only when shutw was really done
In h1_shutr(), to fully close the connection, we must be sure the shutdown write
was already performed on the connection. So we know rely on connection flags
instead of conn_stream flags. If CO_FL_SOCK_WR_SH is already set when h1_shutr()
is called, we can do a full connection close. Otherwise, we just do the shutdown
read.
Without this patch, it is possible to close the connection too early with some
outgoing data in the output buf.
REGTEST: Add a reg test for health-checks over SSL/TLS.
This script runs two tests. One with "httpchk" over SSL/TLS and another
one with "check-ssl" option. As varnishtest does not support SSL/TLS
we use two haproxy processes to run these tests. h2 haproxy process
be2 and be4 backends declare one server each wich are the frontend
of h1 haproxy process. We check the layer6/7 checks thanks to syslog
messages.
PiBa-NL [Sun, 23 Dec 2018 20:06:31 +0000 (21:06 +0100)]
REGTEST: mailers: add new test for 'mailers' section
This test verifies the mailers section works properly by checking that
it sends the proper amount of mails when health-checks are changing and
or marking a server up/down
The test currently fails on all versions of haproxy i tried with varying
results:
- 1.9.0 produces thousands of mails.
- 1.8.14 only sends 1 mail, needs a 200ms 'timeout mail' to succeed
- 1.7.11 only sends 1 mail, needs a 200ms 'timeout mail' to succeed
- 1.6 only sends 1 mail, (does not have the 'timeout mail' setting implemented)
Willy Tarreau [Tue, 8 Jan 2019 08:56:20 +0000 (09:56 +0100)]
DOC: regtest: make it clearer what the purpose of the "broken" series is
The purpose of the "broken" series of reg tests is to integrate scripts
which are known for triggering bugs that are not fixed at the time the
script is merged. These ones are not useful to validate non-regression
after merging a change, but have an important value to help fix the bug
they trigger. This patch updates the description in the Makefile to make
this clearer.
BUG/MINOR: stats/htx: Respect the reserve when the stats page is dumped
As for the cache applet, this one must respect the reserve on HTX streams. This
patch is tagged as MINOR because it is unlikely to fully fill the channel's
buffer. Some tests are already done to not process almost full buffer.
BUG/MEDIUM: cache/htx: Respect the reserve when cached objects are served
It is only true for HTX streams. The legacy code relies on ci_putblk() which is
already aware of the reserve. It is mandatory to not fill the reserve to let
other filters analysing data. It is especially true for the compression
filter. It needs at least 20 bytes of free space, plus at most 5 bytes per 32kB
block. So if the cache fully fills the channel's buffer, the compression will
not have enough space to do its job and it will block the data forwarding,
waiting for more free space. But if the buffer fully filled with input data (ie
no outgoing data), the stream will be frozen infinitely.
This patch must be backported to 1.9. It depends on the following patches:
* BUG/MEDIUM: cache/htx: Respect the reserve when cached objects are served
from the cache
* MINOR: channel/htx: Add HTX version for some helper functions
BUG/MEDIUM: lua: dead lock when Lua tasks are trigerred
When a task is created from Lua context out of initialisation,
the hlua_ctx_init() function can be called from safe environement,
so we must not initialise it. While the support of threads appear,
the safe environment set a lock to ensure only one Lua execution
at a time. If we initialize safe environment in another safe
environmenet, we have a dead lock.
this patch adds the support of the idicator "already_safe" whoch
indicates if the context is initialized form safe Lua fonction.
thank to Flakebi for the report
This patch must be backported to haproxy-1.9 and haproxy-1.8
Willy Tarreau [Mon, 7 Jan 2019 09:38:10 +0000 (10:38 +0100)]
MINOR: stream/cli: report more info about the HTTP messages on "show sess all"
The "show sess all" command didn't allow to detect whether compression
is in use for a given stream, which is sometimes annoying. Let's add a
few more info about the HTTP messages, namely the flags, body len, chunk
len and the "next" pointer.
Willy Tarreau [Mon, 7 Jan 2019 09:10:07 +0000 (10:10 +0100)]
MINOR: stream/cli: fix the location of the waiting flag in "show sess all"
The "waiting" flag indicates if the stream is waiting for some memory,
and was placed on the same output line as the txn for ease of reading.
But since 1.6 the txn is not part of the stream anymore so this output
was placed under a condition, resulting in "waiting" to appear only
when a txn is present. Let's move it upper, closer to the stream's
flags to fix this.
This may safely be backported though it has little value for older
versions.
Willy Tarreau [Mon, 7 Jan 2019 09:01:34 +0000 (10:01 +0100)]
MINOR: stream/htx: add the HTX flags output in "show sess all"
Commit b9af88151 ("MINOR: stream/htx: Add info about the HTX structs in
"show sess all" command") accidently forgot the flags on the request
path, it was only on the response path.
It makes sense to backport this to 1.9 so that both outputs are the same.
Willy Tarreau [Fri, 4 Jan 2019 17:20:32 +0000 (18:20 +0100)]
BUILD: add a new file "version.c" to carry version updates
While testing fixes, it's sometimes confusing to rebuild only one C file
(e.g. a mux) and not to have the correct commit ID reported in "haproxy -v"
nor on the stats page.
This patch adds a new "version.c" file which is always rebuilt. It's
very small and contains only 3 variables derived from the various
version strings. These variables are used instead of the macros at the
few places showing the version. This way the output version of the
running code is always correct for the parts that were rebuilt.
Willy Tarreau [Fri, 4 Jan 2019 16:42:57 +0000 (17:42 +0100)]
BUG/MEDIUM: cli: make "show sess" really thread-safe
This one used to rely on a few spin locks around lists manipulations
only but 1) there were still a few races (e.g. when aborting, or
between STAT_ST_INIT and STAT_ST_LIST), and 2) after last commit
which dumps htx info it became obvious that dereferencing the buffer
contents is not safe at all.
This patch uses the thread isolation from the rendez-vous point
instead, to guarantee that nothing moves during the dump. It may
make the dump a bit slower but it will be 100% safe.
This fix must be backported to 1.9, and possibly to 1.8 which likely
suffers from the short races above, eventhough they're extremely
hard to trigger.
BUG/MEDIUM: server: Defer the mux init until after xprt has been initialized.
In connect_server(), if we're using a new connection, and we have to
initialize the mux right away, only do it so after si_connect() has been
called. si_connect() is responsible for initializing the xprt, and the
mux initialization may depend on the xprt being usable, as it may try to
receive data. Otherwise, the connection will be flagged as having an error,
and we will have to try to connect a second time.
BUG/MEDIUM: h1: In h1_init(), wake the tasklet instead of calling h1_recv().
In h1_init(), instead of calling h1_recv() directly, just wake the tasklet,
so that the receive will be done later.
h1_init() might be called from connect_server(), which is itself called
indirectly from process_stream(), and if the receive fails, we may call
si_cs_process(), which may destroy the channel buffers while process_stream()
still expects them to exist.
BUG/MINOR: cache/htx: Be sure to count partial trailers
When a chunked object is served from the cache, If the trailers are not pushed
in the channel's buffer in one time, we still have to count them in the total
written bytes in the buffer.
BUG/MEDIUM: h1: Get the h1m state when restarting the headers parsing
Since the commit 0f8fb6b7f ("MINOR: h1: make the H1 headers block parser able to
parse headers only"), when headers are not received in one time, a parsing error
is returned because the local state in the function h1_headers_to_hdr_list() was
not initialized with the previous one (in fact, it was not initialized at all).
So now, we start the parsing of headers with the state H1_MSG_HDR_FIRST when the
flag H1_MF_HDRS_ONLY is set. Otherwise, we always get it from the h1m.
Willy Tarreau [Fri, 4 Jan 2019 09:56:26 +0000 (10:56 +0100)]
MEDIUM: mux-h2: emit HEADERS frames when facing HTX trailers blocks
Now the H2 mux will parse and encode the HTX trailers blocks and send
the corresponding HEADERS frame. Since these blocks contain pure H1
trailers which may be fragmented on line boundaries, if first needs
to collect all of them, parse them using the H1 parser, build a list
and finally encode all of them at once once the EOM is met. Note that
this HEADERS frame always carries the end-of-headers and end-of-stream
flags.
This was tested using the helloworld examples from the grpc project,
as well as with the h2c tools. It doesn't seem possible at the moment
to test tailers using varnishtest though.
Willy Tarreau [Fri, 4 Jan 2019 09:48:03 +0000 (10:48 +0100)]
MINOR: h1: make the H1 headers block parser able to parse headers only
Currently the H1 headers parser works for either a request or a response
because it starts from the start line. It is also able to resume its
processing when it was interrupted, but in this case it doesn't update
the list.
Make it support a new flag, H1_MF_HDRS_ONLY so that the caller can
indicate it's only interested in the headers list and not the start
line. This will be convenient to parse H1 trailers.
Willy Tarreau [Fri, 4 Jan 2019 08:28:17 +0000 (09:28 +0100)]
MINOR: mux-h2: make HTX_BLK_EOM processing idempotent
We want to make sure we won't emit another empty DATA frame if we meet
HTX_BLK_EOM after and end of stream was already sent. For now it cannot
happen as far as HTX is respected, but with trailers it may become
ambiguous.
Willy Tarreau [Thu, 3 Jan 2019 20:27:19 +0000 (21:27 +0100)]
BUG/MEDIUM: mux-h1: don't enforce chunked encoding on requests
Recent commit 4710d20 ("BUG/MEDIUM: mux-h1: make HTX chunking
consistent with H2") tried to address chunking inconsistencies between
H1/HTX/H2 and has enforced it on every outgoing message carrying
H1_MF_XFER_LEN without H1_MF_CLEN nor H1_MF_CHNK. But it also does it
on requests, which is not appropriate since a request by default
doesn't have a message body unless explicitly mentioned. Also make
sure we only do this on HTTP/1.1 messages.
The problem is to guarantee the highest level of compatibility between
H1/H1, H1/H2, H2/H1 in each direction regarding the lack of content-
length. We have this truth table (a star '*' indicates which one can
pass trailers) :
For H1 client to H2 server, it will be possible to rely on the presence
of "TE: trailers" in the H1 request to automatically switch to chunks
in the response, and be able to pass trailers at the end. For now this
check is not implemented so an H2 response missing a content-length to
an H1 request will always have a transfer-encoding header added and
trailers will be forwarded if any.
This patch depends on previous commit "MINOR: mux-h1: parse the
content-length header on output and set H1_MF_CLEN" to work properly.
Since the aforementioned commit is scheduled for backport to 1.9 this
commit must also be backported to 1.9.
Willy Tarreau [Thu, 3 Jan 2019 20:52:42 +0000 (21:52 +0100)]
MINOR: mux-h1: parse the content-length header on output and set H1_MF_CLEN
The H1_MF_CLEN flag is needed to figure whether a content-length header is
present or not when producing a request, so let's check it on output just
like we already check the transfer-encoding header.
Willy Tarreau [Thu, 3 Jan 2019 17:39:54 +0000 (18:39 +0100)]
MINOR: h2: add h2_make_htx_trailers to turn H2 headers to HTX trailers
This function is usable to transform a list of H2 header fields to a
HTX trailers block. It takes care of rejecting forbidden headers and
pseudo-headers when performing the conversion. It also emits the
trailing CRLF that is currently needed in the HTX trailers block.
Willy Tarreau [Thu, 3 Jan 2019 15:18:34 +0000 (16:18 +0100)]
MEDIUM: mux-h2: pass trailers to H1 (legacy mode)
When forwarding an H2 request to an H1 server, if the request doesn't
have a content-length header field, it is chunked. In this case it is
possible to send trailers to the server, which is what this patch does.
If the transfer is performed without chunking, then the trailers are
silently discarded.
Willy Tarreau [Thu, 3 Jan 2019 15:18:14 +0000 (16:18 +0100)]
MINOR: h2: add h2_make_h1_trailers to turn H2 headers to H1 trailers
This function is usable to transform a list of H2 header fields to a
H1 trailers block. It takes care of rejecting forbidden headers and
pseudo-headers when performing the conversion.
Willy Tarreau [Wed, 2 Jan 2019 18:38:14 +0000 (19:38 +0100)]
BUG/MEDIUM: mux-h2: decode trailers in HEADERS frames
This is not exactly a bug but a long-time design limitation. We used not
to decode trailers in H2, resulting in broken connections each time a
trailer was sent, since it was impossible to keep the HPACK decompressor
synchronized. Now that the sequencing of operations permits it, we must
make sure to at least properly decode them.
What we try to do is to identify if a HEADERS frame was already seen and
use this indication to know if it's a headers or a trailers. For this,
h2c_decode_headers() checks if the stream indicates that a HEADERS frame
was already received. If so, it decodes it and emits the trailing
0 CRLF CRLF in case of H1, or the HTX_EOD + HTX_EOM blocks in case of HTX,
to terminate the data stream.
The trailers contents are still deleted for now but the request works, and
the connection remains synchronized and usable for subsequent streams.
The correctness may be tested using a simple config and h2spec :
This should definitely be backported to 1.9 given the low impact for the
benefit. However it cannot be backported to 1.8 since the operations cannot
be resumed. The following patches are also needed with this one :
MINOR: mux-h2: make h2c_decode_headers() return a status, not a count
MINOR: mux-h2: add a new dummy stream : h2_error_stream
MEDIUM: mux-h2: make h2c_decode_headers() support recoverable errors
BUG/MINOR: mux-h2: detect when the HTX EOM block cannot be added after headers
MINOR: mux-h2: check for too many streams only for idle streams
MINOR: mux-h2: set H2_SF_HEADERS_RCVD when a HEADERS frame was decoded
Willy Tarreau [Wed, 2 Jan 2019 12:59:43 +0000 (13:59 +0100)]
MINOR: mux-h2: check for too many streams only for idle streams
The HEADERS frame parser checks if we still have too many streams, but
this should only be done for idle streams, otherwise it would prevent
us from processing trailer frames.
Willy Tarreau [Thu, 3 Jan 2019 10:48:45 +0000 (11:48 +0100)]
CLEANUP: mux-h2: clean the stream error path on HEADERS frame processing
In h2c_frt_handle_headers() and h2c_bck_handle_headers() we have an unused
error path made of the strm_err label, while send_rst is used to emit an
RST upon stream error after forcing the stream to h2_refused_stream. Let's
remove this unused strm_err block now.
Willy Tarreau [Thu, 3 Jan 2019 10:41:50 +0000 (11:41 +0100)]
MINOR: mux-h2: remove a misleading and impossible test
In h2c_frt_handle_headers(), we test the stream for SS_ERROR just after
setting it to SS_OPEN, this makes no sense and creates confusion in the
error path. Remove this misleading test.
Willy Tarreau [Thu, 3 Jan 2019 13:50:36 +0000 (14:50 +0100)]
BUG/MINOR: mux-h2: detect when the HTX EOM block cannot be added after headers
In case we receive a very large HEADERS frame which doesn't leave enough
room to place the EOM block after the decoded headers, we must fail the
stream. This test was missing, resulting in the loss of the EOM, possibly
leaving the stream waiting for a time-out.
Note that we also clear h2c->dfl here so that we don't attempt to clear
it twice when going back to the demux.
If this is backported to 1.9, it also requires that the following patches
are backported as well :
MINOR: mux-h2: make h2c_decode_headers() return a status, not a count
MINOR: mux-h2: add a new dummy stream : h2_error_stream
MEDIUM: mux-h2: make h2c_decode_headers() support recoverable errors
Willy Tarreau [Thu, 3 Jan 2019 13:48:18 +0000 (14:48 +0100)]
MEDIUM: mux-h2: make h2c_decode_headers() support recoverable errors
When a decoding error is recoverable, we should emit a stream error and
not a connection error. This patch does this by carefully checking the
connection state before deciding to send a connection error. If only the
stream is in error, an RST_STREAM is sent.
Willy Tarreau [Wed, 2 Jan 2019 14:36:11 +0000 (15:36 +0100)]
MINOR: mux-h2: make h2c_decode_headers() return a status, not a count
This function used to return a byte count for the output produced, or
zero on failure. Not only this value is not used differently than a
boolean, but it prevents us from returning stream errors when a frame
cannot be extracted because it's too large, or from parsing a frame
and producing nothing on output.
This patch modifies its API to return <0 on errors, 0 on inability to
proceed, or >0 on success, irrelevant to the amount of output data.
BUG/MEDIUM: mux-h1: Add a task to handle connection timeouts
The mux h1 mainly depends on the stream to handle errors and timeouts. But,
there is one unhandled case. If a timeout occurred when some outgoing data are
blocked in the output buffer, the stream is detached and the mux waits infinitly
the data are gone before closing the connection. To fix the bug, a task has been
added on the mux to handle connection timeouts. For now, a expiration date is
set only when some outgoing data are blocked. And if a stream is still attached
when the mux's task timed out, an error flag is set on the mux but the
connection is not closed immediatly. We assume the stream will hit the same
timeout just after.
BUG/MEDIUM: proto-htx: Set SI_FL_NOHALF on server side when request is done
In the function htx_end_request, the flag SI_FL_NOHALF must be set on the server
side once the request is in the state HTTP_MSG_DONE. But the response state was
checked before and the flag was only set when the response was also in the state
HTTP_MSG_DONE. Of course, it is not desirable.
BUG/MAJOR: stream-int: Update the stream expiration date in stream_int_notify()
Since a long time, the expiration date of a stream is only updated in
process_stream(). It is calculated, among others, using the channels expiration
dates for reads and writes (.rex and .wex values). But these values are updated
by the stream-interface. So when this happens at the connection layer, the
update is only done if the stream's task is woken up. Otherwise, the stream
expiration date is not immediatly updated. This leads to unexpected
behaviours. Time to time, users reported that the wrong timeout was hitted or
the wrong termination state was reported. This is partly because of this
bug.
Recently, we observed some blocked sessions for a while when big objects are
served from the cache applet. It seems only concern the clients not reading the
response. Because delivered objects are big, not all data can be sent. And
because delivered objects are big, data are fast forwarded (from the input to
the output with no stream wakeup). So in such situation, the stream expiration
date is never updated and no timeout is hitted. The session remains blocked
while the client remains connected.
This bug exists at least since HAProxy 1.5. But recent changes on the connection
layer make it more visible. It must be backported from 1.9 to 1.6. And with more
pain it should be backported to 1.5.
Willy Tarreau [Thu, 3 Jan 2019 16:39:54 +0000 (17:39 +0100)]
BUG/MEDIUM: mux-h1: make HTX chunking consistent with H2
When transfering from H1 to H1, chunking is always indicated by the
presence of the Transfer-encoding header field. But when a message
comes from H2 there is no such header and only HTX_SL_F_XFER_LEN
ought to be relied on. This one will also result in H1_MF_XFER_LEN
to be set, just like transfer-encoding, so let's always rely on
this latter flag to detect the need for chunking (when CLEN is not
here) and automatically add the transfer-encoding header if it was
not present, as reported by H1_MF_CHNK.
Willy Tarreau [Thu, 3 Jan 2019 16:46:56 +0000 (17:46 +0100)]
BUG/MEDIUM: mux-h1: use per-direction flags to indicate transitions
The H1 mux needs to store some information regarding the states that
were met (EOD, trailers, etc) for each direction but currently uses
only one set of flags. This results in failures when both the request
and the response use chunked-encoding because some elements are believed
to have been met already and a trailing 0 CRLF or just a CRLF may be
missing at the end.
The solution here consists in splitting these flags per direction, one
set for input processing and another set for output processing. Only
two flags were affected so this is not a big deal.
Willy Tarreau [Thu, 3 Jan 2019 09:26:23 +0000 (10:26 +0100)]
BUG/MINOR: mux-h2: only update rxbuf's length for H1 headers
In h2c_decode_headers() we update the buffer's length according to the
amount of data produced (outlen). But in case of HTX this outlen value
is not a quantity, just an indicator of success, resulting in the buffer
being added one extra byte and temporarily showing .data > .size, which
is wrong. Fortunately this is overridden when leaving the function by
htx_to_buf() so the impact only exists in step-by-step debugging, but
it definitely needs to be fixed.
Willy Tarreau [Thu, 3 Jan 2019 08:32:20 +0000 (09:32 +0100)]
BUG/MINOR: mux-h2: mark end-of-stream after processing response HEADERS, not before
When dealing with a server's H2 response, we used to set the
end-of-stream flag on the conn_stream and the stream before parsing
the response, which is incorrect since we can fail to process this
response by lack of room, buffer or anything. The extend of this problem
is still limited to a few rare cases, but with trailers it will cause a
systematic failure.
Willy Tarreau [Thu, 3 Jan 2019 08:20:05 +0000 (09:20 +0100)]
BUG/MINOR: mux-h2: don't check the CS count in h2c_bck_handle_headers()
This function handles response HEADERS frames, it is not responsible
for creating new streams thus it must not check if we've reached the
stream count limit, otherwise it could lead to some undesired pauses
which bring no benefit.
Willy Tarreau [Thu, 3 Jan 2019 07:52:09 +0000 (08:52 +0100)]
BUG/MINOR: mux-h2: set the stream-full flag when leaving h2c_decode_headers()
If we exit this function because some data are pending in the rxbuf, we
currently don't indicate any blocking flag, which will prevent the operation
from being attempted again. Let's set H2_CF_DEM_SFULL in this case to indicate
there's not enough room in the stream buffer so that the operation may be
attempted again once we make room. It seems that this issue cannot be
triggered right now but it definitely will with trailers.
This fix should be backported to 1.9 for completeness.
Willy Tarreau [Thu, 3 Jan 2019 07:27:41 +0000 (08:27 +0100)]
BUG/MEDIUM: mux-h2: always restart reading if data are available
h2c_restart_reading() is used at various place to resume processing of
demux data, but this one refrains from doing so if the mux is already
subscribed for receiving. It just happens that even if some incoming
frame processing is interrupted, the mux is always subscribed for
receiving, so this condition alone is not enough, it must be combined
with the fact that the demux buffer is empty, otherwise some resume
events are lost. This typically happens when we refrain from processing
some incoming data due to missing room in the stream's rxbuf, and want
to resume in h2c_rcv_buf(). It will become even more visible with trailers
since these ones want to have an empty rxbuf before proceeding.
Willy Tarreau [Thu, 3 Jan 2019 07:10:14 +0000 (08:10 +0100)]
CLEANUP: mux-h2: fix end-of-stream flag name when processing headers
In h2c_decode_headers() we mistakenly check for H2_F_DATA_END_STREAM
while we should check for H2_F_HEADERS_END_STREAM. Both have the same
value (1) but better stick to the correct flag.
BUG/MAJOR: htx: Return the good block address after a defrag
When an HTX structure is defragmented, it is possible to retrieve the new block
corresponding to an old one. This is useful to do a defrag during a loop on
blocks, to be sure to continue looping on the good block. But, instead of
returning the address of the new block in the HTX structure, the one in the
temporary structure used to do the defrag was returned, leading to unexpected
behaviours.
BUG/MEDIUM: cache: Be sure to end the forwarding when XFER length is unknown
This bug exists in the HTX code and in the legacy one. When the body length is
unknown, the applet hangs. For the legacy code, it hangs because the end of the
cached object is not correctly handled and the applet is never recalled. For the
HTX code, only the begining of the response (the 1st buffer) is sent then the
applet hangs. To work in HTX, The fast forwarding must be correctly handled.
This patch must be backported to 1.9.
[cf: the patch adding the function channel_add_input must be backported with
this one. It does not exist in 1.8 because only responses with a C-L are cached.]
MINOR: stats/htx: Call channel_add_input instead of updating channel state by hand
This way we are sure the channel state is always correctly upadated, especially
the amount of data directly forwarded. For the stats applet, it is not a bug
because the fast forwarding is never used (the response is chunked and the HTX
extra field is always set to 0).
MINOR: channel: Add the function channel_add_input
This function must be called when new incoming data are pushed in the channel's
buffer. It updates the channel state and take care of the fast forwarding by
consuming right amount of data and decrementing "->to_forward" accordingly when
necessary. In fact, this patch just moves a part of ci_putblk in a dedicated
function.
Willy Tarreau [Wed, 2 Jan 2019 19:09:33 +0000 (20:09 +0100)]
BUG/MEDIUM: log: don't mark log FDs as non-blocking on terminals
With the new ability to log to a terminal, it's convenient to be able
to use "log stdout" in a config file, except that it now results in
setting the terminal to non-blocking mode, breaking every utility
relying on stdin afterwards. Since the only reason for logging to a
terminal is to debug, do not set the FD to non-blocking mode when it's
a terminal.
Alex Zorin [Sun, 30 Dec 2018 02:56:28 +0000 (13:56 +1100)]
MINOR: payload: add sample fetch for TLS ALPN
Application-Layer Protocol Negotiation (ALPN, RFC7301) is a TLS
extension which allows a client to present a preference for which
protocols it wishes to connect to, when a single port supports multiple
multiple application protocols.
It allows a transparent proxy to take a decision based on the beginning
of an SSL/TLS stream without deciphering it.
The new fetch "req.ssl_alpn" extracts the ALPN protocol names that may
be present in the ClientHello message.
Olivier Houchard [Fri, 28 Dec 2018 17:50:57 +0000 (18:50 +0100)]
MEDIUM: sessions: Keep track of which connections are idle.
Instead of keeping track of the number of connections we're responsible for,
keep track of the number of connections we're responsible for that we are
currently considering idling (ie that we are not using, they may be in use
by other sessions), that way we can actually reuse connections when we have
more connections than the max configured.
Olivier Houchard [Fri, 28 Dec 2018 15:20:25 +0000 (16:20 +0100)]
MEDIUM: servers: Be smarter when switching connections.
When connecting to a server, and reusing a connection, always attempt to give
the owner of the previous session one of its own connections, so that one
session won't be responsible for too many connections.
Olivier Houchard [Thu, 27 Dec 2018 17:59:46 +0000 (18:59 +0100)]
BUG/MEDIUM: servers: Flag the stream_interface on handshake error.
When creating a new outgoing connection, if we're using ALPN and waiting
for the handshake completion to choose the mux, and for some reason the
handshake failed, add the SI_FL_ERR flag to the stream_interface, so that
process_streams() knows the connection failed, and can attempt to retry,
instead of just hanging.
Olivier Houchard [Thu, 27 Dec 2018 16:20:54 +0000 (17:20 +0100)]
BUG/MAJOR: sessions: Use an unlimited number of servers for the conn list.
When a session adds a connection to its connection list, we used to remove
connections for an another server if there were not enough room for our
server. This can't work, because those lists are now the list of connections
we're responsible for, not just the idle connections.
To fix this, allow for an unlimited number of servers, instead of using
an array, we're now using a linked list.
Olivier Houchard [Thu, 27 Dec 2018 14:35:22 +0000 (15:35 +0100)]
BUG/MAJOR: servers: Correctly use LIST_ELEM().
To access the first element of the list, correctly use LIST_ELEM(), or we
end up getting the head of the list, instead of getting the first connection.
Olivier Houchard [Thu, 27 Dec 2018 14:29:53 +0000 (15:29 +0100)]
BUG/MAJOR: servers: Use the list api correctly to avoid crashes.
In connect_server(), if we looked for an usable connection and failed to
find one, srv_conn won't be NULL at the end of list_for_each_entry(), but
will point to the head of a list, which is not a pointer to a struct
connection, so explicitely set it to NULL.
Olivier Houchard [Fri, 28 Dec 2018 13:45:47 +0000 (14:45 +0100)]
BUG/MEDIUM: servers: Fail if we fail to allocate a conn_stream.
If, for some reason we failed to allocate a conn_stream when reusing an
existing connection, set srv_conn to NULL, so that we fail later, instead
of pretending all is right. This ends up giving a stream_interface with
no endpoint, and so the stream will never end.
Olivier Houchard [Fri, 28 Dec 2018 13:44:41 +0000 (14:44 +0100)]
BUG/MEDIUM: mux_h2: Don't add to the idle list if we're full.
In h2_detach(), don't add the connection to the idle list if nb_streams
is at the max. This can happen if we already closed that stream before, so
its slot became available and was used by another stream.
Jérôme Magnin [Fri, 28 Dec 2018 13:49:08 +0000 (14:49 +0100)]
BUG/MINOR: htx: send the proper authenticate header when using http-request auth
When we use htx and http-request auth rules, we need to send WWW-Authenticate
with a 401 and Proxy-Authenticate with a 407. We only sent Proxy-Authenticate
regardless of status, with htx enabled.
Olivier Houchard [Mon, 24 Dec 2018 12:32:13 +0000 (13:32 +0100)]
BUG/MEDIUM: servers: Don't try to reuse connection if we switched server.
In connect_server(), don't attempt to reuse the old connection if it's
targetting a different server than the one we're supposed to access, or
we will never be able to connect to a server if the first one we tried failed.
Willy Tarreau [Sat, 22 Dec 2018 19:19:26 +0000 (20:19 +0100)]
MEDIUM: mux-h2: handle decoding of CONTINUATION frames
Now that the HEADERS frame decoding is retryable, we can safely try to
fold CONTINUATION frames into a HEADERS frame when the END_OF_HEADERS
flag is missing. In order to do this, h2c_decode_headers() moves the
frames payloads in-situ and leaves a hole that is plugged when leaving
the function. There is no limit to the number of CONTINUATION frames
handled this way provided that all of them fit into the buffer. The
error reported when meeting isolated CONTINUATION frames has now changed
from INTERNAL_ERROR to PROTOCOL_ERROR.
Now there is only one (unrelated) remaining failure in h2spec.
projects / thirdparty / haproxy.git / log
