Willy Tarreau [Tue, 6 May 2014 20:57:53 +0000 (22:57 +0200)]
BUG/MEDIUM: http/session: disable client-side expiration only after body
For a very long time, back in the v1.3 days, we used to rely on a trick
to avoid expiring the client side while transferring a payload to the
server. The problem was that if a client was able to quickly fill the
buffers, and these buffers took some time to reach the server, the
client should not expire while not sending anything.
In order to cover this situation, the client-side timeout was disabled
once the connection to the server was OK, since it implied that we would
at least expire on the server if required.
But there is a drawback to this : if a client stops uploading data before
the end, its timeout is not enforced and we only expire on the server's
timeout, so the logs report a 504.
Since 1.4, we have message body analysers which ensure that we know whether
all the expected data was received or not (HTTP_MSG_DATA or HTTP_MSG_DONE).
So we can fix this problem by disabling the client-side or server-side
timeout at the end of the transfer for the respective side instead of
having it unconditionally in session.c during all the transfer.
With this, the logs now report the correct side for the timeout. Note that
this patch is not enough, because another issue remains : the HTTP body
forwarders do not abort upon timeout, they simply rely on the generic
handling from session.c. So for now, the session is still aborted when
reaching the server timeout, but the culprit is properly reported. A
subsequent patch will address this specific point.
This bug was tagged MEDIUM because of the changes performed. The issue
it fixes is minor however. After some cooling down, it may be backported
to 1.4.
It was reported by and discussed with Rachel Chavez and Patrick Hemmer
on the mailing list.
MINOR: ssl: convert to binary ssl_fc_unique_id and ssl_bc_unique_id.
Previously ssl_fc_unique_id and ssl_bc_unique_id return a string encoded
in base64 of the RFC 5929 TLS unique identifier. This patch modify those fetches
to return directly the ID in the original binary format. The user can make the
choice to encode in base64 using the converter.
MINOR: ssl: adds ssl_f_sha1 fetch to return frontend's certificate fingerprint
ssl_f_sha1 is a binary binary fetch used to returns the SHA-1 fingerprint of
the certificate presented by the frontend when the incoming connection was
made over an SSL/TLS transport layer. This can be used to know which
certificate was chosen using SNI.
MINOR: ssl: adds fetchs and ACLs for ssl back connection.
Adds ssl fetchs and ACLs for outgoinf SSL/Transport layer connection with their
docs:
ssl_bc, ssl_bc_alg_keysize, ssl_bc_cipher, ssl_bc_protocol, ssl_bc_unique_id,
ssl_bc_session_id and ssl_bc_use_keysize.
BUG/MAJOR: http: connection setup may stall on balance url_param
On the mailing list, seri0528@naver.com reported an issue when
using balance url_param or balance uri. The request would sometimes
stall forever.
Cyril Bonté managed to reproduce it with the configuration below :
listen test :80
mode http
balance url_param q
hash-type consistent
server s demo.1wt.eu:80
and found it appeared with this commit : 80a92c0 ("BUG/MEDIUM: http:
don't start to forward request data before the connect").
The bug is subtle but real. The problem is that the HTTP request
forwarding analyzer refrains from starting to parse the request
body when some LB algorithms might need the body contents, in order
to preserve the data pointer and avoid moving things around during
analysis in case a redispatch is later needed. And in order to detect
that the connection establishes, it watches the response channel's
CF_READ_ATTACHED flag.
The problem is that a request analyzer is not subscribed to a response
channel, so it will only see changes when woken for other (generally
correlated) reasons, such as the fact that part of the request could
be sent. And since the CF_READ_ATTACHED flag is cleared once leaving
process_session(), it is important not to miss it. It simply happens
that sometimes the server starts to respond in a sequence that validates
the connection in the middle of process_session(), that it is detected
after the analysers, and that the newly assigned CF_READ_ATTACHED is
not used to detect that the request analysers need to be called again,
then the flag is lost.
The CF_WAKE_WRITE flag doesn't work either because it's cleared upon
entry into process_session(), ie if we spend more than one call not
connecting.
Thus we need a new flag to tell the connection initiator that we are
specifically interested in being notified about connection establishment.
This new flag is CF_WAKE_CONNECT. It is set by the requester, and is
cleared once the connection succeeds, where CF_WAKE_ONCE is set instead,
causing the request analysers to be scanned again.
For future versions, some better options will have to be considered :
- let all analysers subscribe to both request and response events ;
- let analysers subscribe to stream interface events (reduces number
of useless calls)
- change CF_WAKE_WRITE's semantics to persist across calls to
process_session(), but that is different from validating a
connection establishment (eg: no data sent, or no data to send)
The bug was introduced in 1.5-dev23, no backport is needed.
Commit fc6c032 ("MEDIUM: global: add support for CPU binding on Linux ("cpu-map")")
merged into 1.5-dev13 involves a useless test that clang reports as a warning. The
"low" variable cannot be negative here. Issue reported by Charles Carter.
BUG/MINOR: auth: fix wrong return type in pat_match_auth()
Commit 5338eea ("MEDIUM: pattern: The match function browse itself the
list or the tree") changed the return type of pattern matching functions.
One enum was left over in pat_match_auth(). Fortunately, this one equals
zero where a null pointer is expected, so it's cast correctly.
This detected and reported by Charles Carter was introduced in 1.5-dev23,
no backport is needed.
MEDIUM: http: make http-request rules processing return a verdict instead of a rule
Till now we used to return a pointer to a rule, but that makes it
complicated to later add support for registering new actions which
may fail. For example, the redirect may fail if the response is too
large to fit into the buffer.
So instead let's return a verdict. But we needed the pointer to the
last rule to get the address of a redirect and to get the realm used
by the auth page. So these pieces of code have moved into the function
and they produce a verdict.
MEDIUM: http: factorize the "auth" action of http-request and stats
Both use exactly the same mechanism, except for the choice of the
default realm to be emitted when none is selected. It can be achieved
by simply comparing the ruleset with the stats' for now. This achieves
a significant code reduction and further, removes the dependence on
the pointer to the final rule in the caller.
MEDIUM: http: emulate "block" rules using "http-request" rules
The "block" rules are redundant with http-request rules because they
are performed immediately before and do exactly the same thing as
"http-request deny". Moreover, this duplication has led to a few
minor stats accounting issues fixed lately.
Instead of keeping the two rule sets, we now build a list of "block"
rules that we compile as "http-request block" and that we later insert
at the beginning of the "http-request" rules.
The only user-visible change is that in case of a parsing error, the
config parser will now report "http-request block rule" instead of
"blocking condition".
MEDIUM: http: remove even more of the spaghetti in the request path
Some of the remaining interleaving of request processing after the
http-request rules can now safely be removed, because all remaining
actions are mutually exclusive.
So we can move together all those related to an intercepting rule,
then proceed with stats, then with req*.
We still keep an issue with stats vs reqrep which forces us to
keep the stats split in two (detection and action). Indeed, from the
beginning, stats are detected before rewriting and not after. But a
reqdeny rule would stop stats, so in practice we have to first detect,
then perform the action. Maybe we'll be able to kill this in version
1.6.
Till now the Connection header was processed in the middle of the http-request
rules and some reqadd rules. It used to force some http-request actions to be
cut in two parts.
Now with keep-alive, not only that doesn't make any sense anymore, but it's
becoming a total mess, especially since we need to know the headers contents
before proceeding with most actions.
The real reason it was not moved earlier is that the "block" or "http-request"
rules can see a different version if some fields are changed there. But that
is already not reliable anymore since the values observed by the frontend
differ from those in the backend.
This patch is the equivalent of commit f118d9f ("REORG: http: move HTTP
Connection response header parsing earlier") but for the request side. It
has been tagged MEDIUM as it could theorically slightly affect some setups
relying on corner cases or invalid setups, though this does not make real
sense and is highly unlikely.
BUG/MINOR: http: block rules forgot to increment the session's request counter
The session's backend request counters were incremented after the block
rules while these rules could increment the session's error counters,
meaning that we could have more errors than requests reported in a stick
table! Commit 5d5b5d8 ("MEDIUM: proto_tcp: add support for tracking L7
information") is the most responsible for this.
This bug is 1.5-specific and does not need any backport.
BUG/MINOR: http: block rules forgot to increment the denied_req counter
"block" rules used to build the whole response and forgot to increment
the denied_req counters. By jumping to the general "deny" label created
in previous patch, it's easier to fix this.
The issue was already present in 1.3 and remained unnoticed, in part
because few people use "block" nowadays.
MEDIUM: http: jump to dedicated labels after http-request processing
Continue the cleanup of http-request post-processing to remove some
of the interleaved tests. Here we set up a few labels to deal with
the deny and tarpit actions and avoid interleaved ifs.
MEDIUM: http: move reqadd after execution of http_request redirect
We still have a plate of spaghetti in the request processing rules.
All http-request rules are executed at once, then some responses are
built interlaced with other rules that used to be there in the past.
Here, reqadd is executed after an http-req redirect rule is *decided*,
but before it is *executed*.
So let's match the doc and config checks, to put the redirect actually
before the reqadd completely.
MINOR: http: rely on the message body parser to send 100-continue
There's no point in open-coding the sending of 100-continue in
the stats initialization code, better simply rely on the function
designed to process the message body which already does it.
Commit 844a7e7 ("[MEDIUM] http: add support for proxy authentication")
merged in v1.4-rc1 added the ability to emit a status code 407 in auth
responses, but forgot to set the same status in the logs, which still
contain 401.
BUG/MINOR: proxy: unsafe initialization of HTTP transaction when switching from TCP frontend
A switch from a TCP frontend to an HTTP backend initializes the HTTP
transaction. txn->hdr_idx.size is used by hdr_idx_init() but not
necessarily initialized yet here, because the first call to hdr_idx_init()
is in fact placed in http_init_txn(). Moving it before the call is
enough to fix it. We also remove the useless extra confusing call
to hdr_idx_init().
The bug was introduced in 1.5-dev8 with commit ac1932d ("MEDIUM:
tune.http.maxhdr makes it possible to configure the maximum number
of HTTP headers"). No backport to stable is needed.
BUG/MEDIUM: patterns: last fix was still not enough
Last fix did address the issue for inlined patterns, but it was not
enough because the flags are lost as well when updating patterns
dynamically over the CLI.
Also if the same file was used once with -i and another time without
-i, their references would have been merged and both would have used
the same matching method.
It's appear that the patterns have two types of flags. The first
ones are relative to the pattern matching, and the second are
relative to the pattern storage. The pattern matching flags are
the same for all the patterns of one expression. Now they are
stored in the expression. The storage flags are information
returned by the pattern mathing function. This information is
relative to each entry and is stored in the "struct pattern".
Now, the expression matching flags are forwarded to the parse
and index functions. These flags are stored during the
configuration parsing, and they are used during the parse and
index actions.
This issue was introduced in dev23 with the major pattern rework,
and is a continuation of commit a631fc8 ("BUG/MAJOR: patterns: -i
and -n are ignored for inlined patterns"). No backport is needed.
BUG/MAJOR: patterns: -i and -n are ignored for inlined patterns
These flags are only passed to pattern_read_from_file() which
loads the patterns from a file. The functions used to parse the
patterns from the current line do not provide the means to pass
the pattern flags so they're lost.
This issue was introduced in dev23 with the major pattern rework,
and was reported by Graham Morley. No backport is needed.
BUG/MEDIUM: pattern: a typo breaks automatic acl/map numbering
Dmitry Sivachenko reported that nice warning :
src/pattern.c:2243:43: warning: if statement has empty body [-Wempty-body]
if (&ref2->list == &pattern_reference);
^
src/pattern.c:2243:43: note: put the semicolon on a separate line to silence
this warning
It was merged as is with the code from commit af5a29d ("MINOR: pattern:
Each pattern is identified by unique id").
So it looks like we can reassign an ID which is still in use because of
this.
Released version 1.5-dev24 with the following main changes :
- MINOR: pattern: find element in a reference
- MEDIUM: http: ACL and MAP updates through http-(request|response) rules
- MEDIUM: ssl: explicitly log failed handshakes after a heartbeat
- DOC: Full section dedicated to the converters
- MEDIUM: http: register http-request and http-response keywords
- BUG/MINOR: compression: correctly report incoming byte count
- BUG/MINOR: http: don't report server aborts as client aborts
- BUG/MEDIUM: channel: bi_putblk() must not wrap before the end of buffer
- CLEANUP: buffers: remove unused function buffer_contig_space_with_res()
- MEDIUM: stats: reimplement HTTP keep-alive on the stats page
- BUG/MAJOR: http: fix timeouts during data forwarding
- BUG/MEDIUM: http: 100-continue responses must process the next part immediately
- MEDIUM: http: move skipping of 100-continue earlier
- BUILD: stats: let gcc know that last_fwd cannot be used uninitialized...
- CLEANUP: general: get rid of all old occurrences of "session *t"
- CLEANUP: http: remove the useless "if (1)" inherited from version 1.4
- BUG/MEDIUM: stats: mismatch between behaviour and doc about front/back
- MEDIUM: http: enable analysers to have keep-alive on stats
- REORG: http: move HTTP Connection response header parsing earlier
- MINOR: stats: always emit HTTP/1.1 in responses
- MINOR: http: add capture.req.ver and capture.res.ver
- MINOR: checks: add a new global max-spread-checks directive
- BUG/MAJOR: http: fix the 'next' pointer when performing a redirect
- MINOR: http: implement the max-keep-alive-queue setting
- DOC: fix alphabetic order of tcp-check
- MINOR: connection: add a new error code for SSL with heartbeat
- MEDIUM: ssl: implement a workaround for the OpenSSL heartbleed attack
- BUG/MEDIUM: Revert "MEDIUM: ssl: Add standardized DH parameters >= 1024 bits"
- BUILD: http: remove a warning on strndup
- BUILD: ssl: avoid a warning about conn not used with OpenSSL < 1.0.1
- BUG/MINOR: ssl: really block OpenSSL's response to heartbleed attack
- MINOR: ssl: finally catch the heartbeats missing the padding
MINOR: ssl: finally catch the heartbeats missing the padding
Previous patch only focused on parsing the packet right and blocking
it, so it relaxed one test on the packet length. The difference is
not usable for attacking but the logs will not report an attack for
such cases, which is probably bad. Better report all known invalid
packets cases.
BUG/MINOR: ssl: really block OpenSSL's response to heartbleed attack
Recent commit f51c698 ("MEDIUM: ssl: implement a workaround for the
OpenSSL heartbleed attack") did not always work well, because OpenSSL
is fun enough for not testing errors before sending data... So the
output sometimes contained some data.
The OpenSSL code relies on the max_send_segment value to limit the
packet length. The code ensures that a value of zero will result in
no single byte leaking. So we're forcing this instead and that
definitely fixes the issue. Note that we need to set it the hard
way since the regular API checks for valid values.
BUILD: ssl: avoid a warning about conn not used with OpenSSL < 1.0.1
Building with a version of openssl without heartbeat gives this since
latest 29f037d ("MEDIUM: ssl: explicitly log failed handshakes after a
heartbeat") :
src/ssl_sock.c: In function 'ssl_sock_msgcbk':
src/ssl_sock.c:188: warning: unused variable 'conn'
Simply declare conn inside the ifdef. No backport is needed.
The latest commit about set-map/add-acl/... causes this warning for
me :
src/proto_http.c: In function 'parse_http_req_cond':
src/proto_http.c:8863: warning: implicit declaration of function 'strndup'
src/proto_http.c:8863: warning: incompatible implicit declaration of built-in function 'strndup'
src/proto_http.c:8890: warning: incompatible implicit declaration of built-in function 'strndup'
src/proto_http.c:8917: warning: incompatible implicit declaration of built-in function 'strndup'
src/proto_http.c:8944: warning: incompatible implicit declaration of built-in function 'strndup'
Use my_strndup() instead of strndup() which is not portable. No backport
needed.
Sander Klein reported an important performance regression with this
patch applied. It is not yet certain what is exactly the cause but
let's not break other setups now and sort this out after dev24.
The commit was merged into dev23, no need to backport.
MEDIUM: ssl: implement a workaround for the OpenSSL heartbleed attack
Using the previous callback, it's trivial to block the heartbeat attack,
first we control the message length, then we emit an SSL error if it is
out of bounds. A special log is emitted, indicating that a heartbleed
attack was stopped so that they are not confused with other failures.
That way, haproxy can protect itself even when running on an unpatched
SSL stack. Tests performed with openssl-1.0.1c indicate a total success.
MINOR: connection: add a new error code for SSL with heartbeat
Users have seen a huge increase in the rate of SSL handshake failures
starting from 2014/04/08 with the release of the Heartbleed OpenSSL
vulnerability (CVE-2014-0160). Haproxy can detect that a heartbeat
was received in the incoming handshake, and such heartbeats are not
supposed to be common, so let's log a different message when a
handshake error happens after a heartbeat is detected.
This patch only adds the new message and the new code.
The HTTP_REQ_ACT_CUSTOM_STOP action stops evaluation of rules after
your rule, HTTP_REQ_ACT_CUSTOM_CONT permits the evaluation of rules
after your rule.
MEDIUM: http: ACL and MAP updates through http-(request|response) rules
This patch allows manipulation of ACL and MAP content thanks to any
information available in a session: source IP address, HTTP request or
response header, etc...
It's an update "on the fly" of the content of the map/acls. This means
it does not resist to reload or restart of HAProxy.
MINOR: http: implement the max-keep-alive-queue setting
Finn Arne Gangstad suggested that we should have the ability to break
keep-alive when the target server has reached its maxconn and that a
number of connections are present in the queue. After some discussion
around his proposed patch, the following solution was suggested : have
a per-proxy setting to fix a limit to the number of queued connections
on a server after which we break keep-alive. This ensures that even in
high latency networks where keep-alive is beneficial, we try to find a
different server.
This patch is partially based on his original proposal and implements
this configurable threshold.
BUG/MAJOR: http: fix the 'next' pointer when performing a redirect
Commit bed410e ("MAJOR: http: centralize data forwarding in the request path")
has woken up an issue in redirects, where msg->next is not reset when flushing
the input buffer. The result is an attempt to forward a negative amount of
data, making haproxy crash.
This bug does not seem to affect versions prior to dev23, so no backport is
needed.
MINOR: http: add capture.req.ver and capture.res.ver
These ones report a string as "HTTP/1.0" or "HTTP/1.1" depending on the
version of the request message or the response message, respectively.
The purpose is to be able to emit custom log lines reporting this version
in a persistent way.
We used to emit either 1.0 or 1.1 depending on whether we were sending
chunks or not. This condition is useless, better always send 1.1. Also
that way at least clients and intermediary proxies know we speak 1.1.
The "Connection: close" header is still set anyway.
Currently, the parsing of the HTTP Connection header for the response
is performed at the same place as the rule sets, which means that after
parsing the beginning of the response, we still have no information on
whether the response is keep-alive compatible or not. Let's do that
earlier.
Note that this is the same code that was moved in the previous function,
both of them are always called in a row so no change of behaviour is
expected.
A future change might consist in having a late analyser to perform the
late header changes such as mangling the connection header. It's quite
painful that currently this is mixed with the rest of the processing
such as filters.
BUG/MEDIUM: stats: mismatch between behaviour and doc about front/back
In version 1.3.4, we got the ability to split configuration parts between
frontends and backends. The stats was attached to the backend and a control
was made to ensure that it was used only in a listen or backend section, but
not in a frontend.
The documentation clearly says that the statement may only be used in the
backend.
But since that same version above, the defaults stats configuration is
only filled in the frontend part of the proxy and not in the backend's.
So a backend will not get stats which are enabled in a defaults section,
despite what the doc says. However, a frontend configured after a defaults
section will get stats and will not emit the warning!
There were many technical limitations in 1.3.4 making it impossible to
have the stats working both in the frontend and backend, but now this has
become a total mess.
It's common however to see people create a frontend with a perfectly
working stats configuration which only emits a warning stating that it
might not work, adding to the confusion. Most people workaround the tricky
behaviour by declaring a "listen" section with no server, which was the
recommended solution in 1.3 where it was even suggested to add a dispatch
address to avoid a warning.
So the right solution seems to do the following :
- ensure that the defaults section's settings apply to the backends,
as documented ;
- let the frontends work in order not to break existing setups relying
on the defaults section ;
- officially allow stats to be declared in frontends and remove the
warninng
This patch should probably not be backported since it's not certain that
1.4 is fully compatible with having stats in frontends and backends (which
was really made possible thanks to applets).
CLEANUP: general: get rid of all old occurrences of "session *t"
All the code inherited from version 1.1 still holds a lot ot sessions
called "t" because in 1.1 they were tasks. This naming is very annoying
and sometimes even confusing, for example in code involving tables.
Let's get rid of this once for all and before 1.5-final.
Nothing changed beyond just carefully renaming these variables.
BUILD: stats: let gcc know that last_fwd cannot be used uninitialized...
OK, for once it cannot easily know this one, and certain versions are
emitting this harmless warning :
src/dumpstats.c: In function 'http_stats_io_handler':
src/dumpstats.c:4507:19: warning: 'last_fwd' may be used uninitialized in this function [-Wmaybe-uninitialized]
MEDIUM: http: move skipping of 100-continue earlier
It's useless to process 100-continue in the middle of response filters
because there's no info in the 100 response itself, and it could even
make things worse. So better use it as it is, an interim response
waiting for the next response, thus we just have to put it into
http_wait_for_response(). That way we ensure to have a valid response
in this function.
BUG/MEDIUM: http: 100-continue responses must process the next part immediately
Since commit d7ad9f5 ("MAJOR: channel: add a new flag CF_WAKE_WRITE to
notify the task of writes"), we got another bug with 100-continue responses.
If the final response comes in the same packet as the 100, then the rest of
the buffer is not processed since there is no wake-up event.
In fact the change above uncoverred the real culprit which is more
likely session.c which should detect that an earlier analyser was set
and should loop back to it.
A cleaner fix would be better, but setting the flag works fine.
This issue was introduced in 1.5-dev22, no backport is needed.
BUG/MAJOR: http: fix timeouts during data forwarding
Patches c623c17 ("MEDIUM: http: start to centralize the forwarding code")
and bed410e ("MAJOR: http: centralize data forwarding in the request path")
merged into 1.5-dev23 cause transfers to be silently aborted after the
server timeout due to the fact that the analysers are woken up when the
timeout strikes and they believe they have nothing more to do, so they're
terminating the transfer.
MEDIUM: stats: reimplement HTTP keep-alive on the stats page
This basically reimplements commit f3221f9 ("MEDIUM: stats: add support
for HTTP keep-alive on the stats page") which was reverted by commit 51437d2 after Igor Chan reported a broken stats page caused by the bug
fix by previous commit.
BUG/MEDIUM: channel: bi_putblk() must not wrap before the end of buffer
The errors reported by Igor Chan on the stats interface in chunked mode
were caused by data wrapping at the wrong place in the buffer. It could
be reliably reproduced by picking random buffer sizes until the issue
appeared (for a given conf, 5300 with 1024 maxrewrite was OK).
The issue is that the stats interface uses bi_putchk() to emit data,
which relies on bi_putblk(). This code checks the largest part that can
be emitted while preserving the rewrite reserve, but uses that result to
compute the wrapping offset, which is wrong. If some data remain present
in the buffer, the wrapping may be allowed and will happen before the
end of the buffer, leaving some old data in the buffer.
The reason it did not happen before keep-alive is simply that the buffer
was much less likely to contain older data. It also used to happen only
for certain configs after a certain amount of time because the size of
the counters used to inflate the output till the point wrapping started
to happen.
The fix is trivial, buffer_contig_space_with_res() simply needs to be
replaced by buffer_contig_space().
Note that peers were using the same function so it is possible that they
were affected as well.
This issue was introduced in 1.5-dev8. No backport to stable is needed.
BUG/MINOR: http: don't report server aborts as client aborts
Commit f003d37 ("BUG/MINOR: http: don't report client aborts as server errors")
attempted to fix a longstanding issue by which some client aborts could be
logged as server errors. Unfortunately, one of the tests involved there also
catches truncated server responses, which are reported as client aborts.
Instead, only check that the client has really closed using the abortonclose
option, just as in done in the request path (which means that the close was
propagated to the server).
The faulty fix above was introduced in 1.5-dev15, and was backported into
1.4.23.
Thanks to Patrick Hemmer for reporting this issue with traces showing the
root cause of the problem.
Released version 1.5-dev23 with the following main changes :
- BUG/MINOR: reject malformed HTTP/0.9 requests
- MINOR: systemd wrapper: re-execute on SIGUSR2
- MINOR: systemd wrapper: improve logging
- MINOR: systemd wrapper: propagate exit status
- BUG/MINOR: tcpcheck connect wrong behavior
- MEDIUM: proxy: support use_backend with dynamic names
- MINOR: stats: Enhancement to stats page to provide information of last session time.
- BUG/MEDIUM: peers: fix key consistency for integer stick tables
- DOC: fix a typo on http-server-close and encapsulate options with double-quotes
- DOC: fix fetching samples syntax
- MINOR: ssl: add ssl_fc_unique_id to fetch TLS Unique ID
- MEDIUM: ssl: Use ALPN support as it will be available in OpenSSL 1.0.2
- DOC: fix typo
- CLEANUP: code style: use tabs to indent codes instead of spaces
- DOC: fix a few config typos.
- BUG/MINOR: raw_sock: also consider ENOTCONN in addition to EAGAIN for recv()
- DOC: lowercase format string in unique-id
- MINOR: set IP_FREEBIND on IPv6 sockets in transparent mode
- BUG/MINOR: acl: req_ssl_sni fails with SSLv3 record version
- BUG/MINOR: build: add missing objects in osx and bsd Makefiles
- BUG/MINOR: build: handle whitespaces in wc -l output
- BUG/MINOR: Fix name lookup ordering when compiled with USE_GETADDRINFO
- MEDIUM: ssl: Add standardized DH parameters >= 1024 bits
- BUG/MEDIUM: map: The map parser includes blank lines.
- BUG/MINOR: log: The log of quotted capture header has been terminated by 2 quotes.
- MINOR: standard: add function "encode_chunk"
- BUG/MINOR: http: fix encoding of samples used in http headers
- MINOR: sample: add hex converter
- MEDIUM: sample: change the behavior of the bin2str cast
- MAJOR: auth: Change the internal authentication system.
- MEDIUM: acl/pattern: standardisation "of pat_parse_int()" and "pat_parse_dotted_ver()"
- MEDIUM: pattern: The pattern parser no more uses <opaque> and just takes one string.
- MEDIUM: pattern: Change the prototype of the function pattern_register().
- CONTRIB: ip6range: add a network IPv6 range to mask converter
- MINOR: pattern: separe list element from the data part.
- MEDIUM: pattern: add indexation function.
- MEDIUM: pattern: The parse functions just return "struct pattern" without memory allocation
- MINOR: pattern: Rename "pat_idx_elt" to "pattern_tree"
- MINOR: sample: dont call the sample cast function "c_none"
- MINOR: standard: Add function for converting cidr to network mask.
- MEDIUM: sample: Remove types SMP_T_CSTR and SMP_T_CBIN, replace it by SMP_F_CONST flags
- MEDIUM: sample/http_proto: Add new type called method
- MINOR: dumpstats: Group map inline help
- MEDIUM: pattern: The function pattern_exec_match() returns "struct pattern" if the patten match.
- MINOR: dumpstats: change map inline sentences
- MINOR: dumpstats: change the "get map" display management
- MINOR: map/dumpstats: The cli cmd "get map ..." display the "int" format.
- MEDIUM: pattern: The match function browse itself the list or the tree.
- MEDIUM: pattern: Index IPv6 addresses in a tree.
- MEDIUM: pattern: add delete functions
- MEDIUM: pattern: add prune function
- MEDIUM: pattern: add sample lookup function.
- MEDIUM: pattern/dumpstats: The function pattern_lookup() is no longer used
- MINOR: map/pattern: The sample parser is stored in the pattern
- MAJOR: pattern/map: Extends the map edition system in the patterns
- MEDIUM: pattern: merge same pattern
- MEDIUM: pattern: The expected type is stored in the pattern head, and conversion is executed once.
- MINOR: pattern: Each pattern is identified by unique id.
- MINOR: pattern/acl: Each pattern of each acl can be load with specified id
- MINOR: pattern: The function "pattern_register()" is no longer used.
- MINOR: pattern: Merge function pattern_add() with pat_ref_push().
- MINOR: pattern: store configuration reference for each acl or map pattern.
- MINOR: pattern: Each pattern expression element store the reference struct.
- MINOR: dumpstats: display the reference for th key/pattern and value.
- MEDIUM: pattern: delete() function uses the pat_ref_elt to find the element to be removed
- MEDIUM: pattern_find_smp: functions find_smp uses the pat_ref_elt to find the element to be removed
- MEDIUM: dumpstats/pattern: display and use each pointer of each pattern dumped
- MINOR: pattern/map/acl: Centralization of the file parsers
- MINOR: pattern: Check if the file reference is not used with acl and map
- MINOR: acl/pattern: Acl "-M" option force to load file as map file with two columns
- MEDIUM: dumpstats: Display error message during add of values.
- MINOR: pattern: The function pat_ref_set() have now atomic behavior
- MINOR: regex: The pointer regstr in the struc regex is no longer used.
- MINOR: cli: Block the usage of the command "acl add" in many cases.
- MINOR: doc: Update the documentation about the map and acl
- MINOR: pattern: index duplicates
- MINOR: configuration: File and line propagation
- MINOR: dumpstat/conf: display all the configuration lines that using pattern reference
- MINOR: standard: Disable ip resolution during the runtime
- MINOR: pattern: Remove the flag "PAT_F_FROM_FILE".
- MINOR: pattern: forbid dns resolutions
- DOC: document "get map" / "get acl" on the CLI
- MEDIUM: acl: Change the acl register struct
- BUG/MEDIUM: acl: boolean only matches were broken by recent changes
- DOC: pattern: pattern organisation schematics
- MINOR: pattern/cli: Update used terms in documentation and cli
- MINOR: cli: remove information about acl or map owner.
- MINOR: session: don't always assume there's a listener
- MINOR: pattern: Add function to prune and reload pattern list.
- MINOR: standard: Add ipv6 support in the function url2sa().
- MEDIUM: config: Dynamic sections.
- BUG/MEDIUM: stick-table: fix IPv4-to-IPv6 conversion in src_* fetches
- MINOR: http: Add the "language" converter to for use with accept-language
- BUG/MINOR: log: Don't dump empty unique-id
- BUG/MAJOR: session: fix a possible crash with src_tracked
- DOC: Update "language" documentation
- MINOR: http: add the function "del-header" to the directives http-request and http-response
- DOC: add some information on capture.(req|res).hdr
- MINOR: http: capture.req.method and capture.req.uri
- MINOR: http: optimize capture.req.method and capture.req.uri
- MINOR: session: clean up the connection free code
- BUG/MEDIUM: checks: immediately report a connection success
- MEDIUM: connection: don't use real send() flags in snd_buf()
- OPTIM: ssl: implement dynamic record size adjustment
- MINOR: stats: report exact last session time in backend too
- BUG/MEDIUM: stats: the "lastsess" field must appear last in the CSV.
- BUG/MAJOR: check: fix memory leak in "tcp-check connect" over SSL
- BUG/MINOR: channel: initialize xfer_small/xfer_large on new buffers
- MINOR: channel: add the date of last read in the channel
- MEDIUM: stream-int: automatically disable CF_STREAMER flags after idle
- MINOR: ssl: add DEFAULT_SSL_MAX_RECORD to set the record size at build time
- MINOR: config: make the stream interface idle timer user-configurable
- MINOR: config: add global directives to set default SSL ciphers
- MINOR: sample: add a rand() sample fetch to return a sample.
- BUG/MEDIUM: config: immediately abort if peers section has no name
- BUG/MINOR: ssl: fix syntax in config error message
- BUG/MEDIUM: ssl: always send a full buffer after EAGAIN
- BUG/MINOR: config: server on-marked-* statement is ignored in default-server
- BUG/MEDIUM: backend: prefer-last-server breaks redispatch
- BUG/MEDIUM: http: continue to emit 503 on keep-alive to different server
- MEDIUM: acl: fix pattern type for payload / payload_lv
- BUG/MINOR: config: fix a crash on startup when a disabled backend references a peer
- BUG/MEDIUM: compression: fix the output type of the compressor name
- BUG/MEDIUM: http: don't start to forward request data before the connect
- MINOR: http: release compression context only in http_end_txn()
- MINOR: protect ebimtree/ebistree against multiple inclusions
- MEDIUM: proxy: create a tree to store proxies by name
- MEDIUM: proxy: make findproxy() use trees to look up proxies
- MEDIUM: proxy: make get_backend_server() use findproxy() to lookup proxies
- MEDIUM: stick-table: lookup table names using trees.
- MEDIUM: config: faster lookup for duplicated proxy name
- CLEANUP: acl: remove obsolete test in parse_acl_expr()
- MINOR: sample: move smp_to_type to sample.c
- MEDIUM: compression: consider the "q=" attribute in Accept-Encoding
- REORG: cfgparse: move server keyword parsing to server.c
- BUILD: adjust makefile for AIX 5.1
- BUG/MEDIUM: pattern: fix wrong definition of the pat_prune_fcts array
- CLEANUP: pattern: move array definitions to proto/ and not types/
- BUG/MAJOR: counters: check for null-deref when looking up an alternate table
- BUILD: ssl: previous patch failed
- BUILD/MEDIUM: standard: get rid of the last strcpy()
- BUILD/MEDIUM: standard: get rid of sprintf()
- BUILD/MEDIUM: cfgparse: get rid of sprintf()
- BUILD/MEDIUM: checks: get rid of sprintf()
- BUILD/MEDIUM: http: remove calls to sprintf()
- BUG/MEDIUM: systemd-wrapper: fix locating of haproxy binary
- BUILD/MINOR: ssl: remove one call to sprintf()
- MEDIUM: http: don't reject anymore message bodies not containing the url param
- MEDIUM: http: wait for the first chunk or message body length in http_process_body
- CLEANUP: http: rename http_process_request_body()
- CLEANUP: http: prepare dedicated processing for chunked encoded message bodies
- MINOR: http: make msg->eol carry the last CRLF length
- MAJOR: http: do not use msg->sol while processing messages or forwarding data
- MEDIUM: http: http_parse_chunk_crlf() must not advance the buffer pointer
- MAJOR: http: don't update msg->sov anymore while processing the body
- MINOR: http: add a small helper to compute the amount of body bytes present
- MEDIUM: http: add a small helper to compute how far to rewind to find headers
- MINOR: http: add a small helper to compute how far to rewind to find URI
- MEDIUM: http: small helpers to compute how far to rewind to find BODY and DATA
- MAJOR: http: reset msg->sov after headers are forwarded
- MEDIUM: http: forward headers again while waiting for connection to complete
- BUG/MINOR: http: deinitialize compression after a parsing error
- BUG/MINOR: http: deinitialize compression after a compression error
- MEDIUM: http: headers must be forwarded even if data was already inspected
- MAJOR: http: re-enable compression on chunked encoding
- MAJOR: http/compression: fix chunked-encoded response processing
- MEDIUM: http: cleanup: centralize a little bit HTTP compression end
- MEDIUM: http: start to centralize the forwarding code
- MINOR: http: further cleanups of response forwarding function
- MEDIUM: http: only allocate the temporary compression buffer when needed
- MAJOR: http: centralize data forwarding in the request path
- CLEANUP: http: document the response forwarding states
- CLEANUP: http: remove all calls to http_silent_debug()
- DOC: internal: add some reminders about HTTP parsing and pointer states
- BUG/MAJOR: http: fix bug in parse_qvalue() when selecting compression algo
- BUG/MINOR: stats: last session was not always set
- DOC: add pointer to the Cyril's HTML doc in the README
- MEDIUM: config: relax use_backend check to make the condition optional
- MEDIUM: config: report misplaced http-request rules
- MEDIUM: config: report misplaced use-server rules
- DOC: update roadmap with what was done.
Till now there was no check against misplaced use-server rules, and
no warning was emitted, adding to the confusion. They're processed
just after the use_backend rules, or more exactly at the same level
but for the backend.
Recently, the http-request ruleset started to be used a lot and some
bug reports were caused by misplaced http-request rules because there
was no warning if they're after a redirect or use_backend rule. Let's
fix this now. http-request rules are just after the block rules.
MEDIUM: config: relax use_backend check to make the condition optional
Since it became possible to use log-format expressions in use_backend,
having a mandatory condition becomes annoying because configurations
are full of "if TRUE". Let's relax the check to accept no condition
like many other keywords (eg: redirect).
Cyril Bonté reported that the "lastsess" field of a stats-only backend
was never updated. In fact the same is true for any applet and anything
not a server. Also, lastsess was not updated for a server reusing its
connection for a new request.
Since the goal of this field is to report recent activity, it's better
to ensure that all accesses are reported. The call has been moved to
the code validating the session establishment instead, since everything
passes there.
The syntax used to document fetching samples with optional arguments was not
always valid. This commit fixes this issue in order to allow an easier parsing
of the documentation.
BUG/MAJOR: http: fix bug in parse_qvalue() when selecting compression algo
Commit ad90351 ("MINOR: http: Add the "language" converter to for use with accept-language")
introduced a typo in parse_qvalue :
if (*end)
*end = qvalue;
while it should be :
if (end)
*end = qvalue;
Since end is tested for being NULL. This crashes when selecting the
compression algorithm since end is NULL here. No backport is needed,
this is just in latest 1.5-dev.
CLEANUP: http: document the response forwarding states
The forwarding code is never obvious to enter into for newcomers, so
better improve the documentation about how states are chained and what
happens for each of them.
MEDIUM: http: start to centralize the forwarding code
Doing so avoids calling channel_forward() for each part of the chunk
parsing and lowers the number of calls to channel_forward() to only
one per buffer, resulting in about 11% performance increase on small
chunks forwarding rate.
MEDIUM: http: cleanup: centralize a little bit HTTP compression end
The call to flush the compression buffers only needs to be done when
entering the final states or when leaving with missing data. After
that, if trailers are present, they have to be forwarded.
Now we have valid buffer offsets, we can use them to safely parse the
input and only forward when needed. Thus we can get rid of the
consumed_data accumulator, and the code now works both for chunked and
content-length, even with a server feeding one byte at a time (which
systematically broke the previous one).
It's worth noting that 0<CRLF> must always be sent after end of data
(ie: chunk_len==0), and that the trailing CRLF is sent only content
length mode, because in chunked we'll have to pass trailers.
MEDIUM: http: headers must be forwarded even if data was already inspected
Currently, we forward headers only if the incoming message is still before
HTTP_MSG_CHUNK_SIZE, otherwise they'll be considered as data. In practice
this is always true for the response since there's no data inspection, and
for the request there is no compression so there's no problem with forwarding
them as data.
But the principle is incorrect and will make it difficult to later add data
processing features. So better fix it now.
The new principle is simple :
- if headers were not yet forwarded, forward them now.
- while doing so, check if we need to update the state
BUG/MINOR: http: deinitialize compression after a compression error
If for some reason, the compression returns an error, the compression
is not deinitialized which also means that any pending data are not
flushed and could be lost, especially in the chunked-encoded case.
No backport is needed.
BUG/MINOR: http: deinitialize compression after a parsing error
When a parsing error was encountered in a chunked response, we failed
to properly deinitialize the compression context. There was no impact
till now since compression of chunked responses was disabled. No backport
is needed.
MEDIUM: http: forward headers again while waiting for connection to complete
Thanks to the last updates on the message pointers, it is now safe again to
enable forwarding of the request headers while waiting for the connection to
complete because we know how to safely rewind this part.
So this patch slightly modifies what was done in commit 80a92c0 ("BUG/MEDIUM:
http: don't start to forward request data before the connect") to let up to
msg->sov bytes be forwarded when waiting for the connection. The resulting
effect is that a POST request may now be sent with the connect's ACK, which
still saves a packet and may even be useful later when TFO is supported.
MAJOR: http: reset msg->sov after headers are forwarded
In order to avoid abusively relying on buf->o to guess how many bytes to
rewind during a redispatch, we now clear msg->sov. Thus the meaning of this
field is exactly "how many bytes of headers are left to be forwarded". It
is still possible to rewind because msg->eoh + msg->eol equal that value
before scheduling the forwarding, so we can always subtract them.
MEDIUM: http: small helpers to compute how far to rewind to find BODY and DATA
http_body_rewind() returns the number of bytes to rewind before buf->p to
find the message's body. It relies on http_hdr_rewind() to find the beginning
and adds msg->eoh + msg->eol which are always safe.
http_data_rewind() does the same to get the beginning of the data, which
differs from above when a chunk is present. It uses the function above and
adds msg->sol.
The purpose is to centralize further ->sov changes aiming at avoiding
to rely on buf->o.
MINOR: http: add a small helper to compute how far to rewind to find URI
http_uri_rewind() returns the number of bytes to rewind before buf->p to
find the URI. It relies on http_hdr_rewind() to find the beginning and
is just here to simplify operations.
The purpose is to centralize further ->sov changes aiming at avoiding
to rely on buf->o.
MEDIUM: http: add a small helper to compute how far to rewind to find headers
http_hdr_rewind() returns the number of bytes to rewind before buf->p to
find the beginning of headers. At the moment it's not exact as it still
relies on buf->o, assuming that no other data from a past message were
pending there, but it's what was done till there.
The purpose is to centralize further ->sov changes aiming at avoiding
to rely on buf->o.
MINOR: http: add a small helper to compute the amount of body bytes present
http_body_bytes() returns the number of bytes of the current message body
present in the buffer. It is compatible with being called before and after
the headers are forwarded.
MAJOR: http: don't update msg->sov anymore while processing the body
We used to have msg->sov updated for every chunk that was parsed. The issue
is that we want to be able to rewind after chunks were parsed in case we need
to redispatch a request and perform a new hash on the request or insert a
different server header name.
Currently, msg->sov and msg->next make parallel progress. We reached a point
where they're always equal because msg->next is initialized from msg->sov,
and is subtracted msg->sov's value each time msg->sov bytes are forwarded.
So we can now ensure that msg->sov can always be replaced by msg->next for
every state after HTTP_MSG_BODY where it is used as a position counter.
This allows us to keep msg->sov untouched whatever the number of chunks that
are parsed, as is needed to extract data from POST request (eg: url_param).
However, we still need to know the starting position of the data relative to
the body, which differs by the chunk size length. We use msg->sol for this
since it's now always zero and unused in the body.
So with this patch, we have the following situation :
- msg->sov = msg->eoh + msg->eol = size of the headers including last CRLF
- msg->sol = length of the chunk size if any. So msg->sov + msg->sol = DATA.
- msg->next corresponds to the byte being inspected based on the current
state and is always >= msg->sov before starting to forward anything.
Since sov and next are updated in case of header rewriting, a rewind will
fix them both when needed. Of course, ->sol has no reason for changing in
such conditions, so it's fine to keep it relative to msg->sov.
In theory, even if a redispatch has to be performed, a transformation
occurring on the request would still work because the data moved would
still appear at the same place relative to bug->p.
MEDIUM: http: http_parse_chunk_crlf() must not advance the buffer pointer
This function is only a parser, it must start to parse at the next character
and only update the outgoing relative pointers, but not expect the buffer to
be aligned with the next byte to be parsed.
It's important to fix this otherwise we cannot use this function to parse
chunks without starting to forward data.
MAJOR: http: do not use msg->sol while processing messages or forwarding data
There are still some pending issues in the gzip compressor, and fixing
them requires a better handling of intermediate parsing states.
Another issue to deal with is the rewinding of a buffer during a redispatch
when a load balancing algorithm involves L7 data because the exact amount of
data to rewind is not clear. At the moment, this is handled by unwinding all
pending data, which cannot work in responses due to pipelining.
Last, having a first analysis which parses the body and another one which
restarts from where the parsing was left is wrong. Right now it only works
because we never both parse and transform in the same direction. But that
is wrong anyway.
In order to address the first issue, we'll have to use msg->eoh + msg->eol
to find the end of headers, and we still need to store the information about
the forwarded header length somewhere (msg->sol might be reused for this).
msg->sov may only be used for the start of data and not for subsequent chunks
if possible. This first implies that we stop sharing it with header length,
and stop using msg->sol there. In fact we don't need it already as it is
always zero when reaching the HTTP_MSG_BODY state. It was only updated to
reflect a copy of msg->sov.
So now as a first step into that direction, this patch ensure that msg->sol
is never re-assigned after being set to zero and is not used anymore when
we're dealing with HTTP processing and forwarding. We'll later reuse it
differently but for now it's secured.
The patch does nothing magic, it only removes msg->sol everywhere it was
already zero and avoids setting it. In order to keep the sov-sol difference,
it now resets sov after forwarding data. In theory there's no problem here,
but the patch is still tagged major because that code is complex.
Willy Tarreau [Thu, 13 Mar 2014 14:48:45 +0000 (15:48 +0100)]
MINOR: http: make msg->eol carry the last CRLF length
One of the issues we face when we need to either forward headers only
before compressing, or rewind the stream during a redispatch is to know
the proper length of the request headers. msg->eoh always has the total
length up to the last CRLF, and we never know whether the request ended
with a single LF or a standard CRLF. This makes it hard to rewind the
headers without explicitly checking the bytes in the buffer.
Instead of doing so, we now use msg->eol to carry the length of the last
CRLF (either 1 or 2). Since it is not modified at all after HTTP_MSG_BODY,
and was only left in an undefined state, it is safe to use at any moment.
Thus, the complete header length to forward or to rewind now is always
msg->eoh + msg->eol.
CLEANUP: http: prepare dedicated processing for chunked encoded message bodies
Content-length encoded message bodies are trivial to deal with, but
chunked-encoded will require improvements, so let's separate the code
flows between the two to ease next steps. The behaviour is not changed
at all, the code is only rearranged.
projects / thirdparty / haproxy.git / log
