Cleanup: replace urlCanonical() with HttpRequest::effectiveReuqestUri()
We have previously been using the term "canonical URL" in Squid to mean
absolute-URI, but not in all cases and may sometimes mean authority-form.
RFC 7230 introduces a new term "Effective Request URI" which directly
matches our desired usage.
* make urlCanonical() global function a method of class HttpRequest
since it depends on request method for its particular form syntax
* remove the now unnecessary canonical member and HttpRequest::SetHost()
* convert HttpRequest::storeId(), Ftp::UrlWith2f(), and ps_state::url()
to SBuf usage to avoid performance regressions in their use.
* replace many uses of xstrdup() with xstrndup() for performance where
the copy cannot be avoided entirely.
* avoid using urlParse() to do a simple URL data-copy in ICAP handling
* update stub_HttpRequest.cc to match full class HttpRequest API
Bounds limiting was restricting configuration to <1.2 since those are
the currently available TLS versions. But 1.3 is upcoming and may be
used within the lifetime of this Squid version.
Violate RFC 2396 and 3986 URI handling requirements.
For the past 17 years RFC 2396 has specified explicit
characters which are disallowed in any form of URI due
to their use in potential or real malware attacks
against network servers.
10 years ago RFC 3986 updated this by allowing some
previously disallowed characters, and moving to a model
of explicitly listing all characters allowed in each
segment of a URI with explicit %-encoding requirement
for all other characters.
Squid has recently been updated to parse that RFC 3986
syntax closely, with minimal tolerance for garbage as
outlined explicitly in RFC 7230.
However, various major corporations are still building
popular tools that violate the RFC 3986 Security
Considerations surrounding safe encoding of characters
in URI they transmit as HTTP request URLs.
This patch allows Squid with --enable-http-violations
(on by default) and configured for lenient parsing (on
by default) to accept a subset of characters which are
expressly forbidden but actively used un-encoded.
Malware attacks utilizing these characters to perform
URL-injection are mitigated by treating the client
request as an HTTP "0.9" protocol message. Such
messages are not permitted to use nonidempotent HTTP
methods which affect server state, and most Mime
headers from the client are ignored.
... from 8 to 8196 before initial congestion message appears.
Modern networks can be quite busy and even amateur installations have a
much higher I/O throughput than Squid was originally designed for. This
often results in a series of "Queue congestion" warnings appearing on
startup before Squid learns what the local environment requires.
The new limit helps to cater for this and reduce teh frequency of
unnecessary warnings. They may still occur, so debug output is also
updated to show what the queue length has grown to with each warning.
Also updating the congestion counter from 32-bit to 64-bit unsigned
since the new limit already consumes half the available growth bits in
32-bit integer.
Cleanup: Shuffle Squid result codes (aka log tags) into class LogTags
This begins the migration of result codes from enumeration to a
set of flags whih can combine into much more flexible logging of
transation activity than hard-coded labels enumerating every
individual code path.
The existing ABORTED and TIMEDOUT state flags are also moved into
the new class as an example of how such flags would operate.
Crypto-NG: Use Security::PeerOptions for listening port TLS settings
The bulk of this patch is symbol shuffling to de-duplicate the TLS
settings storage and parsing code.
* Shuffle relevant AnyP::PortCfg settings into a Security::PeerOptions
member object.
- removes a lot of duplicate config parsing code.
* Remove the now obsolete and unused Ssl::OpenSSLtoSquidSSLVersion()
The actual logic changes are relatively small:
* Shuffle flags= and options= parsing code from Ssl:: to
Security::PeerOptions and update to use Tokenizer,
- fixes performance regression using c_str() on the stored SBuf,
- fixes performance issue with xstrdup() for option tokens,
- removes several calls to c-string manipulation.
* Add cachemgr 'config' report dumper of Security::PeerOptions for use
by all directives using it to dump tls-* parameter names. The old
parameter names are still accepted, and deprecation will follow in a
separate patch.
- fixes bug where cache_peer was not dumping out its SSL/TLS config
settings at all.
* Change the tls_outgoing_options default value from "disable" to setting
TLS/1.0 minimum version.
- fixes squid.conf parsing error on default value "disable".
* Fix tls-min-version=1.N handling not to alter stored options= config
string. Now updates the binary representation in parsedOptions directly.
* Expose the TLS context creation and configuration to non-OpenSSL builds.
- fixes bug where context creation by OpenSSL failed silently.
Avoid SSL certificate db corruption with empty index.txt as a symptom.
* Detect cases where the size file is corrupted or has a clearly wrong
value. Automatically rebuild the database in such cases.
* Teach ssl_crtd to keep running if it is unable to store the generated
certificate in the database. Return the generated certificate to Squid
and log an error message in such cases.
Background:
There are cases where ssl_crtd may corrupt its certificate database.
The known cases manifest themselves with an empty db index file. When
that happens, ssl_crtd helpers quit, SSL bumping does not work any more,
and the certificate DB has to be deleted and re-initialized.
We do not know exactly what causes corruption in deployments, but one
known trigger that is easy to reproduce in a lab is the block size
change in the ssl_crtd configuration. That change has the following
side-effects:
1. When ssl_crtd removes certificates, it computes their size using a
different block size than the one used to store the certificates.
This is may result in negative database sizes.
2. Signed/unsigned conversion results in a huge number near LONG_MAX,
which is then written to the "size" file.
3. The ssl_crtd helper remoces all certificates from database trying to make
space for new certificates.
4. The ssl_crtd helper refuses to store new certificates because the
database size (as described by the "size" file) still exceeds the
configured limit.
5. The ssl_crtd helper exits because it cannot store a new certificates
to the database. No helper response is sent to Squid in this case.
Most likely, there are other corruption triggers -- the database
management code is of an overall poor quality. This change resolves some
of the underlying problems in hope to address at least some of the
unknown triggers as well as the known one.
Errors served using invalid certificates when dealing with SSL server errors.
When bumping Squid needs to send an Squid-generated error "page" over a
secure connection, Squid needs to generate a certificate for that connection.
Prior to these changes, several scenarios could lead to Squid generating
a certificate that clients could not validate. In those cases, the user would
get a cryptic and misleading browser error instead of a Squid-generated
error page with useful details about the problem.
For example, is a server certificate that is rejected by the certificate
validation helper. Squid no longer uses CN from that certificate to generate
a fake certificate.
Another example is a user accessing an origin server using one of its
"alternative names" and getting a Squid-generated certificate containing just
the server common name (CN).
These changes make sure that certificate for error pages is generated using
SNI (when peeking or staring, if available) or CONNECT host name (including
server-first bumping mode). We now update the ConnStateData::sslCommonName
field (used as CN field for generated certificates) only _after_ the server
certificate is successfully validated.
The bug *_DEPENDENCIES existed to solve no longer exists. Current auto*
toolchains can correctly generate the dependencies data based on *_LDADD
and *_SOURCES Makefile variables.
The incorrect definition of squid_DEPENDENCIES was also causing various
build issues and could result in crashes when conditionaly built features
were changed by users ./configure without a full re-bootstrap.
Currently, Squid cannot redirect intercepted connections that are subject to
SslBump rules to _originserver_ cache_peer. For example, consider Squid that
enforces "safe search" by redirecting clients to forcesafesearch.example.com.
Consider a TLS client that tries to connect to www.example.com. Squid needs to
send that client to forcesafesearch.example.com (without changing the host
header and SNI information; those would still point to www.example.com for
safe search to work as intended!).
The admin may configure Squid to send intercepted clients to an originserver
cache_peer with the forcesafesearch.example.com address. Such a configuration
does not currently work together with ssl_bump peek/splice rules.
This patch:
* Fixes src/neighbors.cc bug which prevented CONNECT requests from going
to originserver cache peers. This bug affects both true CONNECT requests
and intercepted SSL/TLS connections (with fake CONNECT requests). Squid
use the CachePeer::in_addr.port which is not meant to be used for the HTTP
port, apparently. HTTP checks should use CachePeer::http_port instead.
* Changes Squid to not initiate SSL/TLS connection to cache_peer for
true CONNECT requests.
* Allows forwarding being-peeked (or stared) at connections to originserver
cache_peers.
The bug fix described in the first bullet makes the last two changes
necessary.
Amos Jeffries [Tue, 30 Jun 2015 07:21:06 +0000 (00:21 -0700)]
Fix cachemgr 'config' report output for TLS options
Adds Security::PeerOptions::dumpCfg() method to output config
details for the peer TLS settings stored. Takes an optional
prefix string for the option parameters.
- uses prefix "" for tls_outgoing_options
- uses prefix "tls-" for https_port, http_port, and cache_peer
- displays "tls-disable" for directives without TLS settings.
Fixes missing cache_peer config output for TLS settings.
Fixes display of http(s)_port TLS settings 'ssl' prefix.
Fixes tls-min-version= internal operation to avoid polluting
tls-options= config settings.
Also, updates documentation mentions of "SSL" in debugs().
Now that SSL/TLS server method is a fixed API function we do not have to
store it locally. Move the lookup to the SSL support code where it is
actually used.
Alex Rousskov [Sun, 28 Jun 2015 14:08:31 +0000 (07:08 -0700)]
Do not blindly forward cache peer CONNECT responses.
Squid blindly forwards cache peer CONNECT responses to clients. This
may break things if the peer responds with something like HTTP 403
(Forbidden) and keeps the connection with Squid open:
- The client application issues a CONNECT request.
- Squid forwards this request to a cache peer.
- Cache peer correctly responds back with a "403 Forbidden".
- Squid does not parse cache peer response and
just forwards it as if it was a Squid response to the client.
- The TCP connections are not closed.
At this stage, Squid is unaware that the CONNECT request has failed. All
subsequent requests on the user agent TCP connection are treated as
tunnelled traffic. Squid is forwarding these requests to the peer on the
TCP connection previously used for the 403-ed CONNECT request, without
proper processing. The additional headers which should have been applied
by Squid to these requests are not applied, and the requests are being
forwarded to the cache peer even though the Squid configuration may
state that these requests must go directly to the origin server.
This fixes Squid to parse cache peer responses, and if an error response
found, respond with "502 Bad Gateway" to the client and close the
connections.
Amos Jeffries [Tue, 23 Jun 2015 09:21:06 +0000 (02:21 -0700)]
Use relative-URL in errorpage.css for SN.png
Modern browsers now seem to be accepting relative-URLs, and Squid
global_internal_static non-https:// URLs are working. So we can do this
now without as many failures.
Amos Jeffries [Mon, 22 Jun 2015 11:29:39 +0000 (04:29 -0700)]
Replace GNU atomics and related hacks with C++11 std::atomic
With C++11 atomic support by the stdlib is not optional. This
resolves issues determining whether GNU atomics are available,
operational 32-bit vs 64-bit, or cross-compiling (bug 4224).
Amos Jeffries [Sat, 20 Jun 2015 00:24:24 +0000 (17:24 -0700)]
Fix CONNECT failover to IPv4 after trying broken IPv6 servers
This makes CONNECT tunnel connection attempts obey forward_timeout
and continue retrying instead of aborting with a client error when one
possible server hits a connect_timeout.
Alex Rousskov [Fri, 19 Jun 2015 16:57:30 +0000 (10:57 -0600)]
Fixed segmentation fault when freeing https_port clientca on reconfigure
or exit.
AnyP::PortCfg::clientCA list was double-freed because the SSL context takes
ownership of the STACK_OF(X509_NAME) supplied via SSL_CTX_set_client_CA_list(),
but Squid was not aware of that. Squid now supplies a clone of clientCA.
Amos Jeffries [Fri, 19 Jun 2015 07:13:57 +0000 (00:13 -0700)]
Bug 4269: ignore-must-revalidate broken
ignore-must-revalidate appears to prevent revalidation by disabling
storage of objects with must-revalidate/proxy-revalidate header.
However it was also preventing revalidation of objects cached due to
ignore-private, or the presence of no-cache, s-maxage, and use of auth
credentials.
Remove the violation option entirely.
Also cleanup the documentation of ignore-auth which was removed earlier.
Amos Jeffries [Thu, 11 Jun 2015 04:51:10 +0000 (21:51 -0700)]
Convert enum LogTags into class LogTags
- enum sequence re-typed to LogTags_ot
- logTagsIsATcpHit() function converted to method LogTags::isTcpHit()
- strings array lookup converted to method LogTags::c_str()
- constructors and assignment assure valid values, no more need for
external asserts before logging.
Amos Jeffries [Tue, 9 Jun 2015 06:14:43 +0000 (23:14 -0700)]
Bug 1961 partial: Move HttpRequest host:port to class URL
Moves the host:port authority details into class URL for more
modular URI management. Add URL::authority() member to generate
authority-form URIs from the class URL stored details.
Also, shuffle urlDefaultPort() to AnyP::UriScheme::defaultPort()
Amos Jeffries [Tue, 9 Jun 2015 01:54:56 +0000 (18:54 -0700)]
Parser-NG: Transfer-Encoding:chunked Parser
Remove several performance regressions incurred in earlier Parser-NG
updates by refactoring the class ChunkedCodingParser to a class
Http1::TeChunkedParser which parses an SBuf I/O buffer for chunked
encoding data and (for now) copies the chunk payloads into a MemBuf buffer.
The new class is inherited from Http1::Parser and presents the same API.
Chunk Trailers are now available via the Parser API mimeHeader() method
- although none of the rest of Squid makes use of that data yet. It
implements parsing using a ::Parser::Tokenizer for (nearly) compliant
protocol tokenization. With enumerated states instead of a dynamic
function-pointer chain.
Measurements:
Co-Advisor shows no compliance change.
Polygraph shows approx 1% speed improvement over trunk.
Wrong intialization value for clientReplyContext::headers_sz member
The clientReplyContext::headers_sz member after the trunk patch r14078
initialized to wrong value. In many cases inside clientReplyContext class
this is considered as initialized to "0" and setting to "-1" can cause
problems.
This bug can be caused by certificates does not contain a CN field. In this
case the Ssl::ErrorDetail::cn method may return NULL causing this assertion
somewhere inside Ssl::ErrorDetail::buildDetail method, which expects always
a non NULL value from Ssl::ErrorDetail::cn and similar methods.
This patch try to hardening the Ssl::ErrorDetail error formating functions to
avoid always check for NULL values and also avoid sending wrong information
for various certificate fields in the case of an error while extracting the
information from certificate..