* moves the regex pattern state storage to class RegexPattern
in base/RegexPattern.h which is MEMPROXY_CLASS pooled and
constructed with flags and pattern preset.
- for now the regcomp generated data is set separately.
* Replaces ACL storage class RegexList with a std::list
* converts refresh_pattern regex data to class RegexPattern for
its pattern and -i/+i flag details.
The squid_disk_module_candidates_* autoconf variable for disabled
disk I/O modules needs to be explicitly set to 'no' to perform the
disabling. Do this for all modules that were not doing so when module
dependencies were broken or absent.
Also, convert the IpcIo module from a hard ERROR, to a soft auto-disable
when SHM support is not present.
Add temporary SBufToCstring() helper functions for SBuf transition
These functions provide safe replacement for xstrdup() and xstrncpy()
that guarantees 0-termination of the output c-string but do not have
any side effects or behaviour guarantees affecting the source SBuf
internal state.
This lack of side effects is important for the transitional period
where a lot of buffer contents will be copied out of SBuf but are
'read-only' and need to avoid overheads such as the reallocating
twice (or more) that would occur if using SBuf::c_str().
Effective immediately we have a ban on using the xstr*() group of
helper functions to copy data out of SBuf::raw*() accessors. The
xstr*() and all other common system str*() use c-string dependent
operations internally which on non-0-terminated SBuf internals can
result in nasty performance issues (ie. strlen() of 2 MB 'string').
Increasingly code used inside squid.conf parsing is capable of throwing
exceptions to signal errors. Catch any unexpected exceptions that reach
the config parse initiator(s) and report as a FATAL event before self
destructing.
Alex Rousskov [Sun, 26 Jul 2015 18:26:52 +0000 (12:26 -0600)]
Fix ICAP transactions that read a lot of data
by ensuring the read buffer has space [unless it is really full].
Trunk r13995 (Parser-NG: Convert the ICAP read buffer to an SBuf)
broke ICAP transactions that read a lot of data because the new
SBuf::consume() method often does not free buffer space, unlike the
old MemBuf::consume(). Affected transactions failed with mayReadMore()
exceptions because their readBuf.spaceSize() was zero while they
needed to read more data.
Any append,parse,consume;append,parse,consume;... user of SBuf cannot
rely on SBuf::spaceSize() to be meaningful because even consuming the
entire SBuf contents may leave spaceSize() at zero! Instead such code
has to use SBuf::length() to keep buffer from growing too big and
SBuf::rawSpace(1) to ensure some space is available for reading when
the buffer is not too big.
auto-enable for DiskIO modules has been done before the AM_CONDITIONAL
are defined. The checks need to test for explicit "yes" values since
disabled modules will have value "" rather than "no" or "auto" like
other components.
When parsing SMB LanManager packets with invalid protocol ID and the
default set of Squid supported protocols. It may access memory outside
the buffer storing protocol names.
smblib is only used by already deprecated helpers which are deprecated
due to far more significant NTLM protocol issues. It will also only
result in packets being rejected later with invalid protocol names. So
this is a minor bug rather than a vulnerability.
Support %un (any available user name) format code for external ACLs.
The same %un code, with the same meaning is already supported in access.log.
In an external ACL request, it expands to the first available user name
from the following list of information sources:
- authenticated user name, like %ul or %LOGIN
- user name supplied by an external ACL to Squid via the "user=..."
key=value pair, like %ue or %EXT_USER
- SSL client name, like %us
- ident user name, like %ui
Based on Amos Jeffries 2011 patch and "arronax28" design:
http://www.squid-cache.org/mail-archive/squid-dev/201112/0080.html
with TODO completion by Measurement Factory
Improve handling of client connections on shutdown
When Squid which are processing a lot of traffic, using persistent
client connections, or dealing with long duration requests are shutdown
they can exit with a lot of connections still open. The
shutdown_lifetime directive exists to allow time for existing
transactions to complete, but this is not always possible and has no
effect on idle connections.
The result is a large dump of aborted FD entries being logged as the TCP
sockets get abruptly reset. Potentially active transactions cache
objects being "corrupted" in the process.
Makes ConnStateData and its children implement Runner API callbacks
to receive signals about Squid shutdown. Which allows their close()
handlers to be run properly and make use of AsyncCalls API. Idle client
connections are closed immediately on the startShutdown() signal, so
their closure CPU cycles happens during the shutdown grace period.
An extra 0-delay event step is added to SignalEngine shutdown sequence
with a new Runner registry hook 'endingShutdown' is added to signal that
the shutdown_lifetime grace period is over for closure of active
transactions. All network FD sockets should be considered unusable for
read()/write() at that point since close handlers may have already been
scheduled by other Runners. AsyncCall's may still be scheduled to
release resources.
Also adds a DeregisterRunner() API action to remove Runners dynamically
from the registered set.
* shutdown grace period ends:
- remaining client connections closed
* shutdown finishes:
- main signal and Async loop halted
- all memory free'd
Server connections which are PINNED or in active use during the
endingShutdown execution will be closed cleanly as a side-effect of the
client closures. Otherwise there is no change (yet) to server connections
or other FD sockets behaviour on shutdown.
Cleanup: replace urlCanonical() with HttpRequest::effectiveReuqestUri()
We have previously been using the term "canonical URL" in Squid to mean
absolute-URI, but not in all cases and may sometimes mean authority-form.
RFC 7230 introduces a new term "Effective Request URI" which directly
matches our desired usage.
* make urlCanonical() global function a method of class HttpRequest
since it depends on request method for its particular form syntax
* remove the now unnecessary canonical member and HttpRequest::SetHost()
* convert HttpRequest::storeId(), Ftp::UrlWith2f(), and ps_state::url()
to SBuf usage to avoid performance regressions in their use.
* replace many uses of xstrdup() with xstrndup() for performance where
the copy cannot be avoided entirely.
* avoid using urlParse() to do a simple URL data-copy in ICAP handling
* update stub_HttpRequest.cc to match full class HttpRequest API
Bounds limiting was restricting configuration to <1.2 since those are
the currently available TLS versions. But 1.3 is upcoming and may be
used within the lifetime of this Squid version.
Violate RFC 2396 and 3986 URI handling requirements.
For the past 17 years RFC 2396 has specified explicit
characters which are disallowed in any form of URI due
to their use in potential or real malware attacks
against network servers.
10 years ago RFC 3986 updated this by allowing some
previously disallowed characters, and moving to a model
of explicitly listing all characters allowed in each
segment of a URI with explicit %-encoding requirement
for all other characters.
Squid has recently been updated to parse that RFC 3986
syntax closely, with minimal tolerance for garbage as
outlined explicitly in RFC 7230.
However, various major corporations are still building
popular tools that violate the RFC 3986 Security
Considerations surrounding safe encoding of characters
in URI they transmit as HTTP request URLs.
This patch allows Squid with --enable-http-violations
(on by default) and configured for lenient parsing (on
by default) to accept a subset of characters which are
expressly forbidden but actively used un-encoded.
Malware attacks utilizing these characters to perform
URL-injection are mitigated by treating the client
request as an HTTP "0.9" protocol message. Such
messages are not permitted to use nonidempotent HTTP
methods which affect server state, and most Mime
headers from the client are ignored.
... from 8 to 8196 before initial congestion message appears.
Modern networks can be quite busy and even amateur installations have a
much higher I/O throughput than Squid was originally designed for. This
often results in a series of "Queue congestion" warnings appearing on
startup before Squid learns what the local environment requires.
The new limit helps to cater for this and reduce teh frequency of
unnecessary warnings. They may still occur, so debug output is also
updated to show what the queue length has grown to with each warning.
Also updating the congestion counter from 32-bit to 64-bit unsigned
since the new limit already consumes half the available growth bits in
32-bit integer.
Cleanup: Shuffle Squid result codes (aka log tags) into class LogTags
This begins the migration of result codes from enumeration to a
set of flags whih can combine into much more flexible logging of
transation activity than hard-coded labels enumerating every
individual code path.
The existing ABORTED and TIMEDOUT state flags are also moved into
the new class as an example of how such flags would operate.
Fix error-testing condition in log_file_daemon.cc (CID 1256161)
An error condition check would assume that ferror(3) returns negative values if error bit is set.
According to documentation, the only guarantee is that it returns nonzero in case of error.
Crypto-NG: Use Security::PeerOptions for listening port TLS settings
The bulk of this patch is symbol shuffling to de-duplicate the TLS
settings storage and parsing code.
* Shuffle relevant AnyP::PortCfg settings into a Security::PeerOptions
member object.
- removes a lot of duplicate config parsing code.
* Remove the now obsolete and unused Ssl::OpenSSLtoSquidSSLVersion()
The actual logic changes are relatively small:
* Shuffle flags= and options= parsing code from Ssl:: to
Security::PeerOptions and update to use Tokenizer,
- fixes performance regression using c_str() on the stored SBuf,
- fixes performance issue with xstrdup() for option tokens,
- removes several calls to c-string manipulation.
* Add cachemgr 'config' report dumper of Security::PeerOptions for use
by all directives using it to dump tls-* parameter names. The old
parameter names are still accepted, and deprecation will follow in a
separate patch.
- fixes bug where cache_peer was not dumping out its SSL/TLS config
settings at all.
* Change the tls_outgoing_options default value from "disable" to setting
TLS/1.0 minimum version.
- fixes squid.conf parsing error on default value "disable".
* Fix tls-min-version=1.N handling not to alter stored options= config
string. Now updates the binary representation in parsedOptions directly.
* Expose the TLS context creation and configuration to non-OpenSSL builds.
- fixes bug where context creation by OpenSSL failed silently.
Avoid SSL certificate db corruption with empty index.txt as a symptom.
* Detect cases where the size file is corrupted or has a clearly wrong
value. Automatically rebuild the database in such cases.
* Teach ssl_crtd to keep running if it is unable to store the generated
certificate in the database. Return the generated certificate to Squid
and log an error message in such cases.
Background:
There are cases where ssl_crtd may corrupt its certificate database.
The known cases manifest themselves with an empty db index file. When
that happens, ssl_crtd helpers quit, SSL bumping does not work any more,
and the certificate DB has to be deleted and re-initialized.
We do not know exactly what causes corruption in deployments, but one
known trigger that is easy to reproduce in a lab is the block size
change in the ssl_crtd configuration. That change has the following
side-effects:
1. When ssl_crtd removes certificates, it computes their size using a
different block size than the one used to store the certificates.
This is may result in negative database sizes.
2. Signed/unsigned conversion results in a huge number near LONG_MAX,
which is then written to the "size" file.
3. The ssl_crtd helper remoces all certificates from database trying to make
space for new certificates.
4. The ssl_crtd helper refuses to store new certificates because the
database size (as described by the "size" file) still exceeds the
configured limit.
5. The ssl_crtd helper exits because it cannot store a new certificates
to the database. No helper response is sent to Squid in this case.
Most likely, there are other corruption triggers -- the database
management code is of an overall poor quality. This change resolves some
of the underlying problems in hope to address at least some of the
unknown triggers as well as the known one.
projects / thirdparty / squid.git / log
