Alex Rousskov [Sun, 14 Aug 2016 01:17:57 +0000 (19:17 -0600)]
Fixed build on systems needing explicit <utility> in RegexPattern.cc.
... after r14795.
Also removed bogus comments implying that std::move() does something to
its argument. That function is just a type cast; it does nothing else!
We use std::move() so that the compiler picks a move constructor or
assignment (where available). In case of RegexPattern::flags, using
std::move() has no effect because integers do not have those move
methods. However, it may be a good idea to use std::move() anyway
because the type of RegexPattern::flags might change in the future and
for consistency sake. Thus, I did not remove std::move(), only comments.
Amos Jeffries [Sat, 13 Aug 2016 10:12:06 +0000 (22:12 +1200)]
Bug 4561: Replace use of default move operators with explicit implementation
The '= default' syntax for move assignment operator and constructor can
result in double-free crashes when the class type containspointer members
since the trivial move uses std::memmove() which does not perform the
required zeroing/nullptr of the temporary source object being moved.
Make Squid death due to overloaded helpers optional.
Added on-persistent-overload=action option to helpers. Helper overload
is defined as running with an overflowing queue. Persistent helper
overload is [still] defined as being overloaded for more than 3 minutes.
The default behavior is unchanged(*) -- Squid worker dies with a fatal
error at the attempt to submit a new request to a persistenly overloaded
helper. This default behavior can also be configured explicitly using
on-persistent-overload=die.
With on-persistent-overload=ERR, when dealing with a persistently
overloaded helper, Squid immediately skips the helper request and sends
an ERR response to the caller. Squid informs the admin when it starts
and when it stops skipping helper requests due to persistent overload.
The code had conflicting notions of an "overloaded helper". The external
ACL helper, the URL rewriter, and the store ID code used queueFull() to
test whether the new request would overflow the queue (and, hence,
overload the helper), but queueFull() itself did not check whether the
queue was full! It checked whether the queue was already overflowing.
This confusion resulted in that code scheduling one extra helper request
before enabling bypass. The code and its documentation are now more
consistent (and better match the "overload" terminology used by the new
configuration option, which also feels better than calling the helper
"full").
(*) Resolving the above confusion resulted in minor (one request)
differences in the number of helper requests queued by Squid for
external ACL, URL rewriting, and store ID helpers, with the adjusted
behavior [better] matching the documentation.
Amos Jeffries [Sat, 6 Aug 2016 04:49:55 +0000 (16:49 +1200)]
Fix clang errors with global InstanceId initialization
Move static member Last into change() method to avoid initialization order
errors when a caller uses a global InstanceId object before the library
instantiating its template is initialized.
Also, convert the Prefix static member to a constant string in a prefix()
method to avoid the same issue there.
Many web servers do not have complete certificate chains. Many browsers use
certificate extensions of the server certificate and download the missing
intermediate certificates automatically from the Internet.
This patch add this feature to Squid.
The information for missing issuer certificates provided by the Authority
Information Access X509 extension. This describes the format and the location
of additional information provided by the issuer of the certificate.
This patch:
- Implements a class Downloader as an independet AsyncJob class. This new
class can be used by internal squid subsystems to download objects from
the network.
- Modify Ssl::PeerConnector class to use new Downloader class to
retrieve missing certificates from the net. The URIs of missing
certificates from the Authority Information Access X509 extension.
- Implements a new basic certificates parser based on openSSL for the
TLS handshake messages parser.
- Modify the Ssl::ServerBio class to:
* Buffer the Server Hello message and not pass it to the openSSL library
until downloading missing certificates, if any, is finished.
* Extract server certificates from server hello message.
This is required to check if there are missing certificates, and if yes
give the chance to squid to download missing certificates and complete
certificate chains before pass them for processing to openSSL
TODO:
- Add support for certs-only CMS message.
From RFC 4325:
"Where the information is available via HTTP or FTP, accessLocation
MUST be a uniformResourceIdentifier and the URI MUST point to either
a single DER encoded certificate as specified in [RFC2585] or a
collection of certificates in a BER or DER encoded "certs-only" CMS
message as specified in [RFC2797]. "
...
"Conforming applications that support HTTP or FTP for accessing
certificates MUST be able to accept individual DER encoded
certificates and SHOULD be able to accept "certs-only" CMS messages."
Squid segfault via Ftp::Client::readControlReply().
Ftp::Client::scheduleReadControlReply(), which may called from the
asynchronous start() or readControlReply()/handleControlReply()
handlers, does not check whether the control connection is still usable
before using it.
Certain server certificates with a large chain and/or large certificates
(i.e. due to excessive amount of SAN entries) are producing helper requests and
replies which are larger than the 32KB limit set in src/helper.cc.
The major problem for squid is that the helper response should fit in the
helper read buffer. Currently squid starts with 4K request buffer and if
required may increase the buffer size up to 32K.
This patch:
- Uses a constant-size read buffer for helpers and accumulates the helper
response to Helper::Reply object.
- Changes the HelperServerBase::requests list to hold list of the new
Helper::Xaction class which holds pairs of Helper::Request and
Helper::Reply objects
- Modifies the Helper::Reply class to accumulate data and restricts the
required memory allocations
... also address Bug 4311 and partially address Bug 2833 and Bug 4471.
Extend collapsed_forwarding functionality to internal revalidation
requests. This implementation does not support Vary-controlled cache
objects and is limited to SMP-unaware caching environments, where each
Squid worker knows nothing about requests and caches handled by other
workers. However, it also lays critical groundwork for future SMP-aware
collapsed revalidation support.
Prior to these changes, multiple concurrent HTTP requests for the same
stale cached object always resulted in multiple internal revalidation
requests sent by Squid to the origin server. Those internal requests
were likely to result in multiple competing Squid cache updates, causing
cache misses and/or more internal revalidation requests, negating
collapsed forwarding savings.
Internal cache revalidation requests are collapsed if and only if
collapsed_forwarding is enabled. There is no option to control just
revalidation collapsing because there is no known use case for it.
* Public revalidation keys
Each Store entry has a unique key. Keys are used to find entries in the
Store (both already cached/swapped_out entries and not). Public keys are
normally tied to the request method and target URI. Same request
properties normally lead to the same public key, making cache hits
possible. If we were to calculate a public key for an internal
revalidation request, it would have been the same as the public key of
the stale cache entry being revalidated. Adding a revalidation response
to Store would have purged that being-revalidated cached entry, even if
the revalidation response itself was not cachable.
To avoid purging being-revalidated cached entries, the old code used
private keys for internal revalidation requests. Private keys are always
unique and cannot accidentally purge a public entry. On the other hand,
for concurrent [revalidation] requests to find the store entry to
collapse on, that store entry has to have a public key!
We resolved this conflict by adding "scope" to public keys:
* Regular/old public keys have default empty scope (that does not affect
key generation). The code not dealing with collapsed revalidation
continues to work as before. All entries stored in caches continue to
have the same keys (with an empty scope).
* When collapsed forwarding is enabled, collapsable internal
revalidation requests get public keys with a "revalidation" scope
(that is fed to the MD5 hash when the key is generated). Such a
revalidation request can find a matching store entry created by
another revalidation request (and collapse on it), but cannot clash
with the entry being revalidated (because that entry key is using a
different [empty] scope).
This change not only enables collapsing of internal revalidation
requests within one worker, but opens the way for SMP-aware workers to
share information about collapsed revalidation requests, similar to how
those workers already share information about being-swapped-out cache
entries.
After receiving the revalidation response, each collapsed revalidation
request may call updateOnNotModified() to update the stale entry [with
the same revalidation response!]. Concurrent entry updates would have
wasted many resources, especially for disk-cached entries that support
header updates, and may have purged being-revalidated entries due to
locking conflicts among updating transactions. To minimize these
problems, we adjusted header and entry metadata updating logic to skip
the update if nothing have changed since the last update.
Also fixed Bug 4311: Collapsed forwarding deadlocks for SMP Squids using
SMP-unaware caches. Collapsed transactions stalled without getting a
response because they were waiting for the shared "transients" table
updates. The table was created by Store::Controller but then abandoned (not
updated) by SMP-unaware caches. Now, the transients table is not created
at all unless SMP-aware caches are present. This fix should also address
complaints about shared memory being used for Squid instances without
SMP-aware caches.
A combination of SMP-aware and SMP-unaware caches is still not supported
and is likely to stall collapsed transactions if they are enabled. Note
that, by default, the memory cache is SMP-aware in SMP environments.
There are situations when Squid logs nothing to access.log after an
[abnormal] transaction termination. Such "stealthy" transactions may be
a security risk and an accounting problem.
ClientHttpRequest is responsible for logging most transactions but that
object is created only after the HTTP request headers are successfully
parsed. Request header parsing errors may be detected and logged
appropriately, but the job handling the incoming transaction may
terminate for reasons outside the parsing code control (e.g., a job-
killing exception thrown when there are no request headers to start
parsing yet or when the job waits for more request headers to finishing
parsing).
This change adds access logging for three cases:
1. accept(2) system call errors (before ConnStateData job is created).
2. Unexpected ConnStateData job termination, when there is no
ClientHttpRequest to log the failure.
3. Connections which send no bytes: these connections drain Squid
resources and, hence, should be logged.
TODO: make this behavior configurable because some browsers are known to
routinely create such connections(and, hence, logging them may create
too much noise in some environments).
* rename initializeSsl() method to initializeTls()
* use SessionPointer instead of SessionPtr
* return value is best used in boolean expressions, so make it bool
this also resolves some template issues with SessionPointer return type.
* rename the session parameter to serverSession which better documents what
it is than 'ssl', and distinguishes it from clientSession in child methods
which have to deal with both.
- Add/fix basic documentation for Downloader and DownloaderContext classes
- Move DownloaderContext class definition to Downloader.cc file
- fix cbdata leaks inside DownloaderCotnext destructor, caused by simple typo
mistake.
- Do not cbdatalock the ClientHttpRequest object inside DownloaderContext.
This class is responsible to hold the main pointer and finaly delete the
ClientHttpRequest object.
LockingPointer is now a stand-alone class that is understood (and
documented) as a typical shared pointer with OpenSSL-friendly object
importing methods. Replaced its TidyPointer::reset() abuse with an
explicit resetWithoutLocking() method after reviewing that all such
calls needed no locking indeed.
Patrick Welche [Wed, 29 Jun 2016 17:24:24 +0000 (11:24 -0600)]
Bug 4376: clang cannot build Squid eCAP code
This change is not needed for libecap v1.0 that uses tr1::shared_ptr.
However, libecap is being updated to use std::shared_ptr. Once that
update is complete, Squid eCAP code will no longer compile because the
implicit conversion of shared_ptr to bool in tr1 was deemed dangerous
and, hence, made "explicit" when shared_ptr became standard. Unlike "if"
statements, "return" statements do not trigger an explicit conversion.
projects / thirdparty / squid.git / log
