clang++ doesn't support c++ variable arrays for non-pod types.
Change variable arrays to dynamically-allocated arrays, Ipc::QueueReaders, Ipc::StoreMap and Ipc::Mem::PageStack
Amos Jeffries [Wed, 1 Aug 2012 07:01:58 +0000 (19:01 +1200)]
Support -DFAILURE_MODE_TIME=n compiler flag
This value determins at compile-time how long Squid spends in HIT-only
mode after the failure ratio goes over 1.0. see checkFailureRatio() in
src/client_side_request.cc for details on the ratio.
This flag is supported to remove the need for patching when alteration
is required.
This re-enables Squid peer selection algorithms for intercepted
traffic which has failed Host header verification.
When host verification fails Squid will use, in order of preference:
* an already PINNED server connection
* the client ORIGINAL_DST details
* cache_peer as chosen by selection algorithms
NOTE: whenever DIRECT is selected by routing algorithms the
ORIGINAL_DST is used instead.
Peer selection results are updated to display PINNED and
ORIGINAL_DST alongside DIRECT and cache_peer.
SECURITY NOTE:
At this point Squid will pass the request to cache_peer using the
non-trusted Host header in their URLs. Meaning that the peers
may still be poisoned by CVE-2009-0801 attacks. Only the initial
intercepting proxy is protected.
Full protection against CVE-2009-0801 can be enjoyed by building
Squid with the -DSTRICT_HOST_VERIFY compile-time flag. This will
make the peers unreachable for intercepted traffic where the
Host verification has failed.
* list Sponsors who paid for significant developments in the squid code,
donated hardware for devleopment etc.
* exclude cash donations to the project (only cash to a developer for
specific work)
* in an ordered list broken down in reverse-chronological order of the
latest release they sponsored anything.
For simplicity of maintenance I'm adding @Squid-X.Y labels at the
boundaries for the bzr copy. They should be stripped out automatically
by the "make dist" process for release packaging.
Alex Rousskov [Wed, 25 Jul 2012 23:57:51 +0000 (17:57 -0600)]
Allow a ufs cache_dir entry to coexist with a shared memory cache entry
instead of being released when it becomes idle.
The original boolean version of the StoreController::dereference() code
(r11730) was written to make sure that idle unlocked local store_table entries
are released if nobody needs them (to avoid creating inconsistencies with
shared caches that could be modified in a different process).
Then, in r11786, we realized that the original code was destroying non-shared
memory cache entries if there were no cache_dirs to vote for keeping them in
store_table. I fixed that by changing the StoreController::dereference() logic
from "remove if nobody needs it" to "remove if somebody objects to keeping
it". That solved the problem at hand, but prohibited an entry to exist in
a non-shared cache_dir and in a shared memory cache at the same time.
We now go back to the original "remove if nobody needs it" design but also
give non-shared memory cache a vote so that it can protect idle non-shared
memory cached entries from being released if there are no cache_dirs to vote
for them.
Alex Rousskov [Sun, 22 Jul 2012 03:15:02 +0000 (21:15 -0600)]
Stop HttpRequest and AccessLogEntry memory leaks.
The leaks were introduced by recent request_header_add improvements (r12213)
which needed server-side access to client-side information like client
certificate so that it can be stuffed into the outgoing request headers.
Stuffing was done via Format API that uses AccessLogEntry as the source of
information.
This change breaks the HttpRequest->AccessLogEntry->HttpRequest refcounting
loop that prevented both request and ale objects from being destroyed. The ale
object is now delivered to the server side using FwdState::Start() API. Only
HTTP code currently takes advantage of ale availability on the server side.
Also had to modify httpHdrAdd() API because ale is no longer available via
the request pointer.
Alex Rousskov [Fri, 20 Jul 2012 23:11:02 +0000 (17:11 -0600)]
Avoid bogus "Disk space over limit" warnings when rebuidling dirty ufs index.
Subtract sizes of added-then-rejected entries while loading ufs cache index.
Before SMP changes, the ufs code was incorrectly ignoring the size of
loaded-but-not-yet-validated entries, leading to cache disk overflows.
After SMP changes, the ufs code was not subtracting sizes of
loaded-but-then-rejected entries, leading to bogus "Disk space over
limit" warnings. Now we correctly account for both kinds of entries.
author: Alex Rousskov <rousskov@measurement-factory.com>, Christos Tsantilas <chtsanti@users.sourceforge.net>
SslBump: Support bump-ssl-server-first and mimic SSL server certificates.
Summary: These changes allow Squid working in SslBump mode to peek at
the origin server certificate and mimic peeked server certificate
properties in the generated fake certificate, all prior to establishing
a secure connection with the client:
http://wiki.squid-cache.org/Features/BumpSslServerFirst
http://wiki.squid-cache.org/Features/MimicSslServerCert
The changes are required to bump intercepted SSL connections without
excessive browser warnings. The changes allow to disable bumping of some
intercepted SSL connections, forcing Squid to go into a TCP tunnel mode
for those connections.
The changes also empower end user to examine and either honor or bypass
most origin SSL server certificate errors. Prior to these changes, the
responsibility for ignoring certificate validation errors belonged
exclusively to Squid, necessarily leaving users in the dark if errors
are ignored/bypassed.
Squid can still be configured to emulate old bump-ssl-client-first
behavior. However, a manual revision of ssl_bump options is required
during upgrade because ssl_bump no longer supports an implicit "negate
the last one" rule (and it is risky to let Squid guess what the admin
true intent was or mix old- and new-style rules).
Finally, fake certificate generation has been significantly improved.
The new code guarantees that all identically configured Squids receiving
identical origin server certificates will generate identical fake
certificates, even if those Squid instances are running on different
hosts, at different times, and do not communicate with each other. Such
stable, reproducible certificates are required for distributed,
scalable, or fail-safe Squid deployment.
Overall, the changes are meant to make SslBump more powerful and safer.
The code has been tested in several independent labs.
Specific major changes are highlighted below:
Make bumping algorithm selectable using ACLs. Even though
bump-server-first is an overall better method, bumping the client first
is useful for backward compatibility and possibly for serving internal
Squid objects (such as icons inside Squid error pages). The following
example bumps special and most other requests only, using the old
bump-client-first approach for the special requests only:
ssl_bump client-first specialOnes
ssl_bump server-first mostOthers
ssl_bump none all
It allow use the old ssl_bump syntax:
ssl_bump allow/deny acl ...
but warns the user to update squid configuration.
Added sslproxy_cert_adapt squid.conf option to overwrite default mimicking
behavior when generating SSL certificates. See squid.conf.documented.
Added sslproxy_cert_sign squid.conf option to control how generated SSL
certificates are signed. See squid.conf.documented.
Added ssl::certHasExpired, ssl::certNotYetValid, ssl::certDomainMismatch,
ssl::certUntrusted, and ssl::certSelfSign predefined ACLs to squid.conf.
Do not require http[s]_port's key option to be set if cert option is given.
The fixed behavior for bumped connections now matches squid.conf docs.
Generate stable fake certificates by using signing and true certificate
hashes as the serial number and by using the configured CA private key
for all fake certificates.
Use minimal, trusted certificate for serving SSL errors to the client
instead of trying to mimic the broken true certificate (which results
in double error for the user: browser error dialog plus Squid error page).
To mimic "untrusted" true certificates, generate an untrusted CA certificate
from the configured trusted CA certificate. This both reduces configuration
effort (compared to a configuration option) and results in identical
untrusted fake certificates given identical Squid configurations.
Intelligent handling of CONNECT denials: Do not connect to origin
servers unless CONNECT is successfully authenticated. Delay errors.Added sslproxy_cert_sign squid.conf option to control how generated SSL
certificates are signed. See squid.conf.documented.
Provide '%I' error page formatting code with enough information to avoid
displaying '[unknown]' on SQUID_X509_V_ERR_DOMAIN_MISMATCH errors.
Set logged status code (%<Hs) to 200 when establishing a bumped tunnel.
Improved error detailing and logging: Forget most retried errors.
During SslBump errors, the error details are now logged with both the
initial CONNECT transaction and the first tunneled HTTP transaction.
Do not report system errors as custom Squid errors. Do not report
system errors that did not necessarily happen during the transaction
being logged.
Check SSL server certificate when reconnecting to the origin server for
bumped requests. Despite pinning, Squid maintains two separate connections
and the server may disconnect while the client is still sending requests. To
minimize deployment problems, we reconnect to the origin server but check
that its certificate (which we mimicked for the client) has not changed
much.
Forward bumped server connection-close signal to the bumped client to
improve the "dumb tunnel" appearance of the bumped SSL tunnel.
Allow bumping of CONNECT requests without allow-direct set on http_port.
Previously, that flag was required to allow bumped requests to go direct
because they were (and, sometimes, still are) considered "accelerated".
Send SNI information to the server when server-first bumping a non-IP
CONNECT request.
Better helper-to-Squid buffer size management to support large certificates.
Bypass rare OpenSSL certificate serialization failures when composing an
ssl_crtd request by generating the certificate in the Squid process.
When generating certificate CN names, strip [] surrounding host names,
assuming they are for IPv6 addresses. Bracketed CNs confuse browsers.
Disable persistent connections after client-side-detected errors. They cause
"abandoning such and such connection" warnings, stuck ConnStateData jobs, and
other problems.
HttpRequest::SetHost() must invalidate HttpRequest::canonical "cache".
Implement ssl::bump_mode logformat code to log SslBump decisions: prints "none",
"client-first", "server-first" or "-" for no ssl-bump enabled ports
Bug 3478: Partial fix: Connection-auth on intercepted connections is broken
Currenty in the case of intercepted connections each request is open a new
connection to the destination server, even if the connection is a valid pinned
connection.
This patch fixes this problem and reuses pinned connections on intercepted
requests.
Fail with an explanation instead of asserting. The assertion fails when a ufs
cache_dir's swap.state has inconsistencies AND the user starts Squid with a -S
command line option. Normally, such inconsistencies are ignored and many of
them are benign. For example, a missing cache file with an ADD record in
swap.state is such an inconsistency.
The -S option was meant to help developers troubleshoot inconsistencies by
analyzing core dumps, but (a) admins treat assertions as Squid bugs and file
bug reports and (b) in most cases, it is really difficult to find the
inconsistency when Squid asserts after detecting all of them (and leaving the
detection context).
We now explicitly tell the admin what their options are and quit instead of
asserting.
TODO: Consider adding a ufs cache_dir option that checks for and removes
inconsistencies instead of not checking at all (default) or checking and
quitting (-S). This is difficult because some valid cache entries may look
inconsistent while they are being updated and some invalid cache entries
are not visible to Squid without a full directory scan.
Handle Amos comments on bump-ssl-server-first patch
- Use "IF" instead of "IFDEF" in cf.data.pre. The IFDEF already used in the
IDEF<colon> form.
- Add an assertion in setServerBump to assure that even if this method reused in
a client request we will not try to change the existing bump mode
- Polish the code in doCallouts method: wrap if(!calloutContext->error) { ... }
around the whole section of callouts.
- The detailCode parameter in ErrorState::detailError(int detailCode) shadows
the ErrorState::detailCode parameter.
- src/url.cc: Reverting change in urlParseFinish
- Allow old "ssl_bump allow/deny acl ..." syntax converting:
* "ssl_bump allow" to "ssl_bump client-first"
* "ssl-bump deny" to "ssl_bump none"
Prints warning messages to urge users update their configuration.
Does not allow to mix old and new configurations.
Alex Rousskov [Wed, 18 Jul 2012 03:57:10 +0000 (21:57 -0600)]
Bug 3525: Do not resend nibbled PUTs and avoid "mustAutoConsume" assertion.
The connected_okay flag was not set when we were reusing the connection.
The unset flag overwrote bodyNibbled() check, allowing FwdState to retry
a PUT after the failed transaction consumed some of the request body
buffer, triggering BodyPipe.cc:239: "mustAutoConsume" assertion.
We now set the connected_okay flag as soon as we get a usable connection
and do not allow it to overwrite the bodyNibbled() check (just in case).
- Add request_header_add, a new ACL-driven squid.conf option that
allow addition of HTTP request header fields before the request is sent to
the next HTTP hop (a peer proxy or an origin server):
request_header_add <field-name> <field-value> acl1 [acl2]
where:
* Field-name is a token specifying an HTTP header name.
* Field-value is either a constant token or a quoted string containing
%macros. In theory, all of the logformat codes can be used as %macros.
However, unlike logging the transaction may not yet have enough
information to expand a macro when the new header value is needed.
The macro will be expanded into a single dash ('-') in such cases.
Not all macros have been tested.
* One or more Squid ACLs may be specified to restrict header insertion to
matching requests. The request_header_add option supports fast ACLs only.
- Add the %ssl::>cert_subject and %ssl::>cert_issuer logformating codes which
prints the Subject field and Issuer field of the received client SSL
certificate or a dash ('-').
The AccessLogEntry objects currently are only members of the ClientHttpRequest
objects. There are cases where we need to access AccessLogEntry from server
side objects to retrieve already stored informations for the client request and use it in
server side code with format/* interface (eg use Format::Format::assemble
inside http.cc)
This patch convert AccessLogEntry class to RefCountable to allow link it with
other than the ClientHttpRequest objects.
Restore memory caching ability lost since r11969.
Honor maximum_object_size_in_memory for non-shared memory cache.
Since r11969, Squid calls trimMemory() for all entries to release unused
MemObjects memory. Unfortunately, that revision also started trimming objects
that could have been stored in memory cache later. Trimmed objects are no
longer eligible for memory caching.
It is possible that IN_MEMORY check in StoreEntry::trimMemory() was preventing
excessive trimming in the past, but after SMP changes, IN_MEMORY flag is set
only after the object is committed to a memory cache in
StoreController::handleIdleEntry(), so we cannot rely on it. For example:
StoreEntry::unlock()
StoreController::handleIdleEntry() // never get here because entry is
set IN_MEMORY status // already marked for release
This change adds StoreController::keepForLocalMemoryCache() and
MemStore::keepInLocalMemory() methods to check if an entry could be stored in
a memory cache (and, hence, should not be trimmed). The store-related part of
the trimming logic has been moved from StoreEntry::trimMemory() to
StoreController::maybeTrimMemory(). StoreEntry code now focuses on checks that
are not specific to any memory cache settings or admission policies.
These changes may resulted in Squid honoring maximum_object_size_in_memory for
non-shared memory cache. We may have been ignoring that option for non-shared
memory caches since SMP changes because the check for it was specific to a
shared memory cache.
Bug 3577: File Descriptors not properly closed in trunk r12185.
Bug 3583: Server connection stuck after TCP_REFRESH_UNMODIFIED.
These changes fix FD leaks and stuck connections under two conditions:
1) Client aborts while Squid's server-side establishes a connection
Bug 3577: When a client quits while ConnOpener is trying to open the
connection to the next hop, FwdState cancels its ConnOpener callback.
ConnOpener notices that when trying to connect again and quits before
establishing a connection. The ConnOpener cleanup code did not close the
temporary FD used for establishing the connection. It did call fd_close(),
but fd_close() does not close the FD, naturally.
ConnOpener was probably leaking the temporary FD in other error handling
cases as well. It was never closed unless the connection was successful.
2) Client aborts after Squid's server-side established a connection:
Bug 3583: When a client aborts the store entry after receiving an HTTP 304 Not
Modified reply in response to a cache refreshing IMS request, HttpStateData
notices an aborted Store entry (after writing the reply to store), but does
virtually nothing, often resulting in a stuck server connection, leaking a
descriptor. Now we abort the server-side transaction in this case.
Bug 3577: Similarly, when a client disconnects after Squid started talking to
the origin server but before Squid received a [complete] server response,
HttpStateData notices an aborted Store entry (during the next read from the
origin server), but does virtually nothing, often resulting in a stuck server
connection, leaking a descriptor. Now we abort the server-side transaction in
this case.
FwdState now also closes the server-side connection, if any, when the client
aborts the store entry and FwdState::abort() callback is called. This helps
reduce the number of concurrent server-side connections when clients abort
connections rapidly as Squid no longer has to wait for the server-side I/O to
notice that the entry is gone. The code to close the connection was temporary
removed in trunk r10057.1.51.
projects / thirdparty / squid.git / log
