Henrik Nordstrom [Fri, 27 Jan 2012 12:47:06 +0000 (05:47 -0700)]
Send DNS A and AAAA queries in parallel
This implements sending DNS A & AAAA queries in parallel by
creating "slave" idns_query requests. Current implementation
uses the A lookup as "master" and AAAA as "slave" query.
Long term this should probably be restructured to separate "lookup
state" and "query", or even better yet to defer the DNS lookups
until connect time and perform A respective AAAA as needed only and
not look up both before attemting to connect.
This also drops the dns_v4_fallback directive as it have no effect with
parallel DNS lookups. This directive should be reinstanciated in future
to enable pure IPv6 usage.
Inside StoreEntry::swapOut() the StoreEntry::trimMemory() method called to
release unused MemObjects memory. The trimMemory method must called for all
store entries, but current code blocks that call for not-swappable objects,
at least.
This patch trying to fix this bug implementing the following simple logic:
{
bool weAreOrMayBeSwappingOut =
swappingOut() || mayStartSwapout();
trimMemory(weAreOrMayBeSwappingOut);
if (!weAreOrMayBeSwappingOut)
return; // nothing else to do
}
Amos Jeffries [Fri, 27 Jan 2012 12:33:17 +0000 (05:33 -0700)]
Cache Manager migration support
* Add a little bit of XHR script to the CGI cachemgr front page which
probes each of the managed proxies for http:// and https:// capabilities
and produces web links to their internal managers.
* Reserve the template name MGR_INDEX for use by cachemgr scripts.
But do not distribute any preset template. This allows manager apps to
provide their own static template with linked scripts and objects.
* The error page system is updated to create a blanket message
indicating missing template instead of aborting Squid if a template is
not even installed.
Also, add Cross-Origin Request Support for the cache manager API
Now that tools are being implemented to access the cache manager via
http:// scheme we need to accomodate the browser XSS protection
mechanisms which limit XHR based scripts abilities.
This adds CORS headers to manager responses. Permitting XHR to view the
Server header (to detect squid version for known capabilities) and to
flag that the XHR request may need access to credentials for
authenticating with the manager.
This also closes the feature bug 3407 requesting we support the
non-standard "Origin:" header, which is used by the CORS mechanisms.
Future work:
Support the OPTIONS request used by CORS to detect requirements before
POSTing. We do not yet use POST in the Squid code so that is left until
needed.
Alex Rousskov [Sat, 21 Jan 2012 01:13:53 +0000 (18:13 -0700)]
Do not add HTTP 110 and 111 Warnings to TCP_REFRESH_UNMODIFIED responses.
The old "stale if hit" logic did not account for cases where the stored
stale response became fresh due to a successful revalidation with the
origin server.
When the stored response was stale at the time of the request, we were
adding 110 "Response is stale" and even 111 "Revalidation failed"
Warning headers to responses while logging TCP_REFRESH_UNMODIFIED, which
is considered a hit.
Amos Jeffries [Tue, 17 Jan 2012 08:34:49 +0000 (01:34 -0700)]
Cleanup: unlinkd is self-contained
This helper is very minimal. It does not use IP:: or Ipc:: symbols.
If we get linkage issues against it for them those need to be resolved
without pulling in dependencies from squid internal libraries.
Amos Jeffries [Sat, 7 Jan 2012 06:05:46 +0000 (23:05 -0700)]
SourceLayout: shuffle UserRequest into Auth:: namespace
No logic changes, symbols and debug only.
* shuffle AuthUserRequest to Auth::UserRequest
* shuffle AuthBasicUserRequest to Auth::Basic::UserRequest
* shuffle AuthDigestUserRequest to Auth::Digest::UserRequest
* shuffle AuthNegotiateUserRequest to Auth::Negotiate::UserRequest
* shuffle AuthNTLMUserRequest to Auth::NTLM::UserRequest
* polish and add documentation for several parts of the hierarchy
Amos Jeffries [Thu, 22 Dec 2011 07:34:08 +0000 (00:34 -0700)]
Bug 3370: external ACL sometimes skipping
Emit tag/user/log/message/pass details to the request in the case where
the external ACL entry has expired but within graceful revalidate period.
The result of this bug appears as incorrect matches later down the ACL
processing in any config relying on the external ACL output values.
Example; for bypassing auth login, or for filtering tagged traffic.
When the ssl_bump and authentication iare both enabled for an ssl-bumped port
all SSL enabled websites prompt the user for authentication information once
per FQDN.
This patch inherits the authentication info from the CONNECT request to the
sslbumped requests.
Bug 3448: 204 response problem in adaptation chains
When the first ICAP service in a chain respond with 204 the next service
is aborted on Must(old_request->canonical) expression inside Adaptation::Icap::ModXact::encapsulateHead method.
Squid ICAP try to set the request::canonical member of the adapted request
inside Adaptation::Icap::ModXact::prepEchoing when the 204 response received.
The adapted.header->parse(..) call some lines after will set canonical member
to NULL.
This patch calls the urlCanonical() function after parse() method
to build canonical member for the adapted request, instead of trying to copy
this member from the original request.
Alex Rousskov [Thu, 8 Dec 2011 10:57:39 +0000 (03:57 -0700)]
Bug 3420: Request body consumption races and !theConsumer exception.
Also fixes endless waiting for HTTP client to send req body we no longer need.
Before these changes, the client side used a single "closing" state to
handle two different error conditions:
1. We stopped receiving request body because of some error.
2. We stopped sending response because of some error.
When a "directional" error occurred, we try to keep the transaction going in
the other direction (e.g., to give ICAP the entire request or to give HTTP
client the entire response). However, because there was just one "closing"
state, the code failed to correctly detect or process many corner cases,
resulting in stuck transactions and !theConsumer assertions/exceptions due to
races between enableAutoConsumption() and expectNoConsumption() calls.
This patch replaces the "closing" state with two direction-specific "we
stopped sending/receiving" flags.
Now, when the response sending code is done, it now checks whether the
receiving code stopped and closes the connection as needed. This is done both
when we encounter a sending error (ClientSocketContext::initiateClose) and
when we successfully sent the entire response to the client
(ClientSocketContext::keepaliveNextRequest).
Similarly, when the request body reading code is done, it now checks whether
the receiving code stopped and closes the connection as needed. This is done
both when we encounter a receiving error
(ConnStateData::noteBodyConsumerAborted) and when we successfully receive the
entire request body from the client (ClientSocketContext::writeComplete).
TODO: This patch focuses on various error cases. We might still have problems
when there is an early HTTP response and no errors of any kind. I marked the
corresponding old code with an XXX.
External ACL sometimes cannot find the credentials in ACL Checklist even
if they are attached to the HTTPRequest object.
This seems to happen when the checklist is created and the line match
started before the credentials are known. The credentials validation
updates the HTTP request state but is not aware of ACL checklists needing
to be updated so it never happens.
This patch:
* locate the %LOGIN value from either place where credentials can be found,
* updates the checklist if it was unset,
* passes '-' to the helper if no credentials at all were given.
Although the earlier logics forcing a lookup means this '-' case should
not happen it might if the external ACL were processed in 'fast' check.
Amos Jeffries [Thu, 8 Dec 2011 10:51:09 +0000 (03:51 -0700)]
Add FdeCbParams parameter object to CommCalls API.
The problem:
CommCalls API functionality is conflated with comm operational calls
created to do general FD handling (FD as pipe handle, FD as disk handle,
FD as pointer into the fd_table structure). Sometimes because they do
operations mirroring comm handlers and also use FD. None of this actually
requires the CommCalls layer to be involved though. The Comm::Connection
objects which CommCall TCP handlers pass around is also very inappropriate
for these FD types.
This adds FdeCbParams to CommCalls infrastructure, for use internally and
"lower" than comm API to pass around raw FD values. This should be avoided
on TCP socket FD, but may be used by callers needing FD where
Comm::Connection is inappropriate.
Amos Jeffries [Thu, 8 Dec 2011 10:49:51 +0000 (03:49 -0700)]
CBDATA call Dialer template
This adds a template for dialing Unary CBDATA wrapper functions with
type safety. Avoiding the casting that currently occurs in wrappers and
allowing the AsyncCall APIs to be used for these callbacks.
Fix: HttpRequest flags intercepted, spoof_client_ip never set correctly
The request_flags::intercepted,request_flags::spoof_client_ip are 1 bit integers
so when you are try to set to an integer bigger than 1 will overflow and the
results will not be what you are expecting.
The StoreEntry::write in the case of an empty write, calls the StoreEntry
handlers. It is possible one of these handlers will change the state of the
store entry or abort it. The next call of the StoreEntry::write will cause
this assertion.
The block of code which calls the StoreSntry handlers in the case of an empty
write, added to allow forward http headers to the client even if no body data
arrived yet (bug 1750). There is not need for this part of code in the latest
squid releases, so it is safe to be removed.
Alex Rousskov [Thu, 24 Nov 2011 07:03:00 +0000 (00:03 -0700)]
Avoid crashes when processing bad X509 common names (CN).
X509_REQ_get_pubkey() returns a refcounted object that we must clean after use.
X509_REQ_get_subject_name() does not; cleaning the result may cause segfaults.
How we are supposed to tell the difference is beyond me.
Amos Jeffries [Thu, 24 Nov 2011 07:01:41 +0000 (00:01 -0700)]
Cleanup: comm Close handlers
Make handlers take the CommCloseCbParams instead of series of expanded
variables.
Opening access to the other CommCommonCbParams fields with Connection/FD
data. Hiding the deprecated FD parameter from most handlers. Which seem
not to have actually needed it in most cases outside Comm.
Martin Huter [Thu, 24 Nov 2011 06:37:32 +0000 (23:37 -0700)]
Bug 2619: Excessive RAM growth due to unlimited adapted body data consumption
If the client does not read from the open connection (i.e. the user does not
confirm the browsers download-message-box in microsofts IE), squid keeps on
reading data from the ICAP server into the store entry, while no more data
can be delivered to the client.
Thus the store entry in memory is growing and squid may - in worst case -
consume memory up to the size of the users download.
This patch add API to StoreEntry to call the producer back when released
memory/space from the StoreEntry and add code to the ICAP client code to not
consume body data comes from the ICAP server when there is not available space
in the store entry.
Amos Jeffries [Thu, 24 Nov 2011 06:13:51 +0000 (23:13 -0700)]
Log Format token namespace upgrade
This updates the format parser and storage objects in the Format::
namespace and separates some into separate files.
Add a registration API so components can register themselves an array
of tokens in a namespace. Registering the arbitrary namespace "example"
with some tokens ("a","b") will cause the parser to accept those tokens
in a logging format like so: "%example::a %example::b".
Future work:
- use runners registry instead of Init() function
- convert the error pages to use format for the page body macros
- convert the %ssl_* tokens in src/ssl/* to use format and "ssl::"
- convert external_acl_type to use formats for its helper input string.
Mathias Fischer [Thu, 24 Nov 2011 06:12:49 +0000 (23:12 -0700)]
Use the right certificate when detailing SSL certificate validation errors.
When an _intermediate_ SSL server certificate fails validation, we should
report errors using information in that certificate and not in the top-level
"peer" certificate. Otherwise, our details may make no sense. For example, we
may say that the validation failed due to the expired certificate and then show
an expiration date in the future (because the top-level certificate did not
expire but the intermediate certificate did).
OpenSSL X509_STORE_CTX_get_current_cert() returns the certificate that was
being tested when our certificate validation callback was called.
Alex Rousskov [Thu, 17 Nov 2011 09:34:09 +0000 (02:34 -0700)]
Fix Comm::Write closing() assertion when retrying a failed UDP DNS query.
When we receive a UDP DNS response with a truncation (TC) bit set, we retry
using TCP. Since the retry trigger has nothing to do with the TCP connection,
it is possible that the TCP connection is being closed when we are about to
write to it: A call to our connection close callback has been scheduled but
has not fired yet. We must check for and avoid such race conditions.
Amos Jeffries [Thu, 17 Nov 2011 09:26:50 +0000 (02:26 -0700)]
Document and alter the pconn idle timeout directives.
Alters the directive names to clarify what they do and adds some more
description to the config file documentation.
Alters the internal config variables to match the new directive names.
Also alters the well known messages in mgr:filedescriptors report a little
to indicate client/server type and adds a standard "Idle " prefix for
easy automated scanning.
This patch allows Squid to provide details for the %D macro on the secure
connect failed error page when an SSL handshake with the origin server fails.
The default %D text is "Handshake with SSL server failed: XYZ" where XYZ is the
corresponding error string/description returned by OpenSSL if there is any.
Andrew Beverley [Sun, 6 Nov 2011 01:38:17 +0000 (19:38 -0600)]
Add a mask on the qos_flows miss configuration value.
The reason for this is to allow the preserved mark/TOS value from the
server to be altered slightly rather than overwritten completely.
Example usage. The following will preserve the netfilter mark, but will
ensure that the (9th) bit specified in the miss value will be set to 1
in the preserved mark:
Amos Jeffries [Sun, 6 Nov 2011 00:03:50 +0000 (18:03 -0600)]
Fixed two more cases of outdated shared memory cache detection
... which led to "STORE_DISK_CLIENT == getType()" assertions
when running SMP Squid with non-shared memory caching.
UsingSmp() is not the right condition to detect whether we are using a shared
memory cache because shared memory caching may be disabled and because
Coordinator does not use a shared memory cache even if shared caching is
enabled.
This option allows Squid administrator to add custom ICAP request
headers or eCAP options to Squid ICAP requests or eCAP transactions.
Use it to pass custom authentication tokens and other
transaction-state related meta information to an ICAP/eCAP service.
The addition of a meta header is ACL-driven:
adaptation_meta name value [!]aclname ...
Processing for a given header name stops after the first ACL list match.
Thus, it is impossible to add two headers with the same name. If no ACL
lists match for a given header name, no such header is added. For example:
# do not debug transactions except for those that need debugging
adaptation_meta X-Debug 1 needs_debugging
# log all transactions except for those that must remain secret
adaptation_meta X-Log 1 !keep_secret
# mark transactions from users in the "G 1" group
adaptation_meta X-Authenticated-Groups "G 1" authed_as_G1
The "value" parameter may be a regular squid.conf token or a "double
quoted string". Within the quoted string, use backslash (\) to escape
any character, which is currently only useful for escaping backslashes
and double quotes. For example,
"this string has one backslash (\\) and two \"quotes\""
Dmitry Kurochkin [Sat, 29 Oct 2011 11:49:34 +0000 (05:49 -0600)]
Provide "fake" AtomicWordT implementation for non-SMP configurations.
While we can not provide real AtomicWordT implementation on the systems where
atomic operations are not available, we can use a "fake" one if Squid is
running in non-SMP mode. Before the change, the "fake" implementation was
always asserting, which is too restrictive and leads to test failures on
systems without atomic operations.
The new implementation works under conditions similar to "fake" shared memory
segments and allows SMP-using code (e.g. Rock store) to work in non-SMP mode.
In particular, it allows tests to pass on such systems.
AtomicWordT was renamed to WordT and moved to Ipc::Atomic namespace to allow
Ipc::Atomic::Enabled() to be declared outside of the AtomicWordT template
class. This lets us define the Enabled() method in AtomicWord.cc which avoids
dragging protos.h #include into the AtomicWord.h header.
Dmitry Kurochkin [Sat, 29 Oct 2011 11:47:24 +0000 (05:47 -0600)]
Bug 3150: do not start useless unlinkd.
Unlinkd may be used only by UFS storage but, before the change, Squid
always started unlinkd if it was built, even if it was not needed.
Whether a SwapDir may use unlinkd depends on the SwapDir
implementation and DiskIO strategy it uses. The patch adds
unlinkdUseful() method to SwapDir and DiskIOStrategy to decide if
unlinkd should be started.
After the change, unlinkd may be started during reconfiguration and
unlinkdInit() may be called multiple times.
After the change, unlinkdClose() may be called when unlinkd was never
started. The patch removes a warning which was printed in this case
on Windows.
Dmitry Kurochkin [Sat, 29 Oct 2011 11:46:10 +0000 (05:46 -0600)]
Optimization: Make read requests in [Rock] IpcIo bypass max-swap-rate limit.
Before the change, IpcIoFile::WaitBeforePop() delayed both swap ins
(hits) and swap outs (misses). This is suboptimal because reads do
not usually accumulate unfinished I/O requests in OS buffers and,
hence, do not eventually require the OS to block all I/O.
Ideally, a disker should probably dequeue all pending disker requests,
satisfy reads ASAP, and then handle writes, but that is difficult for
several reasons. The patch implements a simpler approach: peek the
next request to be popped, and if it is a swap in (i.e., read or hit),
then pop it without any delay.
When a read is popped, we still adjust the balance member and LastIo,
because we do want to maintain the configured average I/O rate. When a
write request comes in, it will be delayed [longer] if needed.
In the extreme case of a very long stream of read requests (no writes
at all), there will be essentially no I/O rate limit and that is what
we want.
Dmitry Kurochkin [Sat, 29 Oct 2011 11:43:41 +0000 (05:43 -0600)]
Do not allow max-swap-rate and swap-timeout reconfiguration for Rock Store.
These options are used to configure DiskIO module during Rock SwapDir
initialization. During reconfiguration, the values are updated in Rock
SwapDir, but they do not reach the DiskIO module. Thus, while Squid says that
option has a new value, the new value is never really used. This patch fixes
this inconsistency.
In the future, we may support reconfiguration for max-swap-rate and
swap-timeout, but that would require adding reconfiguration support
to DiskIO modules.
Dmitry Kurochkin [Sat, 29 Oct 2011 11:41:43 +0000 (05:41 -0600)]
Independent shared I/O page limit.
Shared memory pages are used for shared memory cache and IPC I/O module.
Before this change, the number of shared memory pages needed for IPC I/O
was calculated from the size of shared memory cache. Moreover, shared
memory cache was required for IPC I/O.
The patch makes the limit for shared I/O pages independent from the
shared memory cache size and presence. IPC I/O pages limit is calculated
from the number of workers and diskers; it does not depend on cache_dir
configuration. This may change in the future if we learn how to compute
it (e.g., by multiplying max-swap-rate and swap-timeout if both are
available).
projects / thirdparty / squid.git / log
