Alex Rousskov [Thu, 4 Aug 2011 07:18:25 +0000 (01:18 -0600)]
Added disk_io_timeout to squid.conf to control approximately how long Squid
allowsDisconnecting: Timeout, server not responding. queuing time. If the
anticipated I/O time exceeds the configured limit, Squid will not swap the
corresponding object in or out, allowing for the disk queues to drain.
Dmitry Kurochkin [Wed, 29 Jun 2011 05:59:07 +0000 (09:59 +0400)]
Always call processReplyBody() after adaptOrFinalizeReply().
The recent fix for "store_status == STORE_PENDING" added a check
with return from HttpStateData::processReply() before
processReplyBody() call. But processReplyBody() does proper
server state cleanup. We could end up with server state that
would not be cleaned (until connection is closed). That could
also introduce other issues. The patch moves the isAccepting()
check from processReply() to processReplyBody(). We do not write
to an entry that is not accepting, but allow processReplyBody()
to do proper cleanup.
Dmitry Kurochkin [Mon, 27 Jun 2011 20:58:29 +0000 (00:58 +0400)]
Separate shared page limits for different purposes.
The patch implements separate level counters for shared pages
with different purpose (i.e. memory cache and IPC I/O). A new
purpose parameter was added to Ipc::Mem::GetPage() function to
update the level and check the limits. After the change, memory
cache and IPC I/O limits are separate and do not affect each
other, i.e. full memory cache does not eat all IPC I/O pages.
Dmitry Kurochkin [Thu, 23 Jun 2011 23:35:43 +0000 (03:35 +0400)]
Fix assert in StoreEntry::write() when aborted entry is written.
HttpStateData::processReply() calls adaptOrFinalizeReply(), which
calls ServerStateData::setFinalReply(), which calls
entry->startWriting(), which calls StoreEntry::invokeHandlers() which
calls client-side code, which may abort or otherwise "close" the
entry.
After adaptOrFinalizeReply() returns to HttpStateData::processReply(),
HttpStateData::processReply() calls HttpStateData::processReplyBody(),
which may try to write to the aborted/closed entry, leading to
asserts.
At least the following scenario triggers the bug:
* client send a regular GET for a cached stale object
* Squid sends an IMS verification request to the origin server
* the origin server replies with 304 (Not Modified)
In this case adaptOrFinalizeReply() in HttpStateData::processReply()
would abort the origin server reply entry, leading to assert.
The patch adds a checks that entry still can accept more data after
the adaptOrFinalizeReply() call.
Dmitry Kurochkin [Tue, 21 Jun 2011 14:35:48 +0000 (18:35 +0400)]
Add shared memory page reserve for Rock and check page availability before IO.
The patch adds a shared memory page reserve for Rock (any IPC IO in
general) that can not be used by the memory cache. Also, Rock checks
if there are free pages available before trying to do IPC IO (both
read and write). This allows to fail early for cache reads and go
MISS route.
Alex Rousskov [Wed, 25 May 2011 03:14:15 +0000 (21:14 -0600)]
Detect support for __sync_add_and_fetch() and friends,
setting HAVE_ATOMIC_OPS accordingly.
Disable shared memory caching by default if atomic operations are not
supported. Prohibit shared memory caching if atomic operations are not
supported.
Alex Rousskov [Wed, 25 May 2011 00:33:52 +0000 (18:33 -0600)]
Added memory_cache_shared to squid.conf. Report whether mem_cache is shared.
Allow the user to explicitly disable shared memory caching in SMP mode.
Added YesNoNone class to allow Squid to compute the default value of a boolean
option (based on other options) instead of using a hard-coded default. The
class is used to automatically enable shared memory caching in SMP
environments (and disable it in non-SMP environments) unless the user says
otherwise.
This needs more work to reduce the number of shared memory pages if shared
memory caching is disabled (the pages may still be needed for SMP I/O).
Amos Jeffries [Tue, 24 May 2011 05:33:47 +0000 (17:33 +1200)]
Docs: fancy up ERR_INVALID_REQ with some javascript
Some of the "possible problems" can be determined as not-relevant.
When these cases are detected hide the text from viewers in a way that
keeps the page operational when javascript is disabled.
Amos Jeffries [Sat, 21 May 2011 01:13:42 +0000 (13:13 +1200)]
URL re-writer handling bug fixes
This patch includes two bug fixes in URL handling which were uncovered
during testing of the URL logging update:
* URL re-write handling was not correctly creating its adapted request
copy. The code here is much reduced by using the clone() method. Still
not completely satisfactory (marked with XXX) since on invalid URL
there is a wasted cycles cloning and deleting almost immediately.
Future cleanups moving the URL parts outside HttpRequest will fix that.
* URL parsing needs to set the canonical field to unset whenever the URI
is re-parsed into a request. This field is an optimization for later
display speed-ups. This has been causing incorrect canonical URL to be
used following re-write. When the cloning above was corrected it caused
asserts in the server-side.
* To prevent memory leaks the urnParse() function internal to URL parsing
is adjusted to accept and update an existing request in identical API
semantics to urlParse() instead of always generating a new one.
Currently, SSL error detail in Squid-generated error pages (%D) contains
both the error name and the explanation text. Some folks using this feature
want to render the two pieces of information differently because the error
name is not something most end-users should read or focus on.
This patch adds the "%x" error page formating code which prints the error name,
and removes the error name (%err_name) from SSL error detail messages.
This patch implements the phase 1 of the ICAP Max-Connections feature as it is
described in squid wiki:
http://wiki.squid-cache.org/Features/ServiceOverload
The behaviour of the patch can be configured using on_overload and max_conn
options of the icap_service configuration parameter. Squid can be configured
to do one of the following:
- Block: send and HTTP error response to the subscriber
- Bypass: ignore the "over-connected" ICAP service
- Wait: wait (in a FIFO queue) for an ICAP connection slot
- Force: proceed, ignoring the Max-Connections limit
Squid warns the first time the service become overloaded
For more information please visit the feature wiki page given above.
Technical informations:
The patch starts count a connections to the ICAP server as active when the
ModXact class receives an FD even if the fd is not really connected to the
server yet, and decrease the active connections to the server when the ModXact
object releases its fd connection.
If the Max-Connection limit is reached squid puts the request to a waiters list.
When one or more connections released squid schedules one or more waiters for
execution and remove them from waiters list.
To handle cases where a waiter gone/canceled before its execution the custom
dialer ConnWaiterDialer used.
The Options connections counted as active connections but are not limited by
the Max-Connections limit. An Option request will be executed even if the
maximum connections number is reached.
Alex Rousskov [Thu, 12 May 2011 03:58:16 +0000 (21:58 -0600)]
Removed Rock-specific code from StoreController.
Allow Store::dereference(e) to indicate whether the entry should
be kept in the global index. Old SwapDirs keep it. Newer code that
maintains entry tables dedicated to each cache_dir does not.
Tilmann Bubeck [Mon, 9 May 2011 12:42:59 +0000 (00:42 +1200)]
Add ext_time_quota_acl helper
Allows an administrator to define time budgets for the users of squid
to limit the time using squid.
This is useful for corporate lunch time allocations, wifi portal
pay-per-minute installations or for parental control of children. The
administrator can define a time budget (e.g. 1 hour per day) which is
enforced through this helper.
Andrew Beverley [Sun, 8 May 2011 23:21:44 +0000 (11:21 +1200)]
QoS: require libcap before enabling netfilter MARK support
As it is not possible to get or set a netfilter mark without libcap, this
patch will disable netfilter marking at compilation time if libcap is not
available (in a similar way to Linux transparent proxying).
Amos Jeffries [Sun, 8 May 2011 13:53:10 +0000 (01:53 +1200)]
Cleanup: sync NTLM and Negotiate UserRequest code
Minor tweaks to reduce diff between the files. No logic changes.
Renames the addHeader() to addAuthentiocationInfoHeader(),
Renames the addTrailer() to addAuthentiocationInfoTrailer() and
document that they add additional *-Info header to the HTTP reply.
Amos Jeffries [Sun, 8 May 2011 06:11:18 +0000 (18:11 +1200)]
Cleanup: Improve Connection Pinning management
Since 1xx handing went in HttpRequest has had two links to the one
ConnStateData managing its client connection.
* Rename the 1xx link to clientConnectionManager (since it is not
actually the connection, but the manager object controlling the FD
usage and stats.
* Convert the pinning code to using the permanent clientConnectionManager
link instead of a temporary pinned_connection link.
This moves all connection pinning state fully into the ConnStateData
manager objects scope.
Side changes that appear to be buggy code previously:
* do not alter pinning state at the point where the pinned connection is
about to start being used. Changes are only relevant at the point of
pinning or unpinning.
* unpin operation now closes the Server FD if still open. Previously
there was the possibility that some code paths would leave server FD
open and pconn it. (especially since the above mentioned state
alteration cleared the "pinned" flag).
Amos Jeffries [Fri, 6 May 2011 16:16:45 +0000 (04:16 +1200)]
Implicit Dependency removal for gcc-4.6.1
GCC 4.6.1 is stricter than 4.6.0. It does not by default include implicit
dependencies. This adds several unit tests .cc files which were implicitly
linked before.
Also adds tests/stub_DiskIOModule.cc to short-circuit the DiskIOModule API
Bug #3214: "helperHandleRead: unexpected read from ssl_crtd" errors.
Squid would read the beginning of a crtd response split across multiple
read operations and treat it as a complete response, causing various
certificate-related errors.
This patch:
- allow the use of other than the '\n' character as the end of message mark
for helper responses.
- Use the '\1' char as end-of-message char for crtd helper. This char looks
safe because the crtd messages are clear text only messages.
Amos Jeffries [Wed, 4 May 2011 03:05:09 +0000 (15:05 +1200)]
Compile fixes for binutils-gold and gcc-4.6 support
These two tools are much stricter about dependency linkages. We have already
had to drop testAuth due to major dependency loops they dislike.
This makes the remainder of the dependency changes needed.
Also adds:
- tests/STUB.h with macros for simpler stub file creation
- stub_libmgr.cc for unit-test stub replacment of mgr/libmgr.la library.
many API functions commented out, but sufficient for the current needs.
Amos Jeffries [Mon, 2 May 2011 13:04:21 +0000 (01:04 +1200)]
Drop testAuth unit-tests
Preparing to move the tests into src/auth.
These old tests construction style also require quite a lot of dependencies
which include several loops causing problems in modern strict linkers.
Opted to remove now and stabilize trunk without it before re-adding simpler
auth unit tests.
Amos Jeffries [Mon, 2 May 2011 01:14:30 +0000 (19:14 -0600)]
Cleanup: base64 coder de-duplication and upgrade
Markus Moeller has re-implemented several of the coder functions for use
by Kerberos helpers.
This patch seeks to de-duplicate them and combine the resulting code
back into the libmiscencoding.la "base64.h" implementation.
Changes include:
* old function API renamed to old_*() and existing code update to use
the names. Some code has been updated to use the new API.
* new estimator base64_encode_len()/base64_decode_len() functions added
to provide details of much much buffer space the output will require.
* new API encoder and decoder functions added which accept caller
provided buffers and encode/decode an arbitrary string into them.
* also fixes a bug where if the input text or output buffer was too
short the coder functions would crop a few bytes off the end of the result.
Noticable in Kerberos where token lengths are not fixed length.
Some optimizations have been added by myself over and above Markus changes:
* optimized to short-circuit on several more variations of empty input
and nil result buffer.
* sub-loop optimizations added to reduce the number of if() calls made
by the new code.
* split encoder into terminated (C-str) and non-terminated variants.
James Bowe [Sun, 1 May 2011 12:10:37 +0000 (00:10 +1200)]
Add external_acl_type %EXT_LOG and %EXT_TAG format options.
%EXT_LOG and %EXT_TAG are filled with the log= and tag= fields
returned by previous external ACLs.
-for a string that never changes after it is set, tag= is suitable.
-for a string that may need updating or overwriting by a later
external_acl, log= is suitable.
Under both circumstances it is conceivable that later external_acls
may need access to the tag= or log= values after they have been set
(e.g. for external_acl debugging, merging log messages, etc).
Alex Rousskov [Thu, 28 Apr 2011 22:45:55 +0000 (16:45 -0600)]
Extraced the write-to-store step from StoreEntry::replaceHttpReply().
This allows the caller to set the reply for the entry and then update the
entry and the reply before writing them to store. For example, the server-side
haveParsedReplyHeaders() code needs to set the entry timestamps and make the
entry key public before the entry starts swapping out, but the same code also
needs access to entry->getReply() and such for timestampsSet() and similar
code to work correctly.
TODO: Calls to StoreEntry::replaceHttpReply() do not have to be modified
because replaceHttpReply() does write by default. However, it is likely that
callers other than ServerStateData::setFinalReply() should take advantage of
the new split interface because they call timestampsSet() and such after
replaceHttpReply().
The bug appeared after commit with revno:11364 which fixes the Bug #3192.
In the case of SSL-bumped connections the ConnStateData::flags.readMore flag
must be reset (set to true) when we are switching to HTTPs,
because we have to read the new unencrypted HTTP request.
This patch reset this flag in ConnStateData::switchToHttps method.
In the common default case there are no reply body limits configured.
There is no need to construct ACL checklists for testing. This saves
one allocation and several locking/unlocking cycles per request.
Rework shared queue for IpcIoFile, further optimize IpcIo notifications.
The patch implements a FewToFewBiQueue class that allows
communication between two group of processes. The queue is used
in IpcIoFile and allows to have a single shared queue reader
state for each process (both diskers and workers). This
continues the optimization started in r11279, see commit log for
more details.
The patch also decreases the number of shared memory segment used
by queues. Before the change, FewToOneBiQueue used
(2*workerCount + 1) number of segments. Now FewToFewBiQueue uses
just three: for shared metadata, for array of one-to-one queues
and for array of queue readers.
Before the patch, each shared object was responsible for allocating
and deallocating shared memory it uses. As a result each object had a
shared and non shared portion. Shared classes provided a pair of
static methods for creating and attaching to existing shared segments.
This is against how normal objects behave: normal objects are not
responsible for managing memory they use, they use the memory they are
given. Besides the old approach mixes shared memory management and
object initialization logic. The patch tries to improve this.
On the user side, the patch provides two functions for managing shared
objects:
* shm_new - allocates/deallocates shared memory, initializes the object
* shm_old - gives refcounted access to the object created by shm_new
Shm_new function returns so called Owner object. It is not used for
working with the shared object, but to do shared memory
allocation/deallocation and object initialization. This function will
be typically used in Squid master process to allocate shared memory on
startup. On exit, the Owner object is deleted and shared object is
deallocated.
Shm_old function returns a refcounted smart pointer to the shared
object. It does not allocate shared memory or initialize the object,
but just points to the object owned by the Owner. Smart pointer
provides a simple way for working with the shared object.
On the internal side, the patch removes shared memory
allocation/deallocation from shared object class. There is no more
local/shared parts. Shared object class implementation is now similar
to an ordinary class. The additional requirements for "shared"
classes are: the object must be a POD with no pointers to or
references; provides a static SharedMemorySize method for shared
memory size calculation; may need to use atomic primitives for safe
updates of data members.
All existing "shared" classes and code were converted to the new API.
Alex Rousskov [Thu, 21 Apr 2011 15:19:31 +0000 (09:19 -0600)]
Temporary fix for coredumps during shutdown cleanup.
For a permanent fix, we need to avoid deleting fd_table while it is still
in use by others, such as DeferredReads, possibly by allowing event loop
to run during shutdown.
projects / thirdparty / squid.git / log
