Alex Rousskov [Wed, 1 Jan 2014 20:13:23 +0000 (13:13 -0700)]
Initial Large Rock and Collapsed Forwarding support.
Large Rock: Support disk (and shared memory) caching of responses exceeding
one db slot (or one shared memory page) in size. A single db slot/page size is
still limited to 32KB (smaller values can be configured for disk caches using
the newly added cache_dir slot-size option). Removal of old rock cache dir
(followed by squid-z) is required -- the on-disk db structure has changed.
Collapsed Forwarding: Optionally merge concurrent cachable requests for the
same URI earlier: After the request headers have been parsed (as before), but
now _before_ the response headers have been received. Merging of requests
received by different SMP workers is supported. Controlled by the new
collapsed_forwarding directive in squid.conf. Disabled by default because all
but one of the merged requests have to be delayed (until the response headers
are received) for the merging to work, which may be worse than forwarding all
concurrent requests immediately. The overall feature idea and request
eligibility conditions are based on Collapsed Forwarding in Squid2.
Summary of other important changes (merged branch log contains the details):
* Tightened StoreEntry locking. Split StoreEntry::lock() into "just lock" and
"update entry reference time" interfaces, addressing an old XXX. Improved
entry lock/unlock debugging. Needs more work.
* Adjusted StoreIOState::write() API to allow callers detect write errors.
* Simplified MemObject::write() API to remove an essentially unused callback.
* Mark client streams that sent everything as STREAM_COMPLETE. The old code
used STREAM_UNPLANNED_COMPLETE if the completed stream was associated with a
non-persistent connection, which did not make sense to me and, IIRC, led to
store entry aborts even though the entries were not damaged in any way.
* mem_hdr::hasContigousContentRange() now returns true for empty ranges.
* Support "appending" ReadWriteLock state that can be shared by readers and
the writer. The writer promises not to update key metadata (except growing
object size and next pointers) and readers promise to be careful when
reading growing slices.
* Fixed StoreEntry::mayStartSwapOut() logic to handle terminated swapouts.
* Improved STORE_MEM_CLIENT detection and documented known (and mostly old)
StoreEntry::storeClientType() problems.
* Removed StoreEntry::hidden_mem_obj hack.
* Polished StoreEntry debugging to report more info, less noise. Use e: prefix.
* Added a script to extract store entry(ies) debugging from cache.log.
Alex Rousskov [Fri, 27 Dec 2013 18:37:26 +0000 (11:37 -0700)]
Updated tests/testRock and the store rebuild stubs it needs.
Synced tests/testUfs after updating the store rebuild stubs.
Store uses StoreController::store_dirs_rebuilding to decide whether the entry
release should be delayed. Thus, storeRebuildComplete() must update it. Also
synced the corresponding CPPUNIT_ASSERT_EQUAL() statements to expect a zero
value after the rebuild is completed.
Do not create an entry just to get its key: Creating forces a public key which
necessarily invalidates the previously cached entry with the same key, if any.
Unlock unused entries. This helps with their release later (and adds more
realism to the test case).
Fixed entry #5 test case to account for the fact that creating a second entry
with the same public key invalidates the first entry.
Release instead of just unlinking entries. StoreEntry::release() is the public
interface which updates internal tables as needed. StoreEntry::unlink() is an
internal call for updating the disk cache-related state; it does not update
the the in-transit index.
Alex Rousskov [Mon, 23 Dec 2013 05:26:23 +0000 (22:26 -0700)]
Bug 3806: Caching responses with Vary header
Various fixes making Vary caching work better.
More work is needed to re-enable shared memory caching of Vary responses.
bag5s r12741: Do not start storing the vary marker object until its key becomes public.
bag5s r12742: Log failed (due to "Vary object loop" or "URL mismatch") hits as TCP_MISSes.
bag5s r12743: Refuse to cache Vary-controlled objects in shared memory (for now).
Jakob Bohm [Sat, 21 Dec 2013 17:31:59 +0000 (09:31 -0800)]
Bug 3985: 60s limit introduced by balance_on_multiple_ip breaks bad IP recovery
Alternative fix if necessary would be to calculate a dynamic timeout
value based on number and timeouts for connections and forwarding.
However, as noted in the bug report the address markings are reset every
TTL when the stored RR are replaced. This meets the criteria for periodic
retries of marked-BAD IPs. Also if all IPs are marked BAD they all get
un-marked and retried. So for now just remove the TTL hack.
Amos Jeffries [Sat, 21 Dec 2013 04:54:54 +0000 (20:54 -0800)]
Fix \-unescaping in quoted strings from helpers
strwordtok() was unescaping '\' values in unquoted tokens. This causes
problems with NTLM helper response user\DOMAIN values which are sent as
a un-quoted un-escaped value.
Restrict un-escaping of '\' to only occur when inside "quoted" words.
Amos Jeffries [Fri, 20 Dec 2013 11:34:44 +0000 (03:34 -0800)]
WCCPv2: fix assertion 'Cannot convert non-IPv4 to IPv4' on FreeBSD
FreeBSD does not set the from-address on recv() if there was an error
reading the packet. This can result in Ip::Address assertion if the
error is not checked for before the IP address is mapped into Squid
internal format.
Alex Rousskov [Wed, 18 Dec 2013 17:19:00 +0000 (10:19 -0700)]
Added send_hit and store_miss squid.conf directives
to control caching of responses using response info.
The existing "cache" directive is checked before Squid has access to the
response and, hence, could not use response-based ACLs such as http_status.
Response-based ACLs may be essential when fine-tuning caching. Squid Bug 3937
(StoreID can lead to 302 infinite loop) is a good use case.
Updated old "cache" directive documentation to provide more information, to
help folks distinguish the three related directives, and to polish for
clarity.
TODO: Support lookup_hit and possibly deprecate/remove "cache".
Moved CharacterSet's backing storage to std::vector<uint8_t> to gain speed at the expense of space.
Improved CharacterSet's constructor
Made CharacterSet's name parameter optional with default value.
Amos Jeffries [Thu, 12 Dec 2013 09:41:39 +0000 (01:41 -0800)]
Fix linker errors "relocation R_X86_64_32 against .rodata"
... "can not be used when making a shared object; recompile with -fPIC"
Caused by some libraries built without libtool in certain environments.
NP: there are other libraries in Squid built without libtool but they
appear not to be having these linker issues at this time.
Also, fixes "undefined symbol eui64_aton" uncovered by these changes.
Amos Jeffries [Wed, 11 Dec 2013 11:12:43 +0000 (03:12 -0800)]
Regression in URL helper API
The backward compatibility logics in redirect.cc are not working as
intended on redirection URLs due to the presence of '=' in the URL and
how the key=value name parsing is performed.
A typical redirection URL looks like:
http://example.com/?url=http://www.example.net/
and 3.4 has a parser that splits tokens at '=' unconditionally and then
passes the bits as a key and value to the redirector logics which
complains that it does not understand the answer of the URL redirector.
Or treats is an an unknown key=value with no redirection URL.
Either case is handled as a no-redirection result from the helper.
This limits the key names to alphanumeric, hyphen and underscore
characters. Valid URL responses contain characters outside this set and
should no longer be interpreted as keys regardless of the '=' character.
This patch add the %tS logformat code to represent master transaction start time
in <seconds>.<milliseconds> format, similar to the existing access.log "current
time" field (%ts.%03tu).
Also allow formated values for adaptation_meta values.
Alex Rousskov [Mon, 2 Dec 2013 00:36:24 +0000 (17:36 -0700)]
Documented that ACLs should be refcounted.
Refcounting would avoid ACL destruction problems (when some ACLs use other
ACLs or when an ACL goes async during reconfigure) and may even reduce
problems with AclMatchedName.
Alex Rousskov [Mon, 2 Dec 2013 00:35:50 +0000 (17:35 -0700)]
Destroy ACLs in the reverse order of creation to avoid destruction segfaults
during reconfiguration.
Group ACLs created later may use other ACLs created earlier. A group ACL must
be deleted first so that its AclDeleter can safely access registration status
(and avoid double deletion) of the ACLs it uses. Since ACLs are remembered (in
Config.aclList) using a singly-linked list, it is difficult to change their
deletion order. Instead, we change their listing order from FIFO to LIFO.
Amos Jeffries [Sun, 1 Dec 2013 07:24:58 +0000 (00:24 -0700)]
Portability: std::string:npos is not always appropriate for String::npos
On some systems such as Windows MinGW the signed/unsigned property of
std::string::npos does not match up with the Squdi String class size_type
signed/unsigned property. Resulting in compiler signed vs unsigned
mismatch errors when compiling.
Nathan Hoad [Fri, 29 Nov 2013 23:26:03 +0000 (16:26 -0700)]
Bug 3972: Segfault when getting the deny info page ID after a reconfigure
Older ACL code was using a stale AclMatchedName value. More recent code resets
the AclMatchedName global to NULL to avoid leaking the stale value, but that
may crash strcmp() in aclGetDenyInfoPage(). Long-term, the global should be
removed, of course.
Alex Rousskov [Fri, 29 Nov 2013 19:47:54 +0000 (12:47 -0700)]
Support libecap v1.0, allowing asynchronous adapters and eCAP version checks.
After these changes, Squid can support eCAP adapters built with libecap v1.0,
but stops supporting adapters built with earlier libecap versions (due to API
changes). The new libecap version allows Squid to better check the version of
the eCAP adapter being loaded as well as the version of the eCAP library being
used. This should help with migration to libecap v1.0.
Expose [running] main event loop as a global so that modules can add engines.
Amos Jeffries [Fri, 29 Nov 2013 04:41:07 +0000 (21:41 -0700)]
Receive annotations from external ACL helpers
This completes the annotation support for common helper interfaces by
making custom key=value pairs sent by external ACL helpers in to
NotePair objects and attaching to the active request.
The other side of this - sending values to the helper is deferred until
the helper format can be converted to logformat codes.
Amos Jeffries [Wed, 27 Nov 2013 00:37:24 +0000 (13:37 +1300)]
Cleanup: convert helper allocation from cbdataAlloc to new
The helper_server and helper_stateful_server classes are already
CBDATA_CLASS2 defined classes with new/delete operators and the free
operation already handled by delete(). However teh allocation was not
using new() and thus not calling the class constructors properly.
Alex Rousskov [Mon, 18 Nov 2013 15:55:05 +0000 (08:55 -0700)]
Re-compute Range response content offset after an FTP response was adapted.
Moved HTTP-only offset calculation code to the general ServerStateData code so
that [converted-to-HTTP] FTP responses can use it. FTP code computes the range
offset on its own earlier, but we must REcalculate in case the response is
adapted into a response with a different (or no) content range.
aclocal on Windows cannot handle line-wrapping \ characters and leaves
them present in the output m4 code. in side AC_CONFIG_FILES macro this
results in a mystery file '\\r' being 'registered'.
The list of Makefiles does not even need ths line-wrap indicator as
demonstrated by the tools/Makefile missing its one for some time without
problems.