* Squid Project lisence without Harvest addendum
* Rosetta Project requires Berkley open distribution lisence
* Squid Translators adopting Berkley open distribution lisence
for compatibility with Squid code lisence and Rosetta lisence
Alex Rousskov [Thu, 16 Jul 2009 18:40:00 +0000 (12:40 -0600)]
Bug fix: Server transaction stuck in RESPMOD ACL check if no services matched.
AccessCheck job code assumed it was running asynchronously. However, the
job was started synchronously. If the access check result was available
immediately, without any async ACL checks, then AccessCheck would call
the callback immediately, before returning from AccessCheck::Start. The
server-side code would then get confused because it uses the
AccessCheck::Start return value to change its state, and that value was
stale when AccessCheck::Start returned _after_ calling the callback.
We now start AccessCheck job properly, via AsyncStart. All checks are now
performed in a detached async state.
TODO: There are other jobs that are not started asynchronously. Fix.
Alex Rousskov [Mon, 13 Jul 2009 01:20:26 +0000 (19:20 -0600)]
Support adaptation sets and chains, including dynamic ICAP chains:
- Support adaptation service sets and chains
(adaptation_service_set and adaptation_service_chain)
- Dynamically form chains based on ICAP X-Next-Services header
(icap_service routing=on)
- Support cross-transactional ICAP header exchange
(adaptation_masterx_shared_names)
An adaptation service set contains similar, interchangeable services. No more
than one service is successfully applied. If one service is down or fails,
Squid can use another service. Think "hot standby" or "spare" ICAP servers.
Sets may seem similar to the existing "service bypass" feature, but they allow
the failed adaptation to be retried and succeed if a replacement service is
available. The services in a set may be all optional or all essential,
depending on whether ignoring the entire set is acceptable. The mixture of
optional and essential services in a set is supported, but yields results that
may be difficult for a human to anticipate or interpret. Squid warns when it
detects such a mixture.
When performing adaptations with a set, failures at a service (optional or
essential, does not matter) are retried with a different service if possible.
If there are no more replacement services left to try, the failure is treated
depending on whether the last service tried was optional or essential: Squid
either tries to ignore the failure and proceed or terminates the master
transaction.
An adaptation chain is a list of different services applied one after another,
forming an adaptation pipeline. Services in a chain may be optional or
essential. When performing adaptations, failures at an optional service are
ignored as if the service did not exist in the chain.
Request satisfaction terminates the adaptation chain.
When forming a set or chain for a given transaction, optional down services
are ignored as if they did not exist.
ICAP and eCAP services can be mixed and matched in an adaptation set or chain.
Merged from 3p1-plus branch at r9513.
* Implementation notes
The notes below focus on _changes_. Adaptation terminology and current layers
are now being documented in src/adaptation/notes.dox
Service sets and chains are implemented as ServiceGroup class kids. They are
very similar in most code aspects. The primary external difference is that
ServiceSet can "replace" a service and ServiceChain can find the "next"
service. The internal search code is implemented in ServiceGroup parent and
is parametrized by the kids.
Before the adaptation starts, Squid calculates the adaptation "plan", which is
just an iterator into the ServiceGroup. The client- and server-side adaptation
initiators used to deal with Service pointers. They now deal with ServiceGroup
pointers. The only interesting difference is that a ServiceGroup does not have
a notion of being optional or essential. Thus, if adaptation start fails, we
do not know whether the failure can be bypassed. Fortunately, starting an
adaptation does not require anything that depends on the adaptation services,
so we now simply assert that the start succeeds.
If the entire adaptation fails, the callers are notified as before. They are
told whether they can ignore the failure as before. No changes there.
A new Adaptation::Iterator class has been added to execute the adaptation
plan. That class is responsible for iterating the services in a service group
until the plan is exhausted or cannot progress due to a final failure.
Dynamically form adaptation chains based on the ICAP X-Next-Services header.
If an ICAP service with the routing=1 option in squid.conf returns an ICAP
X-Next-Services response header during a successful REQMOD or RESPMOD
transaction, Squid abandons the original adaptation plan and forms a new
adaptation chain consisting of services identified in the X-Next-Services
header value (using a comma-separated list of adaptation service names from
squid.conf). The dynamically created chain is destroyed once the new plan is
completed or replaced.
This feature is useful when a custom adaptation service knows which other
services are applicable to the message being adapted.
Limit adaptation iterations to adaptation_service_iteration_limit to protect
Squid from infinite adaptation loops caused by ICAP services constantly
including themselves in the dynamic adaptation chain they request. When the
limit is exceeded, the master transaction fails. The default limit of 16
should be large enough to not require an explicit configuration in most
environments yet may be small enough to limit side-effects of loops.
TODO: Add metadata support to eCAP API and honor X-Next-Services there as
well. Currently, only ICAP services can form dynamic chains but the formed
chains may contain eCAP services.
Other improvements:
Polished adaptation service configuration in squid.conf. Old format with an
anonymous bypass option is deprecated but still supported. Quit with a fatal
message if an adaptation service is misconfigured (debugging level-0 messages
do not seem to work at that stage, but that is probably another, general bug).
Polished HttpRequest::adaptHistory() interface so that the code that knows the
history is needed can force history creation without complex
configuration-time preparations and state. Currently, all adaptation history
users but the logging-related ones know runtime whether the history must be
created (e.g., when a certain ICAP header is received).
Fixed "canonical" Request URL maintenance when ICAP clones requests.
TODO: The urlCanonical() must become HttpRequest::canonical(), hiding the
often out-of-sync canonical data member.
Fixed ICAP request parsing (for ICAP logging). We used to parse Request-Line
as if it were the first header. TODO: optimize by parsing only when needed.
Fixed AccessCheck case where a service group disappears during a nb ACL check.
Replaced "done" member with an existing AsyncJob mustStop mechanism. Removed
extra async call as unneeded because ACL callbacks are already async.
Author: Henrik Nordstrom <henrik@henriknordstrom.net>
http_port allow-direct option to allow direct forwarding in accelerator mode
normally direct forwarding is disabled in accelerator mode unless overridden
by always_direct, to avoid unintentional security loops. But there is setups
where it makes sense to not have this restriction as this has effects on
peer selection as well.
Alex Rousskov [Sat, 11 Jul 2009 05:39:44 +0000 (23:39 -0600)]
Limit X-Forwarded-For growth.
X-Forwarded-For growth leads to String size limit assertions and probably
other problems.
We now replace huge XFF values with a string "error", warn the admin the
first 100 times, and hope that something will stop the loop (if it is a
loop). TODO: we should probably deny requests with huge XFF.
To make growth-associated problems visible during forwarding loops, the
loop breaking code must be disabled (no Via) or not applicable (direct
forwarding) and request_header_max_size has to be raised or disabled.
The X-Forwarded-For header value may also grow too large for reasons
unrelated to forwarding loops.
This change also prevents most cases of pointless computation of the
original X-Forwarded-For value list. That computation can be quite
expensive.
Shuffle cachemgr.cgi.8 into tools/ with cachemgr.cgi code
To perform this shuffle the SUBSTITUTE make code pushed into a sub-include
at doc/manuals/Substitute.am for shared use.
Also clean up the manual titles to reflect the binary they apply to
rather than the generic Squid project release name.
The use of generic @SYSCONFDIR@ and specific filenames allows the
translation strings to be more specific, readable for translators and
reduces Makefile complexity at once. squid.conf etc are widely mentioned
and used enough that allowing for alternative names at this point seems
meaningless.
Attempts to translate, and installs the error pages content if present.
Both srcdir and builddir are scanned used since the file location varies
depending on whether a source bundle or VCS is used to install.
Alex Rousskov [Thu, 2 Jul 2009 16:36:36 +0000 (10:36 -0600)]
Truncate too-long HTTP response bodies to match their Content-Length header.
Sometimes a broken server sends more than Content-Length bytes in the
response. For example, a 302 redirect message with "Content-Length: 0" header
may include an HTML body. Squid used to send "everything" it read to the
client, even if it read more than the Content-Length bytes. That may have
helped in some cases, but we should be more conservative when dealing
with broken servers to combat message smuggling attacks and other bad
side-effects for clients.
We now do not forward more than the advertised content length and declare the
connection with a broken server non-persistent.
Chunked responses (that HTTP/1.0 Squid should not receive and that must not
have a Content-Length header) are not truncated because RFC 2616 says we
MUST ignore their Content-Length header.
TODO: Do not cache the truncated entry and purge the cached version, if any.
Alex Rousskov [Thu, 2 Jul 2009 15:24:24 +0000 (09:24 -0600)]
Break forwarding loops for "transparent" or "intercept" http_ports.
Squid detected forwarding loops in most configurations, but broke
them (using a customizable HTTP_FORBIDDEN response) only when working as
an accelerator. Squid now breaks loops when working as a transparent
proxy as well.
A persistent loop is going to be broken anyway, when the Via and
X-Forwarded-For headers exceed header size limit, but that wastes a lot of
resources and may also crash misconfigured Squids.
TODO: Consider breaking all loops, regardless of the http_port options.
TODO: Consider adding a specific and/or configurable error page for this case
instead of using hard-coded ACCESS_DENIED.
Amos Jeffries [Sun, 28 Jun 2009 08:28:20 +0000 (20:28 +1200)]
Bug 422: RFC 2616 Date header requirements
The Date: header appears to already be implemented on all generated
pages and ICAP processed pages.
This tests and enforces Date: on all other outgoing replies as required.
I'm not certain this is the right place, it appears to be post-caching.
The RFC indicates the Date: should be enforced pre-caching. But was
unable to find a place of input cloning/processing after initial parse.
The storeEntry timestamp is used to estimate correct receiving date.
Amos Jeffries [Sun, 28 Jun 2009 08:11:27 +0000 (20:11 +1200)]
Simpler test for PURGE method
Uses a tri-state setting on enable_purge and acl parsing to
detect PURGE method addition/removal instead of a complicated ACL
creation test post-configure.
This removes the annoying false errors about temp ACL and some minor
speed up in all actions that parse squid.conf.
Alex Rousskov [Sat, 27 Jun 2009 01:14:58 +0000 (19:14 -0600)]
Author: Philip Allison <philip.allison@smoothwall.net>
Bug 2614 fix: Potential loss of adapted body data from eCAP adapters
It was possible for Squid to stop reading buffered adapted body data before it
has all been sent to the browser.
Squid treated a call to noteAbContentDone by an adapter as a signal to stop
consuming and sending adapted body data. The correct behaviour is to use
noteAbContentDone to record the fact that the adapter has stopped producing
new adapted body data, but continue to consume and send data until all
buffered ab content is consumed and sent (i.e., abContent returns an empty
Area).
Amos Jeffries [Fri, 26 Jun 2009 13:32:27 +0000 (01:32 +1200)]
Language Updates: Add aliases from live traffic info
Taking a scan of the last 98 days traffic and locating the country-code
Accept-Language headers used in that traffic to refer to the existing
languages gives a subset we can alias to further improve the coverage.
Amos Jeffries [Fri, 26 Jun 2009 07:02:45 +0000 (19:02 +1200)]
Bug 2674: Remove limit on HTTP headers read.
Headers may be accumulated over more than one read. It does not make
sense to limit the internal copy of the accumulated read buffer to 64KB.
Reverts the internal read buffer to MemBuf defaults. This may cause
issues where headers are of unbounded size. But those are expected to be
caught by the header parser.
Amos Jeffries [Sun, 21 Jun 2009 06:32:16 +0000 (18:32 +1200)]
Fix alias linker dist/install
make requires ';' after a SHELL command apparently.
Make alias-link.sh handle case where the DESTDIR is non-existent.
This occurs on some distro packaging systems (ie using langpack as a
separate package may not install errors).
Amos Jeffries [Sat, 20 Jun 2009 06:50:37 +0000 (18:50 +1200)]
Correct Licensing Credits
Several of the licenses mentioned in the CREDITS file are not relevant
to Squid-3.1 code any more. Several license disclaimers were found to be
missing.
Thanks to the Debian Project for identifying these incorrect entries.
Amos Jeffries [Wed, 17 Jun 2009 14:20:11 +0000 (02:20 +1200)]
Language alias linker/installer/upgrade scripts
alias-link.sh
This is a script set designed to be called via make/Makefile and setup
language codes for those languages which it would be impractical to
bundle duplicate translated files for.
Relies on local environment tools to be detected by automake.
make install
- now also calls generation of aliases after existing install.
Provided in file aliases.
make upgrade
- cleans out legacy files from pre-3.1 and replaces with symlinks
to the new upgraded language codes.
Provided in file alias-upgrade.
NP: this is a destructive process and must be manually run.
Bundle aliasing scripts and Makefile to use them with the langpack.
Amos Jeffries [Sun, 14 Jun 2009 06:18:11 +0000 (18:18 +1200)]
Update .pot and rename errpages.pot
The old name dictionary.pot was picked semi-randomly based on my old
experience in other translation arenas. It does not fit with the l18n
requirements of Squid very well. This renaming adjusts to make things
a bit clearer.
Amos Jeffries [Fri, 12 Jun 2009 08:46:29 +0000 (20:46 +1200)]
Bug 2395: FTP auth errors not displayed
Round 2 for this bug. Now we handle missing auth as an expected result
rather than a failure. FTP operations are now well tested and this patch
does not affect code shared with other components.
Side-effect is that browser authentication popups now appear when the
FTP server needs authentication. This has been a long missed event.
The root cause of the issue is not found so other subsequent errors in
FTP sub-protocol still silently lost due to the same issue.
projects / thirdparty / squid.git / log
