Amos Jeffries [Tue, 16 Nov 2010 05:43:47 +0000 (18:43 +1300)]
Reduce debug level on bodypipe re-write change
- the original reason for adding is unknown
- it is an annoyance for some
- there have been no big problems tracked down to this bodypipe change
over the last few years. It appears relatively harmless.
Amos Jeffries [Wed, 10 Nov 2010 09:50:38 +0000 (22:50 +1300)]
Policy: detect config.h and squid.h include problems
We currently have a policy that config.h MUST be included first so as to
pull in the portability definitions early. It MAY be included via the
legacy squid.h at present.
This alteration to the source maintenance script validates that each .c
and .cc file in the sources includes config.h or squid.h first in its
include order. Also that each .h and .cci do not include config.h which
is a double include with enforced .c/.cc requirement.
* an ERROR: line is produces for each violating file
* as yet the maintenance run is not blocked so as to catch as many
errors as possible in one run
* detection only, as yet no code alterations are performed by this script
* FORMAT: informative lines are silenced to make ERROR: more visible
Alex Rousskov [Sun, 7 Nov 2010 03:07:29 +0000 (21:07 -0600)]
Bug 3091 fix: Bypassed ICAP errors are not counted as service failures.
Notify ICAP service about the failure even if we can bypass it. Otherwise,
a failing service may continue to stay "up", preventing Squid from using a
healthy backup alternative in a service_set (or bypassing ICAP completeley).
Fixed DNS query leaks and increased defense against DNS cache poisoning.
We were leaking (i.e. forgetting about) DNS queries under several conditions.
The most realistic leak case would go like this:
- We send UDP query1.
No response.
- We send UDP query2.
The response for query1 comes, with TC bit.
- We try to connect over TCP, sending TCP query3.
The response for query2 comes, with TC bit, matching TCP query3 ID.
Since we are waiting a response over TCP, we drop the UDP response,
and delete the query from the queue. We leak.
This change avoids forgetting the query under the above scenario.
Moreover, the above steps are hiding another problem: we are accepting responses
to timed out queries, making DNS cache poisoning easier. This change avoids
that by using unique query ID for each sent query. We have also added an
instance ID so that we still can track/identify a single "transaction" from
Squid point of view, even when that transaction involves many DNS query
messages.
When we forget about a DNS query, the caller may get stuck, holding a cbdata
lock. This is typical for ACLs that require domain name resolution, for example.
On a busy server with a long ACL list, the lock counter keeps growing due to
forgotten requests and may overflow, causing a "c->locks < 65535" assertion.
This change fixes the assertion unless there are more DNS leaks or different
lock leaks present.
and response_is_fresh is always false if freshness_lifetime is zero.
The check code was introduced in r5998 with a "Import of fix-ranges
branch" message. The code was commented out at the time of that
commit, for reasons unknown.
Test case:
test_case/rfc2616/noSrv-hit-stale-max-age-req
Amos Jeffries [Mon, 1 Nov 2010 05:44:28 +0000 (23:44 -0600)]
Bug 3038: Detatch libmisc from libcompat
* Migrates many of the remaining libmisc portability wrappers into
libcompat.
* Splits libmisc into:
libprofiler - Squid internal profiler (developer-only)
libmiscencoding - Various binary encoding / crypto algorithms
libmisccontainers - Various data container algorithms
* Makes all binaries which need to link the libmisc* pieces directly instead
of via $(COMPAT_LIB) which now only links the libcompat and internal
profiler due to profiling being used on some libcompat functions.
* Adds a stub_debug for binaries needing the Debug.h API without squid
timers and globals.
Some effort has been made to identify binaries whose dependencies can be
reduced. More of this dependency removal can be done in future.
Amos Jeffries [Mon, 1 Nov 2010 00:21:57 +0000 (18:21 -0600)]
Harden quoted-string parser to RFC requirements
Fix RFC 2616 section 2.2 quote-string handling.
* Restrict the parser to the known length of the value string to prevent
buffer over-reads from specially crafted inputs.
* Drop quoted-string values containing CTL octets.
Alex Rousskov [Thu, 28 Oct 2010 18:52:59 +0000 (12:52 -0600)]
SMP Cache Manager, Phase2 implementation.
Cache Manager actions are forwarded to Coordinator. Coordinator iterates over
Kids, aggregating their stats if possible and/or allowing each kid to dump
non-aggregatable output directly into response if needed. Non-aggregated
output is wrapped in "by kidN { ... } by kidN" markup to ease auto-processing.
Regressions and small output formatting changes are probably unavoidable
because stats are aggregated and passed around as doubles instead of integers
(no more overflows though!) and because many stats collection and formatting
lines had to be touched. These are steps in the right direction though, IMO.
Old code both computed and dumped stats to Store at the same time. To avoid
computing code duplication, we now collect stats in primitive Stats objects
and then either dump those to Store or send them to Coordinator for
aggregation and, eventual Store dump. What stats to collect, when to
aggregate, and when to dump is decided by action-specific Mgr::Action classes.
The Cache Manager menu now consists of ActionProfile objects. ActionProfile
maintains hard-coded information about specific actions. It uses ActionCreator
member to create Action objects when a cache manager request is received.
Added Mgr::ActionParams class to maintain action parameters, including HTTP
request details necessary for Store entry creation (in another strand) and
action-specific parameters (currently just credentials). In Phase3, this class
can be extended to supply more parameters such as kid IDs to which the action
should apply.
Added Mgr::Command that combines hard-coded ActionProfile details with
user-specified ActionParams. This simplifies many interfaces because we no
longer need to supply a long list of parameters, covering various parts of
action config.
Moved Cache Manager registration to Mgr::RegisterAction() globals to reduce
dependency on the CacheManager class, which is a singleton anyway, and which
is unused by most of the registration callers. On the other hand, without
this change, no legacy (function-based actions) code would have been changed!
Enhanced TypedMsgHdr class to simplify storing and loading non-POD classes.
The caller can now easily handle a non-POD class as a series of put/get calls,
one for each POD member. This was necessary to send Mgr::ActionParams to
Coordinator and back. Will probably be useful for sending other complex
structures as well.
Reconfigure, shutdown, and other "basic" actions have been moved to
src/mgr/BasicActions.cc. Mgr::RegisterBasics() registers them.
Most of the Cache Manager code is now in src/mgr/.
Many more polishing touches.
More polishing left for future projects: Move CacheManager to Mgr namespace
and src/mgr/ directory. Use SBuf instead of String for ActionParams and
TypedMsgHdr. Rename Ipc::TypedMsgHdr to Ipc::Msg, Ipc::SocketMsg, or similar
because it maintains more than just msghdr struct. More stats aggregation,
and Phase3 changes.
Alex Rousskov [Wed, 27 Oct 2010 23:29:55 +0000 (17:29 -0600)]
HTTP Compliance: Support If-Match and If-None-Match requests.
Add support for If-Match and If-None-Match headers as described in RFC 2616
(sections 14.24 and 14.26 in particular).
Moved IMS handling from clientReplyContext::cacheHit() to
clientReplyContext::processConditional() while preserving the original IMS
logic, except for the case when a request has both IMS and If-None-Match.
Co-Advisors test cases:
test_clause/rfc2616/ifMatch-mismatch-strong
test_clause/rfc2616/ifMatch-mismatch-weak
test_clause/rfc2616/ifNoneMatch-match-imsNone
and many more
Author: Alex Rousskov <rousskov@measurement-factory.com>
Client-side bandwidth limit (a.k.a., quota or delay pool) implementation.
In mobile environments, Squid may need to limit Squid-to-client bandwidth
available to individual users, identified by their IP addresses. The IP
address pool can be as large as a /10 IPv4 network (4 million unique IP
addresses) and even larger in IPv6 environments. On the other hand, the code
should support thousands of connections coming from a single IP (e.g., a child
proxy).
The implementation is based on storing bandwidth-related "bucket" information
in the existing "client database" hash (client_db.cc). The old code already
assigned each client IP a single ClientInfo object, which satisfies the
client-side IP-based bandwidth pooling requirements. The old hash size is
increased to support up to 32K concurrent clients if needed.
Client-side pools are configured similarly to server-side ones, but there is
only one pool class. See client_delay_pools,
client_delay_initial_bucket_level, client_delay_parameters, and
client_delay_access in squid.conf. The client_delay_access matches the client
with delay parameters. It does not pool clients from different IP addresses
together.
Special care is taken to provide fair distribution of bandwidth among clients
sharing the same bucket (i.e., clients coming from the same IP address).
Multiple same-IP clients competing for bandwidth are queued using FIFO
algorithm. If a bucket becomes empty, the first client among those sharing the
bucket is delayed by 1 second before it can attempt to receive more response
data from Squid. This delay may need to be lowered in high-bandwidth
environments.
This feature has been documented at
http://wiki.squid-cache.org/Features/ClientBandwidthLimit
Amos Jeffries [Sat, 23 Oct 2010 08:50:56 +0000 (02:50 -0600)]
Upgrade process for obsolete options
One problem we currently have with upgrades is leaving the parser able
to avoid its bungled/unknown option message for directives which have
been fully removed or massively syntax altered.
We are able to handle this for flags and option syntax easily but the
parser has been particularly dense and strict on the directives (first
word of each line).
This patch updates the cf_* and cfgman code to allow a special directive
type "obsolete" which causes these directives to be handled specially
without causing the directives to remain in the publicly visible
squid.conf documentation.
It allows DOC_START / DOC_END comments to be written in cf.data.pre
describing the upgrade actions that need to be taken. This text is
dumped to cache.log verbatim when the configuration option is sighted.
If "-k parse" is used the text is displayed at debug level 0, otherwise
displayed at debug level 1. One line indicating a generic "directive X
is obsolete" is always displayed at level 0 for backwards compatibility
with admin expectations of a high level "bungled" message.
After all this text display, parse_obsolete(char*) is called with the
directive name. This function exists in cache_cf.cc and can be coded to
selectivey do more complex handling of the directive. ie for upgrade
actions deeper than removal.
* cf.data.pre has entries added for all the 2.6-3.1 directives I could
find that were removed up to 3.HEAD
Amos Jeffries [Fri, 22 Oct 2010 13:23:09 +0000 (07:23 -0600)]
Bug 3084: IPv6 without Host: header in request causes connection to hang
accel and intercept mode URL re-generation used NtoA instead of ToHostname.
This results in the URL incorrectly wrapping the raw-IPv6 and problems
connecting to non-existent addresses in some cases.
Build failed in Hudson for opensolaris and debian sid
Build failed for opensolaris and debian sid with a linker message that
the TextException::FileNameHash(char const*) is undefined.
This patch trying to resolve the problem
Fixes for the "%err_code and %err_detail logformat codes" patch
The newly added file src/err_detail_type.h must listed in squid_SOURCES
variable in Makefile.am to be included in archives generated using "make dist"
For the same reason the files scripts/{calc-must-ids.pl,calc-must-ids.sh}
must listed in EXTRA_DIST variable.
Author: Alex Rousskov <rousskov@measurement-factory.com>, Christos Tsantilas <chtsanti@users.sourceforge.net>
Added %err_code and %err_detail logformat codes to record details about transaction failures
For example, when Squid responds with 500 Internal Server Error, it is often
useful to know what went wrong.
This patch :
- log a detail string instead of numbers for common errors
- log a label with the detail code number for generic errors (system errors,
or exceptions)
- adds more details about errors, especially those detected with exceptions:
We record a hash of the filename and the source code line number of the
first caught exception.
- adds two scripts which can help the developers to find the exact position
of the caught exception:
1) The calc-must-ids.pl take as argument one or more files and compute for
each Must expression in the given files its id;
2) The calc-must-ids.sh can be used to find the exact position of a Must
expression from its id. Example usage:
# ./scripts/calc-must-ids.sh 0xB79EF14
./src/adaptation/ecap/MessageRep.cc:356: 0xB79EF14 Must(false);
Amos Jeffries [Sun, 17 Oct 2010 03:26:02 +0000 (21:26 -0600)]
Bug 2785: DNS needs to set EDNS options advertising Squid capabilities
... allowing Squid to advertise a larger UDP reply size than 512 bytes.
Internally Squid has a buffer allocated on demand so there is no
practicable limit on individual packets. Network topology and external
software places stricter boundaries on what works and what does not.
Squid does not parse the additional section of replies so for now the
full auto-negotiation EDNS allows is not used. Instead a configuration
option is provided for admin to configure a desirable packet size in
bytes. EDNS defaults to "none" (disabled) until tested in a wider
environment.
Testing so far has brought to light problems with EDNS adverts on A and
IPv4-PTR queries. So support is limited to AAAA and IPv6-PTR queries only.
EDNS compliant resolvers have the option of caching the info between
requests for a short while so this will hopefully leak over to improve
IPv4 responses as well.
Amos Jeffries [Sat, 16 Oct 2010 23:19:46 +0000 (17:19 -0600)]
basic_msnt_auth helper and NTLM/SMBLIB/RFCNB library polish.
Samba smblib/rfcnb code:
* Import the latest copy which I could find a download link to.
This source is from 1997 so I suspect there is something even newer
we should be using. Time was tight is the only excuse for using
these sources. Our originals were from 1995 and 1996 depending on the
helper using it, with a mix of patches.
* These two directories are in ours sources as lib/smblib and lib/rfcnb.
Each has its own convenience library. Kept separate with original
filenames to simplify future upgrades or removal.
* Samba sources have been diffed and compared function by function
against the copies previously in our sources. Functionality extensions
we use have been grafted back on top of the new(er) Samba sources.
- this was mostly around passing extra Unicode, DC hints and pre-crypted
passwords to the login checks.
- some files from libntlmauth have yet to be compared in fine detail,
that will be completed today before merge.
- some basic API function and struct definitions had to be moved to the
API headers to prevent needing to include the *-priv.h private
definitions externally to the library.
* the Samba API headers have been wrapped with #ifndef safety wrappers
* compile errors and include changes required to compile have been
made (code stays C)
* duplicate code in helpers/basic_auth/MSNT/* and libntlmauth/* is
removed.
* abuse of the smblib-priv.h and rfcnb-priv.h headers and all local
re-definitions has been erased from our code. Replaced by includes
of the library API headers: rfcnb/rfcnb.h smblib/smblib.h
libntlmauth:
* smblib/rfcnb bits erased
* moved to lib/ntlmauth in its much reduced form
* built as a convenience library instead of full library
Amos Jeffries [Fri, 15 Oct 2010 10:24:57 +0000 (04:24 -0600)]
Author: Alex Rousskov <rousskov@measurement-factory.com>
Bug 3002 pt2: store initialization (-z) does not work with SMP configs.
In SMP mode, when -z command line option is specified, start kids and allow
them to create their cache_dirs instead of trying to create macro-dependent
cache_dirs in the Master process.
In SMP mode, quit if cache_dir option is found before the workers option.
projects / thirdparty / squid.git / log
