Amos Jeffries [Sat, 21 May 2011 01:13:42 +0000 (13:13 +1200)]
URL re-writer handling bug fixes
This patch includes two bug fixes in URL handling which were uncovered
during testing of the URL logging update:
* URL re-write handling was not correctly creating its adapted request
copy. The code here is much reduced by using the clone() method. Still
not completely satisfactory (marked with XXX) since on invalid URL
there is a wasted cycles cloning and deleting almost immediately.
Future cleanups moving the URL parts outside HttpRequest will fix that.
* URL parsing needs to set the canonical field to unset whenever the URI
is re-parsed into a request. This field is an optimization for later
display speed-ups. This has been causing incorrect canonical URL to be
used following re-write. When the cloning above was corrected it caused
asserts in the server-side.
* To prevent memory leaks the urnParse() function internal to URL parsing
is adjusted to accept and update an existing request in identical API
semantics to urlParse() instead of always generating a new one.
Currently, SSL error detail in Squid-generated error pages (%D) contains
both the error name and the explanation text. Some folks using this feature
want to render the two pieces of information differently because the error
name is not something most end-users should read or focus on.
This patch adds the "%x" error page formating code which prints the error name,
and removes the error name (%err_name) from SSL error detail messages.
This patch implements the phase 1 of the ICAP Max-Connections feature as it is
described in squid wiki:
http://wiki.squid-cache.org/Features/ServiceOverload
The behaviour of the patch can be configured using on_overload and max_conn
options of the icap_service configuration parameter. Squid can be configured
to do one of the following:
- Block: send and HTTP error response to the subscriber
- Bypass: ignore the "over-connected" ICAP service
- Wait: wait (in a FIFO queue) for an ICAP connection slot
- Force: proceed, ignoring the Max-Connections limit
Squid warns the first time the service become overloaded
For more information please visit the feature wiki page given above.
Technical informations:
The patch starts count a connections to the ICAP server as active when the
ModXact class receives an FD even if the fd is not really connected to the
server yet, and decrease the active connections to the server when the ModXact
object releases its fd connection.
If the Max-Connection limit is reached squid puts the request to a waiters list.
When one or more connections released squid schedules one or more waiters for
execution and remove them from waiters list.
To handle cases where a waiter gone/canceled before its execution the custom
dialer ConnWaiterDialer used.
The Options connections counted as active connections but are not limited by
the Max-Connections limit. An Option request will be executed even if the
maximum connections number is reached.
Tilmann Bubeck [Mon, 9 May 2011 12:42:59 +0000 (00:42 +1200)]
Add ext_time_quota_acl helper
Allows an administrator to define time budgets for the users of squid
to limit the time using squid.
This is useful for corporate lunch time allocations, wifi portal
pay-per-minute installations or for parental control of children. The
administrator can define a time budget (e.g. 1 hour per day) which is
enforced through this helper.
Andrew Beverley [Sun, 8 May 2011 23:21:44 +0000 (11:21 +1200)]
QoS: require libcap before enabling netfilter MARK support
As it is not possible to get or set a netfilter mark without libcap, this
patch will disable netfilter marking at compilation time if libcap is not
available (in a similar way to Linux transparent proxying).
Amos Jeffries [Sun, 8 May 2011 13:53:10 +0000 (01:53 +1200)]
Cleanup: sync NTLM and Negotiate UserRequest code
Minor tweaks to reduce diff between the files. No logic changes.
Renames the addHeader() to addAuthentiocationInfoHeader(),
Renames the addTrailer() to addAuthentiocationInfoTrailer() and
document that they add additional *-Info header to the HTTP reply.
Amos Jeffries [Sun, 8 May 2011 06:11:18 +0000 (18:11 +1200)]
Cleanup: Improve Connection Pinning management
Since 1xx handing went in HttpRequest has had two links to the one
ConnStateData managing its client connection.
* Rename the 1xx link to clientConnectionManager (since it is not
actually the connection, but the manager object controlling the FD
usage and stats.
* Convert the pinning code to using the permanent clientConnectionManager
link instead of a temporary pinned_connection link.
This moves all connection pinning state fully into the ConnStateData
manager objects scope.
Side changes that appear to be buggy code previously:
* do not alter pinning state at the point where the pinned connection is
about to start being used. Changes are only relevant at the point of
pinning or unpinning.
* unpin operation now closes the Server FD if still open. Previously
there was the possibility that some code paths would leave server FD
open and pconn it. (especially since the above mentioned state
alteration cleared the "pinned" flag).
Amos Jeffries [Fri, 6 May 2011 16:16:45 +0000 (04:16 +1200)]
Implicit Dependency removal for gcc-4.6.1
GCC 4.6.1 is stricter than 4.6.0. It does not by default include implicit
dependencies. This adds several unit tests .cc files which were implicitly
linked before.
Also adds tests/stub_DiskIOModule.cc to short-circuit the DiskIOModule API
Bug #3214: "helperHandleRead: unexpected read from ssl_crtd" errors.
Squid would read the beginning of a crtd response split across multiple
read operations and treat it as a complete response, causing various
certificate-related errors.
This patch:
- allow the use of other than the '\n' character as the end of message mark
for helper responses.
- Use the '\1' char as end-of-message char for crtd helper. This char looks
safe because the crtd messages are clear text only messages.
Amos Jeffries [Wed, 4 May 2011 03:05:09 +0000 (15:05 +1200)]
Compile fixes for binutils-gold and gcc-4.6 support
These two tools are much stricter about dependency linkages. We have already
had to drop testAuth due to major dependency loops they dislike.
This makes the remainder of the dependency changes needed.
Also adds:
- tests/STUB.h with macros for simpler stub file creation
- stub_libmgr.cc for unit-test stub replacment of mgr/libmgr.la library.
many API functions commented out, but sufficient for the current needs.
Amos Jeffries [Mon, 2 May 2011 13:04:21 +0000 (01:04 +1200)]
Drop testAuth unit-tests
Preparing to move the tests into src/auth.
These old tests construction style also require quite a lot of dependencies
which include several loops causing problems in modern strict linkers.
Opted to remove now and stabilize trunk without it before re-adding simpler
auth unit tests.
Amos Jeffries [Mon, 2 May 2011 01:14:30 +0000 (19:14 -0600)]
Cleanup: base64 coder de-duplication and upgrade
Markus Moeller has re-implemented several of the coder functions for use
by Kerberos helpers.
This patch seeks to de-duplicate them and combine the resulting code
back into the libmiscencoding.la "base64.h" implementation.
Changes include:
* old function API renamed to old_*() and existing code update to use
the names. Some code has been updated to use the new API.
* new estimator base64_encode_len()/base64_decode_len() functions added
to provide details of much much buffer space the output will require.
* new API encoder and decoder functions added which accept caller
provided buffers and encode/decode an arbitrary string into them.
* also fixes a bug where if the input text or output buffer was too
short the coder functions would crop a few bytes off the end of the result.
Noticable in Kerberos where token lengths are not fixed length.
Some optimizations have been added by myself over and above Markus changes:
* optimized to short-circuit on several more variations of empty input
and nil result buffer.
* sub-loop optimizations added to reduce the number of if() calls made
by the new code.
* split encoder into terminated (C-str) and non-terminated variants.
James Bowe [Sun, 1 May 2011 12:10:37 +0000 (00:10 +1200)]
Add external_acl_type %EXT_LOG and %EXT_TAG format options.
%EXT_LOG and %EXT_TAG are filled with the log= and tag= fields
returned by previous external ACLs.
-for a string that never changes after it is set, tag= is suitable.
-for a string that may need updating or overwriting by a later
external_acl, log= is suitable.
Under both circumstances it is conceivable that later external_acls
may need access to the tag= or log= values after they have been set
(e.g. for external_acl debugging, merging log messages, etc).
The bug appeared after commit with revno:11364 which fixes the Bug #3192.
In the case of SSL-bumped connections the ConnStateData::flags.readMore flag
must be reset (set to true) when we are switching to HTTPs,
because we have to read the new unencrypted HTTP request.
This patch reset this flag in ConnStateData::switchToHttps method.
In the common default case there are no reply body limits configured.
There is no need to construct ACL checklists for testing. This saves
one allocation and several locking/unlocking cycles per request.
Use getMyPort() to insert the forward-proxy listening port into error
pages and deny_info redirect URLs. This fixes the current port
hard-coding assumption in ERR_AGENT_CONFIGURE.
The %b option is added for this purpose as a temporary measure until the
codes are merged with the more flexible log formatting set.
This currently depends on squid.conf having a particular http_port
ordering with the forward-proxy port listed first.
SourceLayout: Add enum Direction for AuthUserRequests state logics
The state of credentials lookup and handling is recorded by
authenticateDirection / AuthUserRequest::direction() and its per-scheme
helper methods AuthUserRequest::module_direction().
This formalizes and coordinates the state being returned by using a
shared enum.
The states can generally be considerd as:
- LOOKUP with a helper still needs to validate the credentials
- CHALLENGE if the helepr needs more info from the client
- VALID if everything is fine and the credentials are known Good/Bad
- ERROR if there is any problem with the state or credentials
TODO:
This combination has highlighted a few strange things in the NTLM and
Negotiate states. Where known but Failed credentials are marked as ERROR.
This needs closer investigation why it is not a CHALLENGE in all auth
schemes.
Also there is a little obfuscation of the cases around the generalized
fixHeader() calls. This will be handled in a followup patch.
These macros are required for ./configure to run on an OS such as MingW.
The macro to detect pkg-config being present is usualy only bundled with
pkg-config. When there is no pkg-config installed ./configure will fail.
This allows our configure to detect the absence and mark some components
as unavailable or unusable.
Fixes NTLM and Negotiate auth assertion "RefCountCount() == 2"
It turns out the replay cache and invalid RefCount cases this was added to
protect againt are not present anyway. After some minor cleanup to remove
double-calls in Negotiate things appear to run nicely.
NOTE:
There is still a risk that these problem cases may in future occur, but
meanwhile we need NTLM and Negotiate to be usable and efficient.
The bugs resulting from those can be dealt with if/when they do occur.
Markus Moeller [Fri, 15 Apr 2011 11:51:15 +0000 (05:51 -0600)]
negotiate_wrapper_auth: version 1.0.1
A helper to perform Negotaite authentication in both its Negotiate/NTLM
and Negotiate/Kerberos forms.
Makes use of additional Squid helpers after unwrapping the header token.
Also, shuffle the resulting classes into their own compilation units.
No Logic changes.
Have omitted shuffling or altering two Auth::Basic::User methods handling
the validation short-circuit since these shodul not be part of that class.
Followup patch will move them appropriately.
Uses hard-coded string "cachemgr.cgi/" instead of progname to avoid
complications from alternative names and when running under a browser.
May be elided in transit however teh VERSION sent here will help the
queried proxy respond appropriate to the CGI capabilities as we extend
the types and content of reports coming back from the future releases.
Also, no code shuffling which should normally have been done with namespace.
Config children are currently too intwined with UserRequest children and
helper management. Logic changes are required before that can be done.
ConnStateData::flags.readMoreRequests, do_next_read variables, and
ClientSocketContext::mayUseConnection() methods were used (or unused!)
incorrectly or inconsistently.
This change removes all do_next_read variables to simplify the state. Instead,
the renamed ConnStateData::flags.readMore indicates whether client_side.cc
should call comm_read. The mayUseConnection() methods are now used to indicate
whether the next client-sent byte (buffered or read) should be reserved for
the current request rather than being interpreted as the beginning of the next
request.
Portability Fix: getrlimit() / setrlimit() incompatible type 'struct rlimit'
On Linux (at least) with large file support but not full 64-bit environment.
The getrlimt / setrlimit are #define'd to getrlimite64 / setrlimit64
BUT, the struct rlimit internal fields are updated to 64-bit types individually
instead of a matching #define to struct rlimit64 as a whole.
One can only assume that GCC is casting to void* or some such major voodoo
which hides this type collision.
ICC: support 64-bit environments dirent definitions
struct dirent is not consistently defined for 32-bit and 64-bit enabled
environments. Provide a dirent_t type defined appropriate to the environment
for use instead.
This npending test bug was preventing any poll() errors from being
noticed and displayed. Possibly leading to some of the weird hanging
reports we have been unable to replicate.
Alex Rousskov [Wed, 6 Apr 2011 16:25:36 +0000 (10:25 -0600)]
Fixed chunked request forwarding in ICAP REQMOD presence.
ICAP prohibits forwarding of hop-by-hop headers in HTTP headers. If the virgin
request has a "Transfer-Encoding: chunked" header, the ICAP server will not
receive it. Thus, when the ICAP server responds with a 200 OK and what it
thinks is a copy of the HTTP request, the adapted request will be missing the
Transfer-Encoding header.
One the server side, Squid used to test whether the request had a
Transfer-Encoding header to determine whether request chunking is needed when
talking to the next HTTP hop. That test would fail in ICAP REQMOD presence.
This change implements a more direct/robust check: if we do not know the
request content length, we chunk the request.
We also no longer forward the Content-Length header if we are chunking. It
should not really be there in most cases, but an explicit check is safer and
may also prevent request smuggling attacks via Connection: Content-Length
tricks.
Portability: Provide stdio wrappers for 64-bit in cstdio C++ builds
stdio.h in that case on provides fgetpos64, fopen64 if
__USE_FILE_OFFSET64 is defined. It then checks whether a gcc-specific
__REDIRECT macro is available (defined in sys/cdefs.h, depending on
__GNUC__ begin available).
If it is not available, it does a preprocessor #define.
Which <cstdio> undefines, with this comment:
"// Get rid of those macros defined in <stdio.h> in lieu of real functions.".
When it does a namespace redirection ("namespace std { using ::fgetpos; }")
it goes blam, as fgetpos64 is available, while fgetpos is not.
To fix it, we need to supply global functions matching those
signatures (not macros).
Enable string mempools to work correctly during initialization phase
Makes string mempools work before Mem::Init() was called, as may happen
during global variable initialization or early main.cc processing. If
needed, strings allocated before the Mem::Init() call are given an extra
buffer space to make sure the allocated buffer size will not match any
string pool size during deallocation.
Shortcomings: We now waste RAM on buffer increase for early allocated
strings unless they are already bigger than the maximum supported string
pool size. Statistics for early allocations are broken. Non-string
mempools still do not support early allocations.
Alex Rousskov [Tue, 5 Apr 2011 21:39:53 +0000 (15:39 -0600)]
Fixed %dt logging in the presence of REQMOD.
We use LogEntry::request to save a virgin request for future logging. However,
when that request is adapted and replaced, the adapted request has all the
stats while the saved virgin request lacks them. We have already copied error
details from the adapted to logged/virgin request. Now we copy the DNS wait
time (%dt) as well.
TODO: Move statistics to a stand-alone history object that adapted and
virgin requests can share. Longer term, we should separate HttpRequest
from Master Transaction so that we can store virgin request details without
implicitly storing not-yet-collected master transaction stats.
Display ERROR in cache.log for invalid configured paths
The validator that checks system paths for files and directories in the
configuration file sends error messages to stderr. It should send them to
cache.log for the admin to see easily.
Also, this makes the error display as FATAL ERROR when using -k parse to
indicate that it is fatal to the startup. Other management signals where
it is not necessarily fatal will only display as an ERROR.
projects / thirdparty / squid.git / log
