rousskov [Wed, 13 Feb 2008 06:17:50 +0000 (06:17 +0000)]
Merging async-call branch changes to HEAD:
ConnStateData used a RefCount pointer (the openReference) to keep itself alive
which is not compatible with the cbdata used by CommCalls. This change makes
the ConnStateData class non RefCountable and uses cbdata instead.
rousskov [Wed, 13 Feb 2008 06:12:45 +0000 (06:12 +0000)]
Merging async-call branch changes to HEAD:
Async-call work replaces event-based asynchronous calls with
stand-alone implementation. The common async call API allows Squid
core do call, debug, and troubleshoot all callback handlers in a
uniform way.
An async "job" API is introduced to manage independent logical threads
or work such as protocol transaction handlers on client, server, and
ICAP sides. These jobs should communicate with each other using async
calls to minimize dependencies and avoid reentrant callback loops.
These changes will eventually improve overall code quality, debugging
quality, and Squid robustness.
Most of the changes in the ICAP/ICAP* files are related to removing handler
wrappers and using CommCalls. AsyncJob is now a [virtual] base class for
ICAPInitiator. All ICAP initiators and ICAP transactions are now "jobs".
rousskov [Wed, 13 Feb 2008 06:07:52 +0000 (06:07 +0000)]
Merging async-call branch changes to HEAD:
Async-call work replaces event-based asynchronous calls with
stand-alone implementation. The common async call API allows Squid
core do call, debug, and troubleshoot all callback handlers in a
uniform way.
An async "job" API is introduced to manage independent logical threads
or work such as protocol transaction handlers on client, server, and
ICAP sides. These jobs should communicate with each other using async
calls to minimize dependencies and avoid reentrant callback loops.
These changes will eventually improve overall code quality, debugging
quality, and Squid robustness.
Below you will find log messages from the async-call branch that are
relevant to the file(s) being committed.
Made comm_read and comm_write calls to use the new CommCalls.
Converted the ConnStateData related code in client_side.cc to
use CommCalls.
BodyPipe now uses the new job calls interface. Many changes in
BodyPipe's related code.
rousskov [Wed, 13 Feb 2008 06:02:13 +0000 (06:02 +0000)]
Merging async-call branch changes to HEAD:
Removed calls to comm_iocallbackpending(). They were added to
"Speed up processing of queued events significantly, to not
cause large delays when under low load" but I do not see how
having pending callbacks can be relevant because all pending
callbacks are (should be) executed before FDs are probed.
rousskov [Wed, 13 Feb 2008 05:58:29 +0000 (05:58 +0000)]
Merging async-call branch changes to HEAD:
Async-call work replaces event-based asynchronous calls with
stand-alone implementation. The common async call API allows Squid
core do call, debug, and troubleshoot all callback handlers in a
uniform way.
An async "job" API is introduced to manage independent logical threads
or work such as protocol transaction handlers on client, server, and
ICAP sides. These jobs should communicate with each other using async
calls to minimize dependencies and avoid reentrant callback loops.
These changes will eventually improve overall code quality, debugging
quality, and Squid robustness.
Below you will find log messages from the async-call branch that are
relevant to the file(s) being committed.
Added initial implelentation of AsyncCall-based wrappers for
comm callbacks. The comm layer no longer calls callbacks from
the select loop. Instead, the select loop schedules async
calls. Scheduled calls are then called from the main loop (like
all other async calls), after the select loop completes.
Removed accept loop. We cannot loop because async calls do not
get fired during the loop and, hence, cannot register new
callbacks for new ready FDs.
The loop is implicit now. When the next callback is registered,
we check whether the last accept(2) call was successful or
OPTIMISTIC_IO is defined and call acceptNext() again if yes.
AcceptNext() may schedule another async call (using the
being-submitted callback) if the socket was still ready. Since
callbacks are fired until there are no callabacks left, we
still have an accept loop.
Removed CommDispatcher as unused.
Removed unused IOFCB, IOWCB, and CWCB.
Removed class fdc_t. After CommCallbackData removal, fdc_t was
only used to check that a FD is "active" and to check that a FD
is half_closed. fd_table flags.open seems to be identical to
the "active" state flag so we use that now, via newly added
isOpen() static function. AbortChecker already maintains
half_closed status.
The accept-specific functionality is still implemented by
AcceptFD class. Removed fdc_t used to marshall accept-ralted
calls to AcceptFD anyway. fdc_table now stores AcceptFDs
directly. I did not rename the table to ease merging with other
code, but added a TODO for that.
Removed calls to comm_iocallbackpending(). They were added to
"Speed up processing of queued events significantly, to not
cause large delays when under low load" but I do not see how
having pending callbacks can be relevant because all pending
callbacks are (should be) executed before FDs are probed.
Removed unused nullCallback() methods.
Removed CommCallbackData and related code. It looks like it
remained after one of the big comm rewrites, just to handle
accept errors. We can now schedule an async call to notify of
those errors and do not seem to need CommCallbackData at all.
Removed commfd_completed_events: a list of completed (but not
yet fired) callbacks. We simply schedule the async call now
instead of maintaining our own list of callbacks to call. This
change allows us to eliminate the CommDispatcher class (which
was the motivation behind these changes), but I have not done
that yet.
For comm_io_callback_t, being active seems to be the same as
having a callback. Replaced active data member with a method
that checks for the callback presence.
Relaxed comm_read_cancel() preconditions so that the callers do
not have to check all the assertions inside before calling that
function.
The CommCall code is still more complex than I want it to be.
However, these are low-level interfaces that most developers
will not have to deal with and the complexity will be
significantly reduced if we get rid of old style
function-pointer based callbacks, which is probably desirable
for several reasons.
amosjeffries [Tue, 12 Feb 2008 07:05:11 +0000 (07:05 +0000)]
extension_methods directive is now obsolete.
METHOD_OTHER type now cleanly handles unknown methods in squid.
Part 1:
- removes the directive from squid default config.
- removes the placeholder EXT methods enumerations
- deprecates the code which previously handled config parsing.
part 2: (TODO)
- needs to cleanup the remaining enumerated methods to reduce
memory footprint
amosjeffries [Tue, 12 Feb 2008 06:01:23 +0000 (06:01 +0000)]
Several String fixes.
- Add two missing includes when configured --disable-inline
- SegFault when newly allocated string set to 0-length.
buf_ may be NULL and buffer overruns on truncate attempt
rousskov [Tue, 12 Feb 2008 05:41:52 +0000 (05:41 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
Added sslproxy_cert_error ACL to be able to bypass some
certificate validation errors. The default is to bypass
nothing.
Only a few SSL certificate validation errors are currently
recognized by name, including the newly defined
SQUID_X509_V_ERR_DOMAIN_MISMATCH error which is raised when
Squid certificate domain validation fails.
rousskov [Tue, 12 Feb 2008 05:33:48 +0000 (05:33 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
When SslBump is activated, Squid responds to CONNECT request
with HTTP 200 "Connection established" and switches to SSL
encryption on the connection.
Added ssl_bump ACL. The ACL controls which CONNECT requests to
an http_port marked with an sslBump flag are actually "bumped".
This was needed to bypass the bump and tunnel requests to sites
that Squid cannot handle well (with or without the bump), with
other use cases likely to surface.
By default, no requests are bumped. Squid warns if http_port(s)
have SslBump but no ssl_bump ACL was configured.
The ACL applies to all http_ports, but I suspect that, if
needed, specific ACL rules can distinguish ports using
"myport".
Added ERR_SECURE_CONNECT_FAIL that is returned when we cannot
secure the established connection with the server. Formerly,
ERR_CONNECT_FAIL was returned.
rousskov [Tue, 12 Feb 2008 05:30:10 +0000 (05:30 +0000)]
The connect(2) system call might return "connection ready"
status even for a non-blocking file descriptor. The connection
itself can never be immediately ready in reality because of the
TCP handshake, but I am guessing that in some environments, the
TCP stack fakes/optimizes local connection readiness. We have
seen that for loopback sockets on FreeBSD 6.2, for example, but
the behavior is probably OS- or OS-configuration specific.
If connect(2) is immediately successful, comm module
immediately calls the callback. This means that the callback is
called while the same callback is being registered with comm.
ICAP does not allow this "re-entrance" and other code might not
deal well with it.
The change overwrites connect(2) result so that Squid does not
think that connect(2) was immediately successful. Instead of
calling the callback, Squid then schedules the connection
write-ability check.
The NativeAsyncCall development will fix this and remove the
need to overwrite connect(2) result because comm will always
call callbacks asynchronously.
rousskov [Tue, 12 Feb 2008 05:28:47 +0000 (05:28 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
SslBump needs access to SSL-related options when responding to
CONNECT requests. We could add a new squid.conf option to
accept those options, but I decided to add SSL-related options
to http_port, where CONNECT requests will arrive. This design
avoids the necessity to map SSL options to an HTTP port or to a
CONNECT request when a global default would not do.
Adding SSL options to http_port makes http_port_list and
https_port_list almost identical. I moved stuff around to
reduce duplication between the two classes, moving related
code to ProtoPort.{cc,h} in the process. More work is needed
to remove the distinction completely or, at least, remove the
https_port::http hack.
Added sslproxy_cert_error ACL to be able to bypass some
certificate validation errors. The default is to bypass
nothing.
Only a few SSL certificate validation errors are currently
recognized by name, including the newly defined
SQUID_X509_V_ERR_DOMAIN_MISMATCH error which is raised when
Squid certificate domain validation fails.
rousskov [Tue, 12 Feb 2008 05:26:59 +0000 (05:26 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
Parse/dump the newly added sslBump http_port option. The option
does not imply an accelerated port because it only accelerates
after intercepting a CONNECT request.
Configure SSL Context if SSL certificate or key are specified
for an HTTP port.
rousskov [Tue, 12 Feb 2008 05:26:39 +0000 (05:26 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
Added ERR_SECURE_CONNECT_FAIL that is returned when we cannot
secure the established connection with the server. Formerly,
ERR_CONNECT_FAIL was returned.
Supply the SSL error check ACL when forwarding SSL connections.
The check list is stored as SSL ex_data and is destroyed by
OpenSSL when the SSL struct is destroyed.
rousskov [Tue, 12 Feb 2008 05:26:16 +0000 (05:26 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
When SslBump is activated, Squid responds to CONNECT request
with HTTP 200 "Connection established" and switches to SSL
encryption on the connection.
Added ssl_bump ACL. The ACL controls which CONNECT requests to
an http_port marked with an sslBump flag are actually "bumped".
This was needed to bypass the bump and tunnel requests to sites
that Squid cannot handle well (with or without the bump), with
other use cases likely to surface.
By default, no requests are bumped. Squid warns if http_port(s)
have SslBump but no ssl_bump ACL was configured.
The ACL applies to all http_ports, but I suspect that, if
needed, specific ACL rules can distinguish ports using
"myport".
Added ERR_SECURE_CONNECT_FAIL that is returned when we cannot
secure the established connection with the server. Formerly,
ERR_CONNECT_FAIL was returned.
rousskov [Tue, 12 Feb 2008 05:25:22 +0000 (05:25 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
Added ConnStateData::switchToHttps() method that attempts to
switch regular connection into SSL encryption mode. It is meant
to be called after Squid responds to the CONNECT request,
pretending to be an origin server (SslBump feature).
Moved SSL structure creation to httpsCreate() so that the code
can be used for httpsAccept() and the new switchToHttps() code.
Mark connection that switched to SSL so that others can treat
associated requests specially (e.g., accelerate them).
Use configured SSL context of http_port. This means that we
can no longer rely on connection port protocol when
accelerating request. The port is HTTP but we need to prepend
https:// after switching to SSL mode.
Warn if sslBump is enabled but there is no usable SSL context.
This may need to be a fatal error.
Ignore https_port where we failed to create an SSL context. We
used to warn but did not ignore. Polished the warning message.
Disable SslBump for http_port where we failed to create an SSL
context. We used to warn but did not disable. Polished the
warning message.
Polished the "You have too many 'https_port' lines" warning.
rousskov [Tue, 12 Feb 2008 05:24:38 +0000 (05:24 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
SslBump needs access to SSL-related options when responding to
CONNECT requests. We could add a new squid.conf option to
accept those options, but I decided to add SSL-related options
to http_port, where CONNECT requests will arrive. This design
avoids the necessity to map SSL options to an HTTP port or to a
CONNECT request when a global default would not do.
Adding SSL options to http_port makes http_port_list and
https_port_list almost identical. I moved stuff around to
reduce duplication between the two classes, moving related
code to ProtoPort.{cc,h} in the process. More work is needed
to remove the distinction completely or, at least, remove the
https_port::http hack.
rousskov [Tue, 12 Feb 2008 05:17:41 +0000 (05:17 +0000)]
Importing SslBump feature from Squid3 ssl-bump branch:
Added ERR_SECURE_CONNECT_FAIL that is returned when we cannot
secure the established connection with the server. Formerly,
ERR_CONNECT_FAIL was returned.
I copied new English/ERR_SECURE_CONNECT_FAIL to other
languages. Perhaps it would be better to use translated (but
less specific) ERR_CONNECT_FAIL text instead.
swilton [Sat, 9 Feb 2008 10:48:05 +0000 (10:48 +0000)]
Author: Steven Wilton <swilton@q-net.net.au>
Sort cache list in wccpv2 to ensure a consistent hash allocation across all serv
ices
Under some circumstances different caches were being allocated differnet hash
assignments for different wccpv2 services. This caused problems when using
TPROXY as the reply traffic was being sent to a different cache than the request
causing problems in browsing web pages.
This patch applied a selection sort algorithm to the cache list to ensure that
as long as the same group of caches are registered for any number of wccpv2
services, they will be allocated the same hash assignments in the router.
rousskov [Sat, 9 Feb 2008 01:31:02 +0000 (01:31 +0000)]
Bug 2038: check reply_body_max_size before ICAP
Use newly added HttpReply::*BodyTooLarge() checks to support the
reply_body_max_size ACL when the expected body size may be known and when more
body data is coming. The checks are now shared between server and client
sides.
The server side needs to do the checks before feeding the data to ICAP.
Otherwise, ICAP service may suck a large response in and spit a small response
out. It was decided that the reply_body_max_size ACL should apply to the
virgin response as well.
The server side does not perform receivedBodyTooLarge check yet.
rousskov [Sat, 9 Feb 2008 01:30:18 +0000 (01:30 +0000)]
Bug 2038: check reply_body_max_size before ICAP
Use newly added HttpReply::expectedBodyTooLarge() check to support the
reply_body_max_size ACL when the expected body size may be known. The check
is now shared between server and client sides.
The server side needs to do the checks before feeding the data to ICAP.
Otherwise, ICAP service may suck a large response in and spit a small response
out. It was decided that the reply_body_max_size ACL should apply to the
virgin response as well.
The server side does not perform HttpReply::receivedBodyTooLarge() check yet.
rousskov [Sat, 9 Feb 2008 01:27:59 +0000 (01:27 +0000)]
Bug 2038: check reply_body_max_size before ICAP
Moved maxReplyBodySize-related code from ClientHttpRequest to HttpReply
because server-side needs it too to check limits before ICAP sucks all the
data in.
Calculating limit requires knowing HttpRequest because it affects expected
content length. Since I did not find a single place where any HttpReply would
be guaranteed to be given the request to calculate the limit, we now supply
the request whenever a limit check is performed. The limit calculation result
is cached and the calculation should not be repeated.
checkRetry is now guaranteed to return false for requests that checkRetriable
previously returned false. This prevents retries of requests with
non-idempotent methods, for example.
rousskov [Fri, 8 Feb 2008 01:16:24 +0000 (01:16 +0000)]
Author: Christos Tsantilas <chtsanti@users.sourceforge.net>
Bug 2172: When user fails authentification Squid restarts
Replaced an assertion that there is no auth_user_request, a comment what
to do when there is no auth_user_request, and a comment that there can
be no auth_user_request with what the first comment suggested to do.
amosjeffries [Wed, 6 Feb 2008 13:54:14 +0000 (13:54 +0000)]
Alter caching policy for Dynamic Objects.
- Alters squid caching policy for dynamically generated objects
to allow caching when objects have Cache-Control headers.
- Default policy now follows RFC 2616 section 13.9 exactly as stated.
see http://www.mail-archive.com/squid-dev@squid-cache.org/msg06963.html
amosjeffries [Sun, 3 Feb 2008 17:00:29 +0000 (17:00 +0000)]
Fixes many Unit-test compile errors and testing problems in HttpRequestMethod
- Removes implicit conversion from HttpRequestMethod to other types.
- Adds id() accessor to retrieve an ID for known methods.
- Adds more boolean operators to class library
Also adds auto-documantation in some METHOD-related places.
serassio [Fri, 25 Jan 2008 02:20:43 +0000 (02:20 +0000)]
Windows port: Handle notification of IP address changes for dial-up connections
On Windows 2000 and later, the NotifyAddrChange() function allow a
process to be notified of the changes in the system IP addresses table.
This patch generate a reconfigure request after any notification, this
allow the hot addition/reconfiguration of network interfaces without
manually restart/reconfigure Squid.
amosjeffries [Thu, 24 Jan 2008 03:51:16 +0000 (03:51 +0000)]
Protect String::limitInit() against empty strings.
Is being caleld at least once in squid3 with a NULL ptr and 0 length.
This patch prevents memory allocation on 0 length and a will continue
to assert if a NULL pointer is given with a length.
(We may want to re-think that later)
rousskov [Thu, 24 Jan 2008 02:24:14 +0000 (02:24 +0000)]
String 'call duplication' optimization(?) bypassed calling String::init()
when copying a String and called limitInit() directly. That results in
an assertion when an empty String is copied. The fix avoids calling
limitInit() when an empty String is copied.
rousskov [Wed, 23 Jan 2008 03:54:28 +0000 (03:54 +0000)]
Bug 2168 fix: Avoid tcp_outgoing_address selection for ICAP requests.
Let OS pick the outgoing (i.e., source) address instead.
Code did not supply info required for tcp_outgoing_address ACL match to work,
and tcp_outgoing_address is usually not applicable to ICAP communication.
Eventually, we may add icap_outgoing_address.
amosjeffries [Sat, 19 Jan 2008 14:15:28 +0000 (14:15 +0000)]
EPSV support for FTP and other fixes.
- Adds full EPSV method support for FTP server connections
- Fixes debugging in FTP state machine into specific levels:
* 0: critical problems
* 1: non-critical problems
* 2: FTP protocol chatter
* 3: FTP logic flow debugging
* 5: FTP data parsing flows
- Adds code documentation to some FTP functions.
amosjeffries [Sat, 19 Jan 2008 14:11:34 +0000 (14:11 +0000)]
Miscenllaneous fixes for CNAME, include, and comm.
- Add more wrapping of CNAME code no longer needed.
- Fixes typo in include directive ported code.
- Fixes incorrect socket family setting on some accepted connections.
hno [Mon, 14 Jan 2008 19:13:49 +0000 (19:13 +0000)]
Bug #2153: Use the cache_peer name in CARP hashing to support multiple peers on the same host
The cache_peer name= option was introduced some years ago to allow
peers to be named and to suppot multiple peers on the same host/address.
However, carp still used the hostname/address and not the name making
it fail slightly when forwarding to multiple peers on the same address.
amosjeffries [Fri, 11 Jan 2008 12:04:02 +0000 (12:04 +0000)]
Formal Debug Levels part 1 - Define names for some debug levels.
This allows the use of DBG_CRITICAL and DBG_IMPORTANT instead of magic
numbers 0 and 1 in the debugs() calls. Making code a little more readable
and easier to use. The code itself is not modified to use them yet.
DBG_DATA is added for level 9, though that may change.
TODO: the other levels (2-8) still need deciding. No consensus was reached
amongst the developers for those levels when discussed.
amosjeffries [Fri, 11 Jan 2008 10:49:18 +0000 (10:49 +0000)]
AAAA/A failover fix and CNAME recursion deprecation
A bug in the final version of squid internal DNS resolver logics
caused any failover A results to overwrite the paired previous AAAA.
This patch adds state to store the DNS results between failover queries
and to merge the final sets before passing them out to the requestor.
Lookups should now be seemlessly handled within the DNS resolver stub.
CNAME recursion at the ipcache level should now be obsolete and has been
wrapped in a new ./configure --with-dns-cname option which defaults off.
That code has proven to be problematic anyway and will be no great loss.
Additional counters have been added to the squid statistics to track the
amount of queries of each type have been encountered.