Amos Jeffries [Fri, 17 Jun 2011 16:32:04 +0000 (04:32 +1200)]
Upgrade comm layer Connection handling
The premise underlying this large patch is that instead of copying and
re-copying and re-lookups for the FD related data we can take the
ConnectionDetail class which is generated to store a few bits of IP
information about newly accept()'d connections and make it persist across
the whole of Squid.
It has been renamed from ConnectionDetails to Comm::Connection and has
absorbed a few FD data fields from other classes long the code paths.
Its scope is to hold an FD (or potential FD) plus meta data.
Comm::Connection are valid before, during and after the period when their
particular FD is open. The meta data may be used beforehand to setup the
FD (in the case of peer selection or other TcpAcceptor usage), and it may
remain in use after FD closure until logging or all linked job and call
objects have detected the closure and terminated. A global function
Comm::IsConnOpen() may be used on the pointer objects to detect whether
they point at an active connection.
Most of the patch is simple parameter changes to functions and methods to
pass a "cont Comm::ConnectionPointer &" instead of an "int FD". Along with
class FD fields being converted to these object pointers.
In order to support this alteration there have been behavioral changes to:
The socket accept() Job
Comm::TcpAcceptor altered to spawn Comm::Connection objects and to
operate with one controlling their active/closed state.
FTP data channel handling Calls.
efficiency improvements making use of Comm::Connection as a feedback
channel between TcpAcceptor and FtpStateData to cancel the listening
Job. Most of the underlying logic change is already in trunk to use
the Subscription API. This just streamlines and fixes some race bugs.
Peer selection
updated to spawn a set of Comm::Connection objects. To do this it is
updated to determine *all* peers including DIRECT ones. Doing the DNS
lookup instead of leaving it to comm_connect() on the other side of
FwdState. It also absorbs the outgoing address selection from FwdState
and can now specify details of local+remote ends of an outgoing TCP link.
Forwarding
updated to handle the new outputs from peer selection and to open sequentially.
pconn handling
updated to use destination IP/port and hold a Comm::Connection instead
of domain name indexing an FD. This allows us to maintain idle pools
and re-use FD more efficiently with virtual-hosted servers. Along
with maintaining certainty that the pconn selected actually goes to
the exact destination IP:port needed by forwarding.
comm layer outgoing connections
now have a control job Comm::ConnOpener to do this. Due to the peer
selection and forwarding changes this is a much simpler operation.
HTTP / CONNECT tunnel / gopher / whois / FTP
updated to receive a server and client Comm::Connection object from
forwarding. To operate on those until they close or are finished with.
SNMP / ICP / HTCP / DNS port listeners
updated to work with Comm::Connection holding their listening socket
meta data. This is a side-effect of the ICP and Comm read/write/timeout
changes.
This project adds support for a translatable and customisable error detail file
(errors/templates/error_details.txt). The file is stored like we store error
page templates today. Inside the file, an HTTP-like format used that can be
later extended to other error details (and beyond):
name: value
details: "value"
descr: "value"
or
name: value
details: "multi
line
value"
descr: "value with a \"quoted string\" inside"
The code supports future translations, just like Squid already support error
page translations.
This is a Measurement Factory project
----
Some Technical details:
- The errorpage code which is related to loading and parsing error templates
moved to TemplateFile class. This class is used as base class for
ErrorPageFile class which used to load error page templates.
- The HttpHeader parser used to parse error details
- The error details for various languages cached to memory
- The ErrorDetailsList used to store a list of error details for a
language/locale
- The ErrorDetailsManager is a class used to load and manage multiple error
details list (ErrorDetailsList objects) for many languages. It also
implements a simple cache.
Amos Jeffries [Fri, 17 Jun 2011 02:31:45 +0000 (14:31 +1200)]
Fix squidclient -V option and allow non-HTTP protocols to be tested
The "-" case is for old style HTTP (called 0.9) where there is no version
string. The "-V 0.9" is for testing servers with broken version number
tag "HTTP/0.9". Do not mix these up!
This also adds the ability to send non-HTTP version tags for testing.
ie "-V ICAP/1.0" or "-V ICY/1.0"
Amos Jeffries [Fri, 17 Jun 2011 02:14:01 +0000 (14:14 +1200)]
Upgrade ICAP persistent connection handling
ICAP services use a "service" model of pconn different from the
"TCP destination" model which PconnPool objects are designed for.
This patch alters Adaptation::Icap::ServiceRep to use the simpler
IdleConnList object for pconn storage. IdleConnList stores a
"set of idle connections" more compatible with the ICAP model.
In order to implement ICAP max-connections feature the closeN()
operation is added to IdleConnList.
The result is removal of the complex hash and management operations on
push/pop of the idle conn set. The only expected behaviour change is
more frequent re-use of idle connections on services with multiple IP
addresses. Speed gains are minimal, but positive.
Fixed bypass of SSL certificate validation errors.
The bypass code was calling ACLChecklist::fastCheck() multiple times
if multiple certificate errors were found. That method should not be
called multiple times because it changes the internal ACLChecklist
state, producing wrong answers for repeated calls.
This patch fixes the ACLChecklist::fastCheck() method so it can be called
multiple times. Each fastCheck() call results in an independent access
list check.
author: Alex Rousskov <rousskov@measurement-factory.com>, Christos Tsantilas <christos@chtsanti.net>
Bug 3153 fix: Prevent ICAP RESPMOD transactions getting stuck with the adapted body.
Part 1.
Server is expected to receive adapted response headers and then consume the
adapted response body, if any. If the server receives the headers and then
aborts, it must notify the ICAP side that nobody will consume the body.
Otherwise, the ICAP transaction will fill the BodyPipe buffer and get stuck
waiting for the consumer to free some space.
Part 2:
This fix still leaves one potential race condition unhandled: The ICAP
Initiatee disappears right after sending the adapted headers to the Server
(because there is nothing else for that initiatee to do). After the
noteAdaptationAnswer() call is scheduled by ICAP and before it is received by
the Server job, there is no usable link between Server and ICAP. There is no
way for the Server to notify the ICAP transaction that the Server job is
aborting during that time (and there is no Server job at all after it aborts,
naturally).
The solutions is to develop a custom AsyncCall which will call the
expectNoConsumption() on the message pipe if the call cannot be dialed (i.e.,
the message cannot be delivered to Server).
Amos Jeffries [Mon, 13 Jun 2011 12:25:12 +0000 (06:25 -0600)]
Auth lookup state cbdata upgrade
The authenticators utilize a "statedata" structure to store and pass
the callback and Auth::UserRequest an auth lookup is about.
This patch converts the structure from a CBDATA_GLOBAL_TYPE struct to a
CBDATA_CLASS2 and adds a parameterized constructor for it.
The result is that all the code using it no longer has to explicitly
manage fields assignments and cbdata referencing. Simply new the object
when submitting to the helper system and delete once its handler has
been called.
Amos Jeffries [Sun, 5 Jun 2011 15:43:46 +0000 (03:43 +1200)]
Request-Line log format codes for split client and server view
Since URL-rewrite, adaptation and a number of other things can alter the
request as it goes through Squid it is useful to be able to log both the
original incoming and the final outgoing versions of these details.
This adds logformat codes to extend %rm %ru %rp and %rv with < and >
modifiers for in and outbound display.
The old tag versions are kept for now. There is still work to do cleaning
up the log data object which will clarify if the various data fields
which used to be passed to logging match either of the original or final
request data and can be obsoleted. It is already clear that the logged
path matches neither and contains some normalized version.
Amos Jeffries [Thu, 2 Jun 2011 10:39:53 +0000 (04:39 -0600)]
Translation: auto-generate the lists of available translation files
Use automake includes built during source maintenance and stored in VCS.
Avoiding earlier problems encountered with the approaches attempting to
bootstrap the translation from within Makefile.
This also generates relevant lists for the manual pages. Preparing to
translate those automatically as well.
Currently the httpHeaderParseQuotedString does not support multiline quoted
string fields. According the rfc2616 multiline quoted string fields should
supported:
quoted-string = ( <"> *(qdtext | quoted-pair ) <"> )
qdtext = <any TEXT except <">>
TEXT = <any OCTET except CTLs,
but including LWS>
LWS = [CRLF] 1*( SP | HT )
This patch replaces the "[\r]\n " or "[\r]\n\t" with a single space.
Amos Jeffries [Tue, 24 May 2011 05:33:47 +0000 (17:33 +1200)]
Docs: fancy up ERR_INVALID_REQ with some javascript
Some of the "possible problems" can be determined as not-relevant.
When these cases are detected hide the text from viewers in a way that
keeps the page operational when javascript is disabled.
Amos Jeffries [Sat, 21 May 2011 01:13:42 +0000 (13:13 +1200)]
URL re-writer handling bug fixes
This patch includes two bug fixes in URL handling which were uncovered
during testing of the URL logging update:
* URL re-write handling was not correctly creating its adapted request
copy. The code here is much reduced by using the clone() method. Still
not completely satisfactory (marked with XXX) since on invalid URL
there is a wasted cycles cloning and deleting almost immediately.
Future cleanups moving the URL parts outside HttpRequest will fix that.
* URL parsing needs to set the canonical field to unset whenever the URI
is re-parsed into a request. This field is an optimization for later
display speed-ups. This has been causing incorrect canonical URL to be
used following re-write. When the cloning above was corrected it caused
asserts in the server-side.
* To prevent memory leaks the urnParse() function internal to URL parsing
is adjusted to accept and update an existing request in identical API
semantics to urlParse() instead of always generating a new one.
Currently, SSL error detail in Squid-generated error pages (%D) contains
both the error name and the explanation text. Some folks using this feature
want to render the two pieces of information differently because the error
name is not something most end-users should read or focus on.
This patch adds the "%x" error page formating code which prints the error name,
and removes the error name (%err_name) from SSL error detail messages.
This patch implements the phase 1 of the ICAP Max-Connections feature as it is
described in squid wiki:
http://wiki.squid-cache.org/Features/ServiceOverload
The behaviour of the patch can be configured using on_overload and max_conn
options of the icap_service configuration parameter. Squid can be configured
to do one of the following:
- Block: send and HTTP error response to the subscriber
- Bypass: ignore the "over-connected" ICAP service
- Wait: wait (in a FIFO queue) for an ICAP connection slot
- Force: proceed, ignoring the Max-Connections limit
Squid warns the first time the service become overloaded
For more information please visit the feature wiki page given above.
Technical informations:
The patch starts count a connections to the ICAP server as active when the
ModXact class receives an FD even if the fd is not really connected to the
server yet, and decrease the active connections to the server when the ModXact
object releases its fd connection.
If the Max-Connection limit is reached squid puts the request to a waiters list.
When one or more connections released squid schedules one or more waiters for
execution and remove them from waiters list.
To handle cases where a waiter gone/canceled before its execution the custom
dialer ConnWaiterDialer used.
The Options connections counted as active connections but are not limited by
the Max-Connections limit. An Option request will be executed even if the
maximum connections number is reached.
Tilmann Bubeck [Mon, 9 May 2011 12:42:59 +0000 (00:42 +1200)]
Add ext_time_quota_acl helper
Allows an administrator to define time budgets for the users of squid
to limit the time using squid.
This is useful for corporate lunch time allocations, wifi portal
pay-per-minute installations or for parental control of children. The
administrator can define a time budget (e.g. 1 hour per day) which is
enforced through this helper.
Andrew Beverley [Sun, 8 May 2011 23:21:44 +0000 (11:21 +1200)]
QoS: require libcap before enabling netfilter MARK support
As it is not possible to get or set a netfilter mark without libcap, this
patch will disable netfilter marking at compilation time if libcap is not
available (in a similar way to Linux transparent proxying).
Amos Jeffries [Sun, 8 May 2011 13:53:10 +0000 (01:53 +1200)]
Cleanup: sync NTLM and Negotiate UserRequest code
Minor tweaks to reduce diff between the files. No logic changes.
Renames the addHeader() to addAuthentiocationInfoHeader(),
Renames the addTrailer() to addAuthentiocationInfoTrailer() and
document that they add additional *-Info header to the HTTP reply.
Amos Jeffries [Sun, 8 May 2011 06:11:18 +0000 (18:11 +1200)]
Cleanup: Improve Connection Pinning management
Since 1xx handing went in HttpRequest has had two links to the one
ConnStateData managing its client connection.
* Rename the 1xx link to clientConnectionManager (since it is not
actually the connection, but the manager object controlling the FD
usage and stats.
* Convert the pinning code to using the permanent clientConnectionManager
link instead of a temporary pinned_connection link.
This moves all connection pinning state fully into the ConnStateData
manager objects scope.
Side changes that appear to be buggy code previously:
* do not alter pinning state at the point where the pinned connection is
about to start being used. Changes are only relevant at the point of
pinning or unpinning.
* unpin operation now closes the Server FD if still open. Previously
there was the possibility that some code paths would leave server FD
open and pconn it. (especially since the above mentioned state
alteration cleared the "pinned" flag).
projects / thirdparty / squid.git / log
