Alex Rousskov [Fri, 14 Dec 2012 16:10:05 +0000 (09:10 -0700)]
Polished helper messages debugging slightly.
Made stateful and stateless debugging a little bit more similar/symmetric.
Debug helperStatefulSubmit message contents length.
Debug accumulated helperStatefulHandleRead buffer.
Polished helperHandleRead buffer debugging.
Alex Rousskov [Fri, 14 Dec 2012 16:07:18 +0000 (09:07 -0700)]
Added Raw, an std::ostream manipulator to print possibly non-terminated buffers
and their labels/sizes.
The Raw manipulator tries to be smart about printing buffers at various
debugging levels: Large buffers are only printed at DBG_DATA by default.
This allows the caller to mix higher-level debug messages with dumping
of potentially large volumes of data. This smartness can be overruled
using an explicit minLevel() method call.
This patch add cache to cert validation helper. The following new options
added to "sslcrtvalidator_program" configuration parameter to control cache
behaviour:
ttl=n TTL in seconds for cached results.The default is 60 secs
cache=n limit the result cache size. The default value is 2048
To implement the cert validation cache a new template class investigated,
the LruMap which implements a simple lru cache.
The LruMap templete class also used to replace the old Ssl::LocalContextStorage
class which implements a SSL contexts cache.
Amos Jeffries [Sat, 8 Dec 2012 01:50:46 +0000 (14:50 +1300)]
Fix helper buffer management
Identified by Tsantilas Christos.
"One byte remains in helpers io buffer after parsing the response. This is
will cause problems when the next response from helper will enter squid.
After the srv->rbuf parsed, the srv->rbuf still contains on byte (a '\0'
char) and the srv->roffset is 1."
Perform memmove() in helperHandleRead() instead of helperReturnBuffer()
* helperReturnBuffer is only dealing with the message sub-string, but has
been made to memmove() the buffer contents which means it has to become
aware of these problematic terminal \0 octet(s). However
helperHandleRead() is where the termination is being done and the
buffer information is all being handled.
Pass the first of the termination \0 octets to helperReturnBuffer() not
the last \0. This is made possible after fixing (1a).
* helpers which return \r\n were still passing the location of the \n as
endpoint to workaround (1a) even though the \r is also replaced with
\0 and shortens the msg portion by one octet. This affects the
HelperReply parser length checks on responses without kv-pair.
Also, clear out an obsolete TODO.
Also, document a remaining design issue in stateful helpers assuming each
read() operation is on response and ignoring any octets in the buffer
after parsing one reply.
SSL server certificate validator implementation: Ssl::CertValidationMsg::parseResponse fix
The parsed SSL errors returned by cert validator checking is wrong and causes
the Ssl::CertValidationMsg::parseResponse method to return false on valid
messages
This patch add the "server_ssl_cert_fingerprint" acl type to match against
server SSL certificate fingerprint.
The new acl type has the form:
acl aclname server_ssl_cert_fingerprint [-sha1] fingerprint1 ...
The fingerprint must given in the form:
XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX
where X are any valid hexadecimal number
Example usage:
acl BrokeServer dst 192.168.1.23
acl GoodCert server_ssl_cert_fingerprint AB:2A:82:AF:46:AE:1F:31:21:74:65:BF:56:47:25:D1:87:51:41:AE
sslproxy_cert_error allow BrokeServer GoodCert
sslproxy_cert_error deny all
This patch implements the certificate validation helper interface described at:
http://wiki.squid-cache.org/Features/SslServerCertValidator
The helper consulted after the internal OpenSSL validation, regardless of the
validation results. The helper will receive:
1) the origin server certificate [chain],
2) the intended domain name, and
3) a list of OpenSSL validation errors (if any).
If the helper decides to honor an OpenSSL error or report another validation
error(s), the helper will return:
1) A list of certificates.
2) A list of items consists the the validation error name (see %err_name
error page macro and %err_details logformat code), error reason
(%ssl_lib_error macro), and the offending certificate.
The returned information mimics what the internal OpenSSL-based validation code
collects now. Returned errors, if any, fed to sslproxy_cert_error, triggering
the existing SSL error processing code.
The helper invocation controlled by the "sslcrtvalidator_program" and
"sslcrtvalidator_children" configurations options which are similar to the
ssl_crtd related options.
A simple testing cert validation helper developed in perl included in this
patch. This helper just echo back the certificate errors.
Amos Jeffries [Fri, 30 Nov 2012 23:02:11 +0000 (12:02 +1300)]
Add kv-pair support to url_rewrite_helper interface
This stage of the helper reply protocol adds kv-pair support to the
url_rewrite_helper interfacefor URL redirect and rewrite operations.
It uses the new Notes objects and kv-pair field added by the stage 2
helper protocol instead of parsing the 'other' field. Although,
the 'other' field is still parsed when *no* result field is received
for backward compatibility with older helpers.
The response syntax for URL helpers becomes:
[channel-ID SP] result [SP kv-pair ...] [SP other] EOL
NP: 'other' field is now deprecated and will be ignored/discarded on any
response containing a result code field.
When result code "OK" is presented by the helper several kv-pairs are
reserved to control Squid actions:
* "rewrite-url=" is added to return a re-written URL.
- When this key is presented the URL is re-written to the new one by
Squid without client interaction.
- The 'url' keys presence will override this key.
* "url=" is added to return the redirected-to URL.
- When this key name is presented an HTTP redirect response is generated
for the client.
- This keys presence overrides the 'rewrite-url' key actions.
* "status=" is added to hold the HTTP status code for use in redirect
operations.
- This field is optional and status is no longer required for marking
redirect actions.
- If no redirect status is provided Squid will assign one (currently the
default is 302, that may change in the future).
- This key is only relevant when 'url' key is also presented. In all
other uses it is currently ignored.
When result codes BH or ERR are presented by the helper no redirect or
rewrite action is performed and no kv-pair key names are reserved for use
at this time.
Any other keys MAY be sent on any response. The URL helper interface
makes no other use of them, but this patch does pass them on to the ALE
object for logging as transaction Notes. All kv-pairs returned by the
helper (including the url, stauts rewrite-url keys) are available for
logging via the %{...}note log format option.
As with changes to other helpers interfaces in earlier updates, only
the first value presented for any of the reserved kv-pairs is used.
Multiple values are accepted as notes, but otherwise ignored by Squid and
do not affect the transaction outcome.
Additionally, when the BH result code is received from the helper a
simple recovery is attempted. The lookup request will be re-scheduled (up
to once) in an attempt to find a better responding helper.
NOTE: Helper notes are *not* passed to adaptation interfaces.
- REQMOD adaptation happens before URL helpers are used.
- REPMOD adaptation may at some point receive them, but that change is
not done by this update.
Amos Jeffries [Fri, 30 Nov 2012 11:57:23 +0000 (04:57 -0700)]
Polish: Improve the messages output by UFS swap log management.
It also includes a small local variable symbol change from "new_path" to
"tmp_path" to prevent future developer mistakes like the one seen in
bug 3663 mistaking "new_path" for the new destination path of xrename().
Amos Jeffries [Fri, 30 Nov 2012 11:08:47 +0000 (00:08 +1300)]
Add kv-pir support to url_rewrite_helper interface
This stage of the helper reply protocol adds kv-pair support to the
url_rewrite_helper interfacefor URL redirect and rewrite operations.
It uses the new Notes objects and kv-pair field added by the stage 2
helper protocol instead of parsing the 'other' field. Although,
the 'other' field is still parsed when *no* result field is received
for backward compatibility with older helpers.
The response syntax for URL helpers becomes:
[channel-ID SP] result [SP kv-pair ...] [SP other] EOL
NP: 'other' field is now deprecated and will be ignored/discarded on any
response containing a result code field.
When result code "OK" is presented by the helper several kv-pairs are
reserved to control Squid actions:
* "rewrite-url=" is added to return a re-written URL.
- When this key is presented the URL is re-written to the new one by
Squid without client interaction.
- The 'url' keys presence will override this key.
* "url=" is added to return the redirected-to URL.
- When this key name is presented an HTTP redirect response is generated
for the client.
- This keys presence overrides the 'rewrite-url' key actions.
* "status=" is added to hold the HTTP status code for use in redirect
operations.
- This field is optional and status is no longer required for marking
redirect actions.
- If no redirect status is provided Squid will assign one (currently the
default is 302, that may change in the future).
- This key is only relevant when 'url' key is also presented. In all
other uses it is currently ignored.
When result codes BH or ERR are presented by the helper no redirect or
rewrite action is performed and no kv-pair key names are reserved for use
at this time.
Any other keys MAY be sent on any response. The URL helper interface
makes no other use of them, but this patch does pass them on to the ALE
object for logging as transaction Notes. All kv-pairs returned by the
helper (including the url, stauts rewrite-url keys) are available for
logging via the %{...}note log format option.
As with changes to other helpers interfaces in earlier updates, only
the first value presented for any of the reserved kv-pairs is used.
Multiple values are accepted as notes, but otherwise ignored by Squid and
do not affect the transaction outcome.
Additionally, when the BH result code is received from the helper a
simple recovery is attempted. The lookup request will be re-scheduled (up
to once) in an attempt to find a better responding helper.
NOTE: Helper notes are *not* passe to adaptation interfaces.
- REQMOD adaptation happens before URL helpers are used.
- REPMOD adaptation may at some point receive them, but that change is
not done by this update.
Amos Jeffries [Tue, 27 Nov 2012 22:02:02 +0000 (11:02 +1300)]
Helper protocol upgrade: add optional kv-pair field to responses
This adds an optional kv-pair field immediately following the result field
on all helper interface responses. This field contains a list of key=value
pairs where the value is a RFC1738 (URL) encoded token or a quoted string.
Quoted-string MAY contain whitespace and shell escaping.
The response syntax for all helpers becomes:
[channel-ID SP] result [SP key-pair ...] [SP other] EOL
The parser for HelperReply is also updated to map the old AF and NA
NTLM/Negotiate response fields into the HelperReply notes:
* "token=" is added to supply the NTLM and Negotiate server blob/token.
* "user=" is added to supply the user label field.
The relevant callback handlers are updated for these helpers to make use
of these new keys.
The bundled Digest authentication helpers are all upgraded to send the new
format responses. They now use ERR for failed lookup, BH for internal
errors, and OK with "ha1=" key added to supply a HA1 response. The handler
for Digest authentication is updated to process the new HelperReply fields
with failover the old format on Unknown result codes.
The external ACL handler is updated to pull its key=value pairs out of the
Notes list. The old parser loop becomes useless with this and is removed.
Taking with it support for several long deprecated keys "login=", "passwd=",
and "error=" which are now ignored.
Any other keys MAY be sent on any response. However at this stage 2 patch
they are ignored. As are repeated / secondary values for the expected key
names, only the first instance sent in the response is used.
Amos Jeffries [Tue, 27 Nov 2012 21:19:46 +0000 (10:19 +1300)]
Audit Review updates
* guarantee that note values output by the HelperReply parser are ""
nil-terminated string and not undefined String.
* rename HelperReply::responseKeys to 'notes'
* rename Notes::findByName() to find()
* add Note::firstValue() to locate first value provided for a given key
and present it as a char* terminated string
* various documentation updates
Amos Jeffries [Tue, 27 Nov 2012 10:57:12 +0000 (23:57 +1300)]
Treat no-cache and must-revalidate in Authentication
Wrapped as a violation because this operation is off-spec.
CC:no-cache was omitted from the HTTP spec apparently on grounds that
changing its caching effects on authentication would come as a surprise.
The actual operation is safe enough to use when parameterless no-cache
is treated strictly as an alias for must-revalidate (as done by Squid now).
Alex Rousskov [Mon, 26 Nov 2012 18:54:43 +0000 (11:54 -0700)]
Make it possible to match empty header field values using req_header and rep_header.
Warning: Some req_header and rep_header ACLs that were [accidentally] not
matching empty headers (e.g., "^$" or ".*") will now start matching them.
A new HttpHeader::getByNameIfPresent() method is added to be able to detect
presence of empty header fields while ACLHTTPHeaderData::match() is adjusted
to convert undefined String values into empty c-strings ("") for
ACLRegexData::match() to work.
Prior to these changes, when trying to match an empty header value with a
regex like "^$", ACLHTTPHeaderData::match() would return false because:
* HttpHeader::getStrOrList() and getByName() return an undefined String.
* String::termedBuf() returns NULL for undefined Strings; and
* ACLRegexData::match() always fails on NULL c-strings.
Amos Jeffries [Sat, 24 Nov 2012 14:30:02 +0000 (03:30 +1300)]
Fix helper reply length detection
Stateless helper reply handler was incrementing 't' position before
passing the details of response length to HelperReply parser.
Stateful helper reply handler was pointing at position of \n insteaf of
position of \r when \r\n received.
Both of these cause the response length parameter sent to the reply
parser to be longer than the actual response characters length. Breaking
parse on responses with just result code followed by \n since the string
comparison EOL is now tested against input length 2/3/4.
Amos Jeffries [Sun, 18 Nov 2012 10:37:19 +0000 (03:37 -0700)]
Fix several buffer termination bugs
* strcpy() replaced in several places with strncpy() to ensure destination
buffers are not overflowed.
* strncpy() does not nul-terminate the destination when the string being
copied in exactly fills the buffer. Ensure we have terminated strings
where it may matter.
Detected by Coverity Scan. Issues 740309, 740310, 740311, 740481, 740483
Amos Jeffries [Sun, 18 Nov 2012 10:02:38 +0000 (03:02 -0700)]
Remove MemPoolChunked::memPID
This member variable appears to have been missed when MemPool was split
into generic framework and specific Chunked implementation.
(rev:10513.1.1 aka trunnk rev:10517)
The memPID and its maintenance code was moved into MemImplementingAllocator
but this definitio left here un-initialized and shadowing the framework
member.
Amos Jeffries [Sun, 18 Nov 2012 07:06:26 +0000 (00:06 -0700)]
negotiate_kerberos_auth: better bounds checking
* sysconf() may return -N values on some platforms or values larger than
the hard-coded 1024 buffer size for hostname. Use sizeof() instead
since the buffer is hardcoded anyway.
* also, use return instead of exit() on the test binary to reduce
warnings from static analysis compilers.
Amos Jeffries [Sat, 17 Nov 2012 12:58:59 +0000 (05:58 -0700)]
Various memory leaks in configuration parsing
This lot are all small issues derived from allocating new memory and
assigning to a pointer already pointing at previous allocation, or
passing xstrdup() output to a caller which does not directly hold the
passed memory.
Both cases will disappear once we clean up teh string handlign in Squid
but for now these still need fixing to avoid leaking memory.
Detected by Coverity Scan. Issues 740430, 740432, 740439.
Amos Jeffries [Fri, 16 Nov 2012 06:45:56 +0000 (23:45 -0700)]
Fix various assertion with side effects
When compiled with high optimization and assert disabled these operations
would have disappeared. The side effects being:
* Disk I/O failure protection disabled. Allowing loops in diskd write.
* squidpurge error handling on command line parse gone. Causing segfault.
* squidpurge 'I am Alive' ticker feature cease working.
Detected by Coverity Scan. Issues 740299, 740300, 740301, 740302, 740303
projects / thirdparty / squid.git / log
