This patch investigates the new note ACL type, to match transaction annotation.
Syntax:
acl aclname note name [value ...]
Without values, matches any annotation with a given name. With value(s), matches
any annotation with a given name that also has one of the given values.
Annotation sources include note and adaptation_meta directives as well as helper
and eCAP responses.
Alex Rousskov [Fri, 24 May 2013 15:14:50 +0000 (09:14 -0600)]
Ask for SSL key password when started with -N but without sslpassword_program.
Do not give SSL a password-asking callback if sslpassword_program is not
configured. Without a callback, OpenSSL itself asks for the password (which
works if Squid runs in foreground because of -N).
The fix applies to Ssl::readCertChainAndPrivateKeyFromFiles() context only.
This is not the only place where we read private keys. Some other places are
working correctly, but others may need more work. Also,
Ssl::readCertChainAndPrivateKeyFromFiles() may not really work if
sslpassword_program _is_ configured because "user data" pointer will be nil.
Cleanup: Merge AccessLogEntry 'helperNotes' and 'configNotes' members to 'notes' member
There is not any need to store notes added using Note cfg option and notes added
from helper to separated member. This patch merge them to the same
AccessLogEntry::note member.
Amos Jeffries [Wed, 22 May 2013 06:33:05 +0000 (00:33 -0600)]
Add pt-bz (Belize Portuguee) dialect to translations
In absence of information on whether Belize Portuguese dialect can be
presented with Brazillian or European Portuguese texts or whether they
require a third translation use a symlink entry for pt-bz to pt-br based
on the geographical distance for dialect locale alternatives.
Alex Rousskov [Wed, 22 May 2013 01:04:34 +0000 (19:04 -0600)]
Honor zero-padded logformat codes again.
Trunk r12628 "Turn flags to bool" set Token::zero to false instead of true,
resulting in extra spaces occasionally logged in front of such popular and
default fields as %03tu, screwing up log parsers.
Amos Jeffries [Tue, 21 May 2013 05:39:18 +0000 (23:39 -0600)]
Detect and use -march=natuve when possible
Clang++ 3.2 fails to detect some CPUs correctly and requires the
additional checks enabled by this option to build working executables.
This option supported by GCC 4.3 and later enables additional CPU
detection and enables CPU-specific optimizations. In the interests of
better performance this patch enables it whenever is it available and
possible to use (cross-compilers cannot use it).
Ming Fu [Tue, 21 May 2013 05:17:23 +0000 (23:17 -0600)]
Bug 1991: kqueue causes SSL to hang
Compare the code in normal select and epoll v.s. kqueue. The select use a 0
wait time to get out of select wait in order to handle a list of read_pendings.
However, epoll add the read_pending to read and write event monitor. At a first
look, this seems strange as why read pending has anything to do with write. It
became obvious when the write ready event is triggered. During a write ready
event, if read_pending is on, the read callback is called before the write
callback. As the write buffer is unlikely to be full for an extended period, a
write callback is guaranteed in the immediate future for the read_pending
socket by waiting on write.
The patch follows that same logic as epoll and applies it on kqueue.
The code initializing default values and determining helper detection
was significantly different from other helpers detection logics. Combined
with rev.12786 logic changes resulted in the detection always failing.
Update the --enable-log-deemon-heleprs option to work identically to the
--enable-external-acl-helpers option.
Silamael [Fri, 17 May 2013 08:36:45 +0000 (02:36 -0600)]
Add missing piece omitted from rev.9677.
rev.9677 created forward_max_tries direcive but omitted one of the
checks. This adds that check and allows forward_max_tries to be set
to values greater than 10.
Bug 3744: squid terminated: FATAL: Bungled (null) line 3: sslproxy_cert_sign signTrusted all
This bug is a Makefile dependencies problem.
- The cf_gen includes the cf_gen_defines.cci so this file should included in
cf_gen dependencies.
- Currently the cf_gen_defines.cci exist in cf_gen.$(OBJEXT) dependencies but
does not have any effect because the obj file never build and used.
- Also the cf_gen_defines.cci file depends on autoconf.h so this file should
added to to cf_gen_defines.cc dependencies.
All of the sources has the autoconf.h file in their dependencies.
But the cf_gen_defines.cci is auto-generated and does not exist when the
dependencies computed.
Change the libraries order in LIBS variable inside SQUID_CHECK_OPENSSL_GETCERTIFICATE_WORKS check.
Looks that play a role in some cases (when openssl provided only as static
library in my tests).
Bug 3759: OpenSSL compilation error on stock Fedora17, RHEL, CentOS 6 systems
OpenSSL-1.0.x has changes in TXT_DB interface over the earlier openSSL releases.
Also looks that the IMPLEMENT_LHASH_* macros are not correctly implemented and
causes compile failures.
Some of the linux distributions to overcome the above problems trying to patch
openSSL SDK. For squid this is means that the current checks based on openSSL
version can not work.
This patch try to detect at configure time:
- if the TXT_DB uses the new implementation investigated in openSSL-1.0.x
releases
- If the IMPLEMENT_LHASH_* openSSL macros are correctly implemented.
Then uses the autoconf defines to implement the correct workarounds for used
openSSL SDK.
This patch try to avoid using the SSL_get_certificate function. While configures
squid run tests:
- to examine if the workaround code can be used
- to detect buggy SSL_get_certificate
Inside Ssl::verifySslCertificate try to use workarround code and if this is not
possible uses the SSL_get_certificate if it is not buggy, else hit an assertion
Amos Jeffries [Tue, 14 May 2013 15:31:15 +0000 (09:31 -0600)]
Log an ERROR instead of halting on unknown cache_dir types
Squid-3 can run fine without any configured cache_dir. This assists with
upgrade from older Squid-2 where COSS or NULL cache types may be present.
It also assists with backward compatibility for any future cache types
which may be added in future.
Amos Jeffries [Mon, 13 May 2013 16:21:23 +0000 (10:21 -0600)]
Remove origin_tries limiter on forwarding
This limit seems to have been set to prevent large amount of looping when
DIRECT attempts fail under the old model of constant DNS lookups and
retries.
However it is hard-coded and has no configuration knob visible. Under
the curent model of all destinations being enumerated once and tried
sequentially this protection would seem to be no longer necessary and
somewhat harmful as it will be preventing retries reaching destinations
with more than 2 unreachable IPs (think 3 IPv6 and an IPv4 on IPv4-only
network).
Alex Rousskov [Mon, 13 May 2013 14:07:55 +0000 (08:07 -0600)]
Fixed leaking configurable SSL error details.
Trunk r11496 "Configurable SSL error details messages" correctly disabled
collection of HTTP statistics for non-HTTP header fields, such as configurable
SSL error details. However, it also incorrectly disabled deletion of those
non-HTTP header fields.
Configurable SSL error details are only created during [re]configuration time,
so the leak went unnoticed since 2011-06-17, but the same bug caused a major
runtime annotation leak later (r12413) until the new annotation code was
redesigned to avoid using HttpHeader (r12779).
Steve Hill [Mon, 13 May 2013 03:57:03 +0000 (21:57 -0600)]
Add spoof_client_ip access control
... to control whether TPROXY requests have their source IP address
spoofed by Squid. The ACL defaults to allow (i.e. the current
behaviour), but using an ACL that results in a deny result will
disable spoofing for that request.
Example config to disable spoofing for all requests:
spoof_client_ip deny all
Also, polish the TPROXY related flags:
1. The flags.spoofClientIp flag was a general "this is a TPROXY
request" flag, which was a bit confusing given the name of the
flag. So the flags.spoofClientIp flag now only indicates
whether we want to spoof the source IP or not.
2. The flags.interceptTproxy is added to flag whether the request
was received on a "tproxy" port or not.
Nathan Hoad [Sat, 11 May 2013 20:59:44 +0000 (14:59 -0600)]
Bug 3389: Auto-reconnect for tcp access_log.
Major changes:
1. Squid reconnects to TCP logger as needed. Squid keeps trying to connect
forever, using a hard-coded 0.5 second delay between attempts.
2. Squid buffers log records while there is no connectivity. The buffering
limit is configurable.
3. On buffer overflows, Squid worker either dies or starts dropping log
records. The choice is configurable.
4. The tcp logging module honors buffered_logs setting. Old code was flushing
each record.
5. Squid reports changes in logging state to cache.log. Except for every 100th
consecutive connection failures, routine connection retries are not reported
at level 1, to reduce noise level.
6. A new access_log configuration format/style has been added. It allows us to
easily add named options such as buffer-size or on-error. The same format can
be used to add module-specific options in the future, but doing so would
require changes to the high-level logging code. All old configuration
formats/styles are still supported.
7. squid.conf buffered_log option documentation now reflects reality. It used
to talk about cache.log but I do not think Squid uses that option for
cache.log maintenance.
Known minor side-effects of these changes:
i) All access_log logs can now be configured to bypass errors because the old
"fatal" flag is now configurable via log-specific on-error option in
squid.conf. The default is still "die". I have not checked whether modules
other than TCP logger honor that flag.
ii) All access_log logs now use 8*MAX_URL (64KB) instead of a 4*MAX_URL (32KB)
or smaller buffer size by default. The ICAP logger was using 2*MAX_URL buffer
size. The TCP logger was using 64KB buffer size before so no change for TCP. I
decided that it is better to raise the default buffering level for some logs
rather than decrease it for other logs, but it is not clear what the best
default is. The buffer size is now configurable via buffer-size so admins can
control it on individual log basis.
iii) Some access_log configuration styles overlap. To resolve ambiguities,
Squid may need to assume that the first logging ACL name (if any) does not
contain '=' and is not equal to an existing logformat name. It is possible to
use 'all' as the first ACL name if these heuristics cause problems.
TODO: We have attempted to solve more TCP logging problems, but it turns out
that correct solutions would require fixing higher-level logging code, not
specific to TCP logger or Bug 3389 scope. Those unsolved problems include:
A. During reconfiguration, all logs are closed and reopened, even if there
have been no changes to their configuration that necessitate such a drastic
action (or no changes at all!). For TCP logger, this means that the old
connection is used to flush remaining buffered records (if any), and the new
connection is used to log new records, possibly at the same time. Nathan wrote
clever code that keeps logging going using the same job/connection. However,
we had to yank that code out because it clashed with higher-level logging
state in subtle ways.
B. During shutdown, all connections are put in the closing state before logs
are told to flush remaining records. For TCP logger, this means that the
remaining buffered records (if any) are lost. The correct fix may require
rearranging shutdown sequence AND letting EventLoop run during shutdown (among
other things).
C. When logger connectivity is lost, Squid does not notice the problem until
the second TCP socket write (or later). This results in lost records. This is
due to TCP-level buffering. I suspect the only cure for this is adding
logger-to-Squid "I got your records" feedback, which requires changes in the
logging protocol (currently there is no logger-to-Squid communication at all).
Alex Rousskov [Sat, 11 May 2013 15:23:59 +0000 (09:23 -0600)]
Avoid !closing assertions when helpers call comm_read [during reconfigure].
While helper reading code does check for COMM_ERR_CLOSING, it is not sufficient
because helperReturnBuffer() called by the reading code may notice the helper
shutdown flag (set earlier by reconfigure) and start closing the connection
underneath the reading code feet.
Amos Jeffries [Wed, 8 May 2013 03:02:01 +0000 (21:02 -0600)]
Polish client-side: make TcpAcceptor fill out EUI details
The scope of TcpAcceptor is to receive new TCP connections and to fill
out the state data about the new connectino as much as possible.
EUI details about the client come under that state management.
This change makes EUI details available about https_port and FTP data
connections.
Amos Jeffries [Sun, 5 May 2013 08:38:06 +0000 (02:38 -0600)]
Fix parsing error in rev.12620
The ConfigParser::strtokFile() methods added by rev.12620 use
an undo mechanism to pull the next token if anything has been
undone.
The new parseFlags() method also added in that revision uses
the undo mechanism for any token which is not a flag token
it also returns the token in its nextToken parameter.
In ACLIP the data parser uses parseFlags() and the nextToken
received then goes on to use strtokFile() and the undo copy
of that same token. The result being that all dst/src ACLs
had a double-entry for the first IP address tokens.
Since IP address tokens are de-duplicated with loud WARNING
on -k parse this was resulting in confusing noise from the
built-in ACL definitions.
Since parseFlags() is so new and ACLIP was the only user of the
nextToken parameter we can drop it entirely and rely on only
ConfigParser undo mechanism from now on.
The stricter xato*() parsing bounds checks are halting on the ','
delimiters. Fix this by adding an optional end-of-value parameter to the
relevant parse functions and sending the delimiter in.
This fix makes xatoui() and xatoll() more friendly to parsing
unterminated strings.
previous to this fix the latter two values of tcpkeepalive= were
undocumented optional. This makes Squid enforce the documented format
where all three values are required if any is set.
Amos Jeffries [Sat, 4 May 2013 06:34:24 +0000 (00:34 -0600)]
autoconf: upgrade all helper config.test to M4 scriptlets
This carries on from the success of basic auth heleprs upgrade to perform
the update on all helpers bundled with Squid.
It also polishes some of the basic auth M4 macro tests a little.
TODO:
* Convert the duplicate code in the modules.m4 files into a macro
for simpler management.
* Update the tests. This patch replaces all config.test tests with an
equivalent autoconf macro. There are some helpers which could be
extended with better tests now.
Convert the basic_auth bundled helpers from using bash to m4 scripts
when testgn which helpers are able to be built.
This conversion allows us to leverage existing information found by
./configure in the helper test scritps, and to de-duplicate and simplify
many of the testing checks.
This required shuffling the SASL checks into a custom macro, which is
altered from a halting ERROR when SASL is absent to a softer WARNING,
but prevents the SASL helpers building - which the old checks did not
guarantee. Resolving Bug 3793.
The nature of m4 macros adds two small long term issues. Both
well-known issues with m4_include() expansion:
* the macro is expanded in-place without scoping separation.
So any brokenness in the child scripts will be a little harder to
identify and may result in breakage of modues other than the one whose
child script was the cause.
* the macro takes literal arguments without variable expansion.
So a "full" path relative to the main configure.ac script is required
in every include. Meaning the modules.m4 file cannot simply
sub-include using the $helper script variable, but must be a full
expansion of the modules set.
This patch try to fix current current Notes interface and usage.
The changes done having in mind that we need:
1) to add multiple notes with the same key
2) to support 3 different note types: adaptation meta headers, helper notes
and custom notes added by the system administrator
3) to log notes using the %note formating code
4) to use the %note formating code everywhere the formating API is used. For
example use the %note with the request_header_add configuration parameter.
5) to use notes with ACLs.
Details:
- The NotePairs class is not a kid of HttpHeader class anymore. It is
implemented from scratch to cover Helper/adaptation and custom notes needs.
* The new class stores key:value pairs in list. It allow multiple entries
with the same key.
* Includes a find method which return a coma separated list of values
for a given key
- The HttpRequest::helperNotes is now a Refcount of a HttpPairs object
- The HelperReply::notes is now a HttpPairs object
- The AccessLogEntry::notes now is a RefCount of a HttpPairs object, and
stores only the custom notes add by the "note" configuration parameter
- Add the AccessLogEntry::helperNotes which is a RefCount of a HttpPairs object
to store notes added by helpers.
Now the notes added by adaptation or helpers are accessible to format/* code
imediatelly after added. Before this patch are accessible only for logging.
Future work:
- Posible merge AccessLogEntry::notes and AccessLogEntry::helperNotes
- Performance fixes
Alex Rousskov [Wed, 24 Apr 2013 21:22:39 +0000 (15:22 -0600)]
Prevent external_acl.cc "inBackground" assertion on queue overloads.
The enqueue check for external ACL lookups was inconsistent with the final
queue length check in ExternalACLLookup::Start(). The former allowed adding to
the already full (but not yet overflowing) queue while the latter
rightfully(?) asserted that the queue should not overflow.
The code changes to prevent several useless allocations on missing
sslproxy_cert_error directive left a few lines out of place.
Shuffle cert.sslErrors setup back to the original code sequence and
define allowDomainMismatch early as false for the default handling.
Some systems like GNU Hurd provide teh mmap() API but lack MAP_NORESERVE
support. This option is an optimization, so we can define the macro
ourselves to nil and apparently not suffer (many) bad side effects.
Bug 3816: SSL_get_certificate call inside Ssl::verifySslCertificate crashes squi
d
The SSL_get_certificate implementation in OpenSSL 1.0.1d and 1.0.1e releases,
will crash if called before the certificate sent to the client.
This patch add a hack when one of the problematic OpenSSL versions used to
retrieve the certificate directly from SSL_CTX object, instead of creating
a temporary SSL object, and call SSL_get_certificate.
Docs: remove Squid prefix from ntlm_auth example paths
There is no longer a helper called ntlm_auth shipped with Squid and the
example configs are supposed to be referring to the Samba helper now
as demonstrated by the command line options.
/usr/bin/ is probably not the only location where Samba installs its
helper, but is easily recognised and better than prefixing with a custom
Squid-specific path from --prefix.
* Mostly adding DEFAULT_DOC directive to hide strange default values.
This wll help us move to different internal values for no-limit etc
at some point in the future.
* Migrates some access controls to "DEFAULT: none" instead of "deny all".
This reduces run-time CPU cycles running useless ACL tests.
NP: some access controls have been left unchanged due to complexity in
the code testing them (ie icap_access).
* added documentation for several directives which were missing it
* corrected buffered_logs documentation (text by Alex Rousskov)
* Updated cf_gen tool to produce more descriptive error messages.
Bug 3817: Memory leak in SSL cert validate for alt_name peer certs
Inside function Ssl::matchX509CommonNames which checks a domain name against
certificate common name and alternate names, if the domain matches any of the
alternate names the function return without releasing allocated data.
Current OpenBSD implementation of PF divert-to works similarly to TPROXY
and only requires a getsockname() lookup to locate the TCP packet
original destination.
We can use the same PF configuration to preform "intercept" option but
the old PF transparent code does lookups on /dev/pf which fails badly
on the new PF versions. getsockname() is what is really required and
already performed by TcpAcceptor on all incoming connections, so there
is no need for a special PF lookup code now.
Add a new ./configure option --with-nat-devpf to enable the old /dev/pf
NAT lookup code in a backward-compatible way for older OS versions and
OpenBSD based distros which have not yet ported the new PF code. The
option is disabled by default since the systems requiring it are fairly
old now.
Also remove the getsockname() lookup in the IPFW lookup implementation
which is redundant behind TcpAcceptor.
Michal Luscon [Tue, 16 Apr 2013 00:26:10 +0000 (18:26 -0600)]
Bug 3825: ncsa_auth segfaulting with glibc-2.17
It appears the crypt() function may return NULL strings. Check for those
before all strcmp() operations.
NOTE: The MD5 output checks are probably not needed but since SquidMD5 is
an object build-time switched between several encryption library API
definitions it is better to be safe here as well.
Polish: upgrade TunnelStateData to CBDATA_CLASS2()
CBDATA_CLASS2() removes the need to define new/delete operators and
removes soem uses of cbdataFree()/cbdataAlloc()
Also replaces several abuses of the cbdataInternal*() locking API with
CbcPointer<> auto-pointers. The existence of some of these locks needing
to be done is questionable since AsyncCalls scheduling protects better
against the 'deleted under our feets' problem.
For now the locks are retained since it is not yet easy to track down
which are safe and which are removable.
* convert the C-style tunnelStateFree() functio to a proper destructor.
* create a proper constructor for TunnelStateData
* include debugging for trace-job.pl to track tunnel jobs setup/teardown
projects / thirdparty / squid.git / log
