Bug 3405: ssl_crtd crashes failing to remove certificate
- Try to update the index file in all cases the database modified
rows. Currently we are using the new operator.
- The find operator in database should not modify the database. Currently
if an entry is expired, ssl_crtd removes the cert file but does not
update the index file.
- Fix a small memory leak when remove entries from database: A row object
removed from TXT_DB indexes but never released.
This patch:
* Use OPENSSL_malloc and OPENSSL_free to allocate/release memory for
TXT_DB rows. OpenSSL SDK assumes that always allocated using these
functions.
* Add code in Ssl::CertificateDb::Row destructor to correctly release
a TXT_DB row.
* Add the sq_TXT_DB_delete and sq_TXT_DB_delete_row functions which
removes a row from TXT_DB indexes.
Amos Jeffries [Tue, 6 Nov 2012 12:45:17 +0000 (05:45 -0700)]
Bug 3677: Port un-pinning logic changes from 3.3
Several design changes have been made to pinning in 3.3. One of them
added a safety check to proect close handler removal in the event that
the server connection was already closed.
This ports that protection from 3.3 and fixes at least one code path
leading to bug 3677 assertions.
Amos Jeffries [Sun, 28 Oct 2012 09:54:40 +0000 (03:54 -0600)]
Remove 'Bungled' warning on missing component directives
* Make Squid display a ERROR notice and ignore squid.conf directives
which belong to components disabled at build time instead of throwing
up 'FATAL: Bungled' message and aborting.
* Polish the 'requires X' message text for better display of cases like
'requires MS Windows' and 'requires Linux'
* Add 'requires X' message for --enable-auth, --enable-eui, and
--enable-ssl-crtd which were missing.
* Add debug 'filename' for if-none and postscriptum logging
Amos Jeffries [Sat, 20 Oct 2012 12:32:21 +0000 (06:32 -0600)]
Remove the refresh_pattern ignore-no-cache option
With the HTTP/1.1 Cache-Control:no-cache compliance upgrade this option became
obsolete. Its main use-case was to force caching of responses which contained this
control. Revalidation is now happening and no-cache objects are cached by default.
The one remaining effect of ignoring no-cache is to prevent no-cache from allowing
authenticated responses being cached. Which is the opposite of its usually expected
result and allowing that change of behaviour would be counter productive.
Amos Jeffries [Sat, 20 Oct 2012 10:07:11 +0000 (04:07 -0600)]
HTTP/1.1: Cache-Control compliance upgrade
trunk rev 11361 converted Cache-Control header from using a single mask
bitmap (shared by request and response) to separate CC header objects in
the request response. This conversion contained several regressions like
the one bug 3670 reports.
This patch:
* documents HttpStateData::cacheableReply() clarifying the overall
method action and what each individual check it doing.
* resolves several visible regressions, including bug 3670.
* extends the caching to handle the "no-cache" controls as per HTTP/1.1
(MAY store, but MUST revalidate before use).
* extends the caching for several lesser known cases of "MAY store"
exemptions handling authenticated transactions.
* removes an old hack handling Pragme:no-cache (undefined in HTTP/1.1)
One side effect of now caching transactions utilizing "no-cache" is that
hacks around Pragma:no-cache are reduced to only having any effect when
Cache-Control is absent. Reducing their performance cost. And now require
--enable-http-violations is built in since HTTP/1.1 states that
response Pragma are not defined and thus SHOULD be ignored.
Wolfgang Breyha [Thu, 18 Oct 2012 04:23:10 +0000 (22:23 -0600)]
Bug 3655: pinning failure breaks NTLM and Negotiate authentication
Some additional bits for squid-3.2 ensuring that already pinned connections
are cleaned up properly before re-pinning. Also that the pointer is not
NULL before closing when un-pinning.
Alex Rousskov [Wed, 17 Oct 2012 01:02:52 +0000 (19:02 -0600)]
Allow a ufs cache_dir entry to coexist with a shared memory cache entry
... instead of being released when it becomes idle.
The original boolean version of the StoreController::dereference() code
(r11730) was written to make sure that idle unlocked local store_table entries
are released if nobody needs them (to avoid creating inconsistencies with
shared caches that could be modified in a different process).
Then, in r11786, we realized that the original code was destroying non-shared
memory cache entries if there were no cache_dirs to vote for keeping them in
store_table. I fixed that by changing the StoreController::dereference() logic
from "remove if nobody needs it" to "remove if somebody objects to keeping
it". That solved the problem at hand, but prohibited an entry to exist in
a non-shared cache_dir and in a shared memory cache at the same time.
We now go back to the original "remove if nobody needs it" design but also
give non-shared memory cache a vote so that it can protect idle but suitable
for memory cache entries from being released if there are no cache_dirs to
vote for them.
This is a second revision of the fix. The first one (r12231) was reverted
because it did not pass tests/testRock unit tests on some platforms. The unit
tests assume that the entry slot is not locked after the entry is stored, but
the first revision of the fix allowed idle entries to remain in store_table
and, hence, their slots were locked and could not be replaced, causing
assertions. This revision allows the idle entry to be destroyed (and its slot
unlocked) if [non-shared] memory caching is disabled.
It is not clear why only some of the platforms were affected by this. Should
not memory caching be disabled everywhere during testRock (because testRock
does not set memory cache capacity and memory cache entry size limits)?
Amos Jeffries [Wed, 17 Oct 2012 00:43:57 +0000 (18:43 -0600)]
HTTP/1.1: honour Cache-Control before Pragma:no-cache
HTTPbis WG clarifications on the interaction between Cache-Control and
Pragma specify that Cache-Control may be explicitly crafted to allow
conditional caching by HTTP/1.1 caches and Pragma:no-cache supplied to
prevent caching by older HTTP/1.0 caches unable to understand the
Cache-Control properly.
Squid does obey Cache-Control. So we can deprecate Pragma to the side
cases when Cache-Control is absent entirely.
Alex Rousskov [Wed, 17 Oct 2012 00:02:26 +0000 (18:02 -0600)]
Reverted trunk r12255 changes. Provided a portable flexible arrays replacement.
Trunk r12255 made Clang compiler happy by removing flexible nonPod[] arrays.
Unfortunately, it also moved shared memory items into local memory (in some
cases uninitialized).
This change provides a Clang-friendly flexible array replacement while keeping
items in the shared memory (and using placement-new initialization). The code
may have become even less readable, but hopefully more portable.
N.B. Flexible arrays were introdiced in C99 standard, after C++ was
standardized in 1998. They are not yet in any revised C++ standard, but most
C++ compilers support them, at least for PODs.
Alex Rousskov [Sun, 14 Oct 2012 00:13:18 +0000 (18:13 -0600)]
Do not check 'reply content_length is too big' condition twice.
Saves a few CPU cycles. No visible runtime effects are expected.
The semi-duplicate check was introduced in trunk r4410 while trying to
prevent huge objects from being swapped out. It is not clear to me why
that change helped at the time (but it could have something to do with
negative content_length values and their effect on then-current size
types).
Amos Jeffries [Sat, 6 Oct 2012 02:26:32 +0000 (20:26 -0600)]
Fix website config display of default settings
The website page cfgman builder script was omitted when adding the
DEFAULT_DOC parameter and making DEFAULT / DEFAULT_IF_NONE support
multi-line values.
Amos Jeffries [Sat, 6 Oct 2012 02:18:46 +0000 (20:18 -0600)]
Small memory leak in src ACL parse
Fixes one minor memory leak when IPv6 is disabled and parsing an IPv6
address. For example the default localhost ACL ::1 value.
Caught by Valgrind:
==26647== 384 bytes in 4 blocks are definitely lost in loss record 1,132 of 1,726
==26647== at 0x4C25A28: calloc (vg_replace_malloc.c:467)
==26647== by 0x65B441: xcalloc (xalloc.cc:75)
==26647== by 0x657B99: MemPoolMalloc::allocate() (MemPoolMalloc.cc:62)
==26647== by 0x5A95B1: acl_ip_data::FactoryParse(char const*) (Ip.h:66)
==26647== by 0x5AA8BD: ACLIP::parse() (Ip.cc:523)
==26647== by 0x5E0A80: ACL::ParseAclLine(ConfigParser&, ACL**) (Acl.cc:174)
==26647== by 0x4B0C0F: parse_line(char*) (cache_cf.cc:1252)
==26647== by 0x4B2076: parseOneConfigFile(char const*, unsigned int) (cache_cf.cc:518)
==26647== by 0x4B29D0: parseConfigFile(char const*) (cache_cf.cc:558)
==26647== by 0x546B81: SquidMain(int, char**) (main.cc:1372)
==26647== by 0x547445: main (main.cc:1215)
Regression fix: Handle dstdomain duplicates and overlapping names better
Since 3.2 changes to dstdomain overlap detection teh case of duplicate
wildcards has become a fatal error needlessly.
This silently ignores all exact duplicates, even if they are wildcards.
Also, adjust the message display to always display the longer of the
domains first. Since we are dealing with sub-domains it is the most
reliable indicator of which should be removed to safely fix the detected
issue.
Alex Rousskov [Sat, 22 Sep 2012 03:12:33 +0000 (21:12 -0600)]
Do not reuse persistent connections for PUTs to avoid ERR_ZERO_SIZE_OBJECT.
A compliant proxy may retry PUTs, but Squid lacks the [rather complicated]
code required to protect the PUT request body from being nibbled during the
first try or [also tricky] code to send 100-continue expectation requiredto
delay body sending. Thus, Squid cannot safely retry some PUTs today, and
FwdState::checkRetriable() must return false for all PUTs, to avoid
bogus ERR_ZERO_SIZE_OBJECT errors (especially for clients that did not
reuse a pconn and, hence, may not be ready to handle/retry an error response).
In theory, requests with safe or idempotent methods other than PUT might have
bodies so we apply the same logic to them as well.
This reopens Squid bug #3398, undoing trunk r11859 commit which attempted
to close that bug.
Alex Rousskov [Sat, 22 Sep 2012 03:11:24 +0000 (21:11 -0600)]
Do not chunk responses carrying a Content-Range header.
When Squid forwards a response with a Content-Range header,
ClientSocketContext::socketState() detects the end of the response range(s)
and returns STREAM_*COMPLETE to ClientSocketContext::writeComplete().
The latter thinks that the writing of the response to the client must be
over and calls keepaliveNextRequest() instead of writing the last-chunk
(if any). If the to-client response was chunked, the client gets stuck
waiting for that missing last-chunk.
The multipart Range request case was already excluded from chunking (or it
would probably suffer from the same problem). With this change, no
Content-Range responses will be chunked.
N.B. Some servers send Content-Range responses to basic GET requests
without a Range header, so the problem affects more than just Range requests.
TODO: A proper fix would be to rewrite ClientSocketContext::writeComplete()
and other code so that it does not mix internal ClientStream completion with
[possibly chunk-encoded] writing completion. This should probably be done
along with fixing ClientSocketContext::socketState() and other state-checking
code to ignore to-client persistence (flags.proxy_keepalive), which is not
related to the internal ClientStream state.
Changes to interception handling in 3.2 series (namely the preference
for using ORIGINAL_DST) have increased the chances that misconfigured
network systems involving Squid will hit forwarding loops.
Two instances are currently known:
* passing forward-proxy traffic to a interception port.
* NAT performed on a separate box.
This enacts an old TODO by removing the loop detection bypass for
intercepted traffic and accelerated traffic. Now we always check for
loops regardless of how the request was received.
NOTE: accel mode was only included due to the TODO.
If problems are found there it can be re-instated.
According to CPU_SET(3) and, apparently, on some systems (e.g.,
OpenSuSE 10.3) CPU_COUNT macro expects a non-const argument. The
patch fixes build error on these systems.
Bug 3616: Retrieve client connection for ACL checks from the related HttpRequest object
This patch enable SSL client certificate ACL checks (user_cert and ca_cert)
in all cases the client connection information can retrieved from the related
HttpRequest object, eg when making peering decisions (peer_cache_access ACL).
Also fixes assertion failure caused by acl dst on transparent proxy.
Discussed under the "Supply client connection and IDENT information to
peer_cache_access ACL check" thread on squid-dev.
Amos Jeffries [Wed, 29 Aug 2012 03:36:29 +0000 (21:36 -0600)]
Regression: login=PASS send no credentials when none available.
login=PASS should act like PASSTHRU, sending no credentials header, when
no client supplied OR external ACL credentials are available.
3.2 has been found wrongly adding the username "PASS" in this case.
clang++ doesn't support c++ variable arrays for non-pod types.
Change variable arrays to dynamically-allocated arrays, Ipc::QueueReaders,
Ipc::StoreMap and Ipc::Mem::PageStack
Amos Jeffries [Thu, 2 Aug 2012 03:31:52 +0000 (21:31 -0600)]
Support -DFAILURE_MODE_TIME=n compiler flag
This value determins at compile-time how long Squid spends in HIT-only
mode after the failure ratio goes over 1.0. see checkFailureRatio() in
src/client_side_request.cc for details on the ratio.
This flag is supported to remove the need for patching when alteration
is required.
This re-enables Squid peer selection algorithms for intercepted
traffic which has failed Host header verification.
When host verification fails Squid will use, in order of preference:
* an already PINNED server connection
* the client ORIGINAL_DST details
* cache_peer as chosen by selection algorithms
NOTE: whenever DIRECT is selected by routing algorithms the
ORIGINAL_DST is used instead.
Peer selection results are updated to display PINNED and
ORIGINAL_DST alongside DIRECT and cache_peer.
SECURITY NOTE:
At this point Squid will pass the request to cache_peer using the
non-trusted Host header in their URLs. Meaning that the peers
may still be poisoned by CVE-2009-0801 attacks. Only the initial
intercepting proxy is protected.
Full protection against CVE-2009-0801 can be enjoyed by building
Squid with the -DSTRICT_HOST_VERIFY compile-time flag. This will
make the peers unreachable for intercepted traffic where the
Host verification has failed.
The AccessLogEntry objects currently are only members of the ClientHttpRequest
objects. There are cases where we need to access AccessLogEntry from server
side objects to retrieve already stored informations for the client request and use it in
server side code with format/* interface (eg use Format::Format::assemble
inside http.cc)
This patch convert AccessLogEntry class to RefCountable to allow link it with
other than the ClientHttpRequest objects.
When '-k parse' is used deprecation notices and upgrade help messages etc
need to be bumped consistently up to level-0 and this macro will help
reducing the (?:) code mistakes.
Alex Rousskov [Sat, 21 Jul 2012 01:28:42 +0000 (19:28 -0600)]
Avoid bogus "Disk space over limit" warnings when rebuidling dirty ufs index.
Subtract sizes of added-then-rejected entries while loading ufs cache index.
Before SMP changes, the ufs code was incorrectly ignoring the size of
loaded-but-not-yet-validated entries, leading to cache disk overflows.
After SMP changes, the ufs code was not subtracting sizes of
loaded-but-then-rejected entries, leading to bogus "Disk space over
limit" warnings. Now we correctly account for both kinds of entries.
Bug 3478: Partial fix: Connection-auth on intercepted connections is broken
Currenty in the case of intercepted connections each request is open a new
connection to the destination server, even if the connection is a valid pinned
connection.
This patch fixes this problem and reuses pinned connections on intercepted
requests.
Fail with an explanation instead of asserting. The assertion fails when a ufs
cache_dir's swap.state has inconsistencies AND the user starts Squid with a -S
command line option. Normally, such inconsistencies are ignored and many of
them are benign. For example, a missing cache file with an ADD record in
swap.state is such an inconsistency.
The -S option was meant to help developers troubleshoot inconsistencies by
analyzing core dumps, but (a) admins treat assertions as Squid bugs and file
bug reports and (b) in most cases, it is really difficult to find the
inconsistency when Squid asserts after detecting all of them (and leaving the
detection context).
We now explicitly tell the admin what their options are and quit instead of
asserting.
TODO: Consider adding a ufs cache_dir option that checks for and removes
inconsistencies instead of not checking at all (default) or checking and
quitting (-S). This is difficult because some valid cache entries may look
inconsistent while they are being updated and some invalid cache entries
are not visible to Squid without a full directory scan.
Alex Rousskov [Sat, 21 Jul 2012 01:14:53 +0000 (19:14 -0600)]
Bug 3525: Do not resend nibbled PUTs and avoid "mustAutoConsume" assertion.
The connected_okay flag was not set when we were reusing the connection.
The unset flag overwrote bodyNibbled() check, allowing FwdState to retry
a PUT after the failed transaction consumed some of the request body
buffer, triggering BodyPipe.cc:239: "mustAutoConsume" assertion.
We now set the connected_okay flag as soon as we get a usable connection
and do not allow it to overwrite the bodyNibbled() check (just in case).
Restore memory caching ability lost since r11969.
Honor maximum_object_size_in_memory for non-shared memory cache.
Since r11969, Squid calls trimMemory() for all entries to release unused
MemObjects memory. Unfortunately, that revision also started trimming objects
that could have been stored in memory cache later. Trimmed objects are no
longer eligible for memory caching.
It is possible that IN_MEMORY check in StoreEntry::trimMemory() was preventing
excessive trimming in the past, but after SMP changes, IN_MEMORY flag is set
only after the object is committed to a memory cache in
StoreController::handleIdleEntry(), so we cannot rely on it. For example:
StoreEntry::unlock()
StoreController::handleIdleEntry() // never get here because entry is
set IN_MEMORY status // already marked for release
This change adds StoreController::keepForLocalMemoryCache() and
MemStore::keepInLocalMemory() methods to check if an entry could be stored in
a memory cache (and, hence, should not be trimmed). The store-related part of
the trimming logic has been moved from StoreEntry::trimMemory() to
StoreController::maybeTrimMemory(). StoreEntry code now focuses on checks that
are not specific to any memory cache settings or admission policies.
These changes may resulted in Squid honoring maximum_object_size_in_memory for
non-shared memory cache. We may have been ignoring that option for non-shared
memory caches since SMP changes because the check for it was specific to a
shared memory cache.
Bug 3577: File Descriptors not properly closed in trunk r12185.
Bug 3583: Server connection stuck after TCP_REFRESH_UNMODIFIED.
These changes fix FD leaks and stuck connections under two conditions:
1) Client aborts while Squid's server-side establishes a connection
Bug 3577: When a client quits while ConnOpener is trying to open the
connection to the next hop, FwdState cancels its ConnOpener callback.
ConnOpener notices that when trying to connect again and quits before
establishing a connection. The ConnOpener cleanup code did not close the
temporary FD used for establishing the connection. It did call fd_close(),
but fd_close() does not close the FD, naturally.
ConnOpener was probably leaking the temporary FD in other error handling
cases as well. It was never closed unless the connection was successful.
2) Client aborts after Squid's server-side established a connection:
Bug 3583: When a client aborts the store entry after receiving an HTTP 304 Not
Modified reply in response to a cache refreshing IMS request, HttpStateData
notices an aborted Store entry (after writing the reply to store), but does
virtually nothing, often resulting in a stuck server connection, leaking a
descriptor. Now we abort the server-side transaction in this case.
Bug 3577: Similarly, when a client disconnects after Squid started talking to
the origin server but before Squid received a [complete] server response,
HttpStateData notices an aborted Store entry (during the next read from the
origin server), but does virtually nothing, often resulting in a stuck server
connection, leaking a descriptor. Now we abort the server-side transaction in
this case.
FwdState now also closes the server-side connection, if any, when the client
aborts the store entry and FwdState::abort() callback is called. This helps
reduce the number of concurrent server-side connections when clients abort
connections rapidly as Squid no longer has to wait for the server-side I/O to
notice that the entry is gone. The code to close the connection was temporary
removed in trunk r10057.1.51.
Alex Rousskov [Fri, 13 Jul 2012 11:53:18 +0000 (05:53 -0600)]
[request|reply]_header_* manglers fixes to handle custom headers
This patch fix the [request|reply]_header_[access|replace] configuration
parameters to support custom headers. Before this patch the user was able
to remove/replace/allow all custom headers using the "Other" as header name.
projects / thirdparty / squid.git / log
