Handle Amos comments on bump-ssl-server-first patch
- Use "IF" instead of "IFDEF" in cf.data.pre. The IFDEF already used in the
IDEF<colon> form.
- Add an assertion in setServerBump to assure that even if this method reused in
a client request we will not try to change the existing bump mode
- Polish the code in doCallouts method: wrap if(!calloutContext->error) { ... }
around the whole section of callouts.
- The detailCode parameter in ErrorState::detailError(int detailCode) shadows
the ErrorState::detailCode parameter.
- src/url.cc: Reverting change in urlParseFinish
- Allow old "ssl_bump allow/deny acl ..." syntax converting:
* "ssl_bump allow" to "ssl_bump client-first"
* "ssl-bump deny" to "ssl_bump none"
Prints warning messages to urge users update their configuration.
Does not allow to mix old and new configurations.
Alex Rousskov [Wed, 18 Jul 2012 03:57:10 +0000 (21:57 -0600)]
Bug 3525: Do not resend nibbled PUTs and avoid "mustAutoConsume" assertion.
The connected_okay flag was not set when we were reusing the connection.
The unset flag overwrote bodyNibbled() check, allowing FwdState to retry
a PUT after the failed transaction consumed some of the request body
buffer, triggering BodyPipe.cc:239: "mustAutoConsume" assertion.
We now set the connected_okay flag as soon as we get a usable connection
and do not allow it to overwrite the bodyNibbled() check (just in case).
- Add request_header_add, a new ACL-driven squid.conf option that
allow addition of HTTP request header fields before the request is sent to
the next HTTP hop (a peer proxy or an origin server):
request_header_add <field-name> <field-value> acl1 [acl2]
where:
* Field-name is a token specifying an HTTP header name.
* Field-value is either a constant token or a quoted string containing
%macros. In theory, all of the logformat codes can be used as %macros.
However, unlike logging the transaction may not yet have enough
information to expand a macro when the new header value is needed.
The macro will be expanded into a single dash ('-') in such cases.
Not all macros have been tested.
* One or more Squid ACLs may be specified to restrict header insertion to
matching requests. The request_header_add option supports fast ACLs only.
- Add the %ssl::>cert_subject and %ssl::>cert_issuer logformating codes which
prints the Subject field and Issuer field of the received client SSL
certificate or a dash ('-').
The AccessLogEntry objects currently are only members of the ClientHttpRequest
objects. There are cases where we need to access AccessLogEntry from server
side objects to retrieve already stored informations for the client request and use it in
server side code with format/* interface (eg use Format::Format::assemble
inside http.cc)
This patch convert AccessLogEntry class to RefCountable to allow link it with
other than the ClientHttpRequest objects.
Restore memory caching ability lost since r11969.
Honor maximum_object_size_in_memory for non-shared memory cache.
Since r11969, Squid calls trimMemory() for all entries to release unused
MemObjects memory. Unfortunately, that revision also started trimming objects
that could have been stored in memory cache later. Trimmed objects are no
longer eligible for memory caching.
It is possible that IN_MEMORY check in StoreEntry::trimMemory() was preventing
excessive trimming in the past, but after SMP changes, IN_MEMORY flag is set
only after the object is committed to a memory cache in
StoreController::handleIdleEntry(), so we cannot rely on it. For example:
StoreEntry::unlock()
StoreController::handleIdleEntry() // never get here because entry is
set IN_MEMORY status // already marked for release
This change adds StoreController::keepForLocalMemoryCache() and
MemStore::keepInLocalMemory() methods to check if an entry could be stored in
a memory cache (and, hence, should not be trimmed). The store-related part of
the trimming logic has been moved from StoreEntry::trimMemory() to
StoreController::maybeTrimMemory(). StoreEntry code now focuses on checks that
are not specific to any memory cache settings or admission policies.
These changes may resulted in Squid honoring maximum_object_size_in_memory for
non-shared memory cache. We may have been ignoring that option for non-shared
memory caches since SMP changes because the check for it was specific to a
shared memory cache.
Bug 3577: File Descriptors not properly closed in trunk r12185.
Bug 3583: Server connection stuck after TCP_REFRESH_UNMODIFIED.
These changes fix FD leaks and stuck connections under two conditions:
1) Client aborts while Squid's server-side establishes a connection
Bug 3577: When a client quits while ConnOpener is trying to open the
connection to the next hop, FwdState cancels its ConnOpener callback.
ConnOpener notices that when trying to connect again and quits before
establishing a connection. The ConnOpener cleanup code did not close the
temporary FD used for establishing the connection. It did call fd_close(),
but fd_close() does not close the FD, naturally.
ConnOpener was probably leaking the temporary FD in other error handling
cases as well. It was never closed unless the connection was successful.
2) Client aborts after Squid's server-side established a connection:
Bug 3583: When a client aborts the store entry after receiving an HTTP 304 Not
Modified reply in response to a cache refreshing IMS request, HttpStateData
notices an aborted Store entry (after writing the reply to store), but does
virtually nothing, often resulting in a stuck server connection, leaking a
descriptor. Now we abort the server-side transaction in this case.
Bug 3577: Similarly, when a client disconnects after Squid started talking to
the origin server but before Squid received a [complete] server response,
HttpStateData notices an aborted Store entry (during the next read from the
origin server), but does virtually nothing, often resulting in a stuck server
connection, leaking a descriptor. Now we abort the server-side transaction in
this case.
FwdState now also closes the server-side connection, if any, when the client
aborts the store entry and FwdState::abort() callback is called. This helps
reduce the number of concurrent server-side connections when clients abort
connections rapidly as Squid no longer has to wait for the server-side I/O to
notice that the entry is gone. The code to close the connection was temporary
removed in trunk r10057.1.51.
Author: Alex Rousskov <rousskov@measurement-factory.com>
[request|reply]_header_* manglers fixes to handle custom headers
This patch fix the [request|reply]_header_[access|replace] configuration
parameters to support custom headers. Before this patch the user was able
to remove/replace/allow all custom headers using the "Other" as header name.
When '-k parse' is used deprecation notices and upgrade help messages etc
need to be bumped consistently up to level-0 and this macro will help
reducing the (?:) code mistakes.
Alex Rousskov [Sun, 1 Jul 2012 03:55:21 +0000 (21:55 -0600)]
Added ssl::bump_mode logformat code to log SslBump decisions.
Reshuffled (a little) how the bumping decision for CONNECT is made to
streamline the code. This was necessary to consistently distinguish '-' from
'none' logged modes.
Do not evaluate ssl_bump when we are going to respond to a CONNECT request
with an HTTP 407 (Proxy Authentication Required) or with a redirect response.
Such evaluation was pointless because the code never bumps in such cases
because, unlike regular errors, these responses cannot be delayed and served
later inside the bumped tunnel.
Alex Rousskov [Thu, 28 Jun 2012 18:26:44 +0000 (12:26 -0600)]
Removed FilledChecklist::checkCallback() as harmful and not needed.
The method was resetting the authentication state (auth_user_request) of a
connection just before notifying the caller of the async ACL check result. The
reset restarts authentication sequence, creating a 407 "loop" for auth schemes
that require more than one step (e.g., NTLM).
The method is no longer needed because we do not need to explicitly "unlock"
auth_user_request anymore. It is refcounted now.
There are other cases where connection authentication state () is reset from
within the ACL code. Some of those cases are not needed and some might even
cause similar bugs, but I did not risk misclassifying them in this commit.
Dmitry Kurochkin [Fri, 22 Jun 2012 03:49:38 +0000 (21:49 -0600)]
Fix build with GCC 4.7 (and probably other C++11 compilers).
User-defined literals introduced by C++11 break some previously valid
code, resulting in errors when building with GCC v4.7. For example:
error: unable to find string literal operator 'operator"" PRIu64'
In particular, whitespace is now needed after a string literal and
before something that could be a valid user-defined literal. See
"User-defined literals and whitespace" section at [1] for more details.
The patch adds spaces between string literals and macros.
Amos Jeffries [Tue, 19 Jun 2012 23:16:13 +0000 (11:16 +1200)]
Cleanup: disconnect Authentication and URL-rewrite callback handlers
The authentication handlers were for some reason using RH (rewrite helper)
callback typedef. But specifying it as a fatal error if the char*
parameter was used in auth.
Assign a new callback typedef AUTHCB for use by authentication callers.
This allows auth callers to use different parameters (none) and to avoid
possibly fatal mistakes when coding new auth modules.
Alex Rousskov [Mon, 18 Jun 2012 23:13:05 +0000 (17:13 -0600)]
Account for Store disk client quota when bandwidth-limiting the server.
It is not clear why the store client type matters when
MemObject::mostBytesAllowed() is trying to find the maximum delay pool
quota for reading from the next hop HTTP server. Whether the client(s)
are reading from disk or RAM, the corresponding server-side bandwidth
ought to be limited.
This code was removed as a part of bug 3462 investigation, but it is not
needed to fix bug 3462.
Julien Pinon [Mon, 18 Jun 2012 23:08:56 +0000 (17:08 -0600)]
Bug 3462: Delay Pools and ICAP
Allow StoreEntry::bytesWanted() API to ignore delay pools. Use that
feature when shoveling adapted response body from ICAP/eCAP BodyPipe
into Store.
If we do not ignore delay pools in
ServerStateData::handleMoreAdaptedBodyAvailable() context, and our pool
quota is exhausted, we will stop reading from the adaptation BodyPipe,
but there is no code/API to notify a BodyPipe reader when the quota
becomes available again. With no reads from the pipe, the ICAP service
stops sending more adapted body data and possibly stops reading the
virgin body data from Squid, resulting in a stuck ICAP RESPMOD and the
master HTTP transaction.
We do need to call StoreEntry::bytesWanted() in this context because
otherwise Squid may run out of RAM (Squid bug #2619). The fix for that
problem inadvertently created this bug when delay pools were enabled.
Long-term, a "kick BodyPipe consumer when there is quota" code should be
added, and delay pools should be enabled for ICAP responses, but with
enough knobs for admins to disable ICAP pooling where needed. Most ICAP
environments will probably want to disable bandwidth controls because
ICAP service traffic is usually "local".
Removed StoreEntry::bytesWanted() TRY_FWD_HDR_WAIT guard that disabled
delay pool application until the final HTTP response headers are
received (HttpStateData::haveParsedReplyHeaders() is called). Nobody
could fully explain why that condition was really needed (and we
obviously want to limit bandwidth consumption at all response processing
stages, in general). The now-removed guard was bypassing delay pool
checks for virgin HTTP response (until the ICAP server responded with
adapted headers), causing bandwidth overuse.
Possibly fixes or helps with Squid bug #2606 as well.
Bug 2976: squid reports ERR_INVALID_URL for transparently captured requests when reconfiguring
During reconfigure the configured AnyP::PortCfg objects in http_port_list
may deleted so it is not safe to use them while processing Http requests.
For this reason inside prepareTransparentURL (file client_side.cc) function
the protocol was hard-coded to "http" instead of read it from the related
AnyP::PortCfg object.
But this is breaks the intercepted https traffic.
This patch:
1. Inside prepareTransparentURL read the protocol from the related
AnyP::PortCfg object
2. add_http_port() locks the new port pointer before linking it.
3. parsePortCfg() locks the new port pointer before linking it.
4. free_PortCfg() unlock the old port pointer before unlinking
it. It does not delete the old pointer.
This patch also discussed in squid-dev user mailing list in
"Re: [PATCH] Squid host rewrite for intercepted https requests"
thread.
Alex Rousskov [Sat, 16 Jun 2012 15:03:46 +0000 (09:03 -0600)]
Fix several ACL-related bugs including broken default rules and ACCESS_DUNNO.
For example:
# broken when "goodGuys" matches (denies good guys)
acl_driven_option deny !goodGuys
and
# broken if badGuys fails to match or mismatch (allows bad guys)
acl_driven_option allow !badGuys
Fixing the above resulted in significant changes (and more fixes!)
detailed below.
* Revised ACLChecklist::fastCheck() and nonBlockingCheck() APIs to
clarify all possible outcomes and to specify that exceptional ACL
check outcomes (not ALLOW or DENIED) are not ignored/skipped but
result in the same exceptional final answer. I believe this is the
right behavior even if it is going to break some [already broken
IMHO] existing configurations. Skipping failed ACLs is insecure and
may lead to confusing results.
* Correctly handle cases where no rules were matched and, hence, the
keyword/action of the last seen rule (if any) has to be "reversed".
* Do not ignore non-allow/deny outcomes of rules in fastCheck().
* Move away from setting the "default" (and usually wrong) "current"
answer and then sometimes adjusting it. Set the answer only when we
know what it is. This is done using markFinished() call which now
accepts the [final] answer value and debugging reason for selecting
that answer.
* Streamline and better document ACLChecklist::matchAclList()
interface. Use it in a more consistent fashion.
* Rewrote ACLChecklist::matchAclList() implementation when it comes to
handling ACLList::matches() outcomes. Better document and restrict
supported outcomes. Assert on unsupported outcomes (for now).
* Removed ACLChecklist::lastACLResult(). It was doing nothing but
duplicating nodeMatched value as far as I could tell.
* Removed ProxyAuthNeeded class. It is an async state that does not
perform async operations and, hence, is not needed.
* Move IdentLookup::checkForAsync() connection check into
ACLIdent::match() to avoid creating an async state that is not
needed.
* Polished aclMatchExternal() and greatly simplified
ACLExternal::ExternalAclLookup() to avoid creating async state under
non-async conditions, to avoid checking for the same conditions
twice, to fix wrong debugging messages, and to simplify (and possibly
fix) the overall algorithm.
The aclMatchExternal() call now checks most of the corner cases,
discards stale cached entries, and schedules either a background
cache update or a regular external lookup as needed.
ACLExternal::ExternalAclLookup() code is now
ExternalACLLookup::Start(). It initiates an external lookup. It does
not deal with the cached entry at all. It relies on
aclMatchExternal() to check various preconditions.
Some of the old code made little sense to me, and this is the biggest
ACL-specific change in this project, with the highest probability of
new bugs or unintended side-effects.
My goal here was to prevent aclMatchExternal() from creating an async
state where none was needed because new ACLChecklist::matchAclList()
code prohibited such self-contradictory outcomes. However, I later
discovered that it is not possible to prohibit them without rewriting
how Squid DNS cache lookups are working -- ipcache_nbgethostbyname()
and similar code may call back immediately if the item is in the
cache. Since I did not want to rewrite DNS code as well, I ended up
relaxing the ACLChecklist::matchAclList() code requirements, going a
step back to where we sometimes call ACLList::matches() twice for the
same ACL node.
Thus, it is probably possible to undo most of the external_acl.cc
changes. I left them in because I think they improve the quality of
the code and possibly fix a bug or two.
* Adjusted ACLMaxUserIP::match(), ACLProxyAuth::match(), and
ACLExternal::match() to explicitly stop ACL processing when an
exceptional state is discovered instead of just setting the current
answer and proceeding as if more ACLs could be checked. On the other
hand, we now do not set the answer if the corresponding internal
matching code (e.g., AuthenticateAcl()) needs an async operation
because we do not know the answer yet.
* Fixed HttpStateData::handle1xx() and
HttpStateData::finishingBrokenPost() to correctly handle
fastCheck(void) return values. They were assuming that there are only
two possible return values (ACCESS_DENIED/ALLOWED), potentially
subjecting more messages to invasive adaptations than necessary.
TODO:
* Rename currentAnswer() to finalAnswer(). We probably never change the
"current" answer any more.
When an intercepted SSL connection matches "ssl_bump none" in
squid.conf, Squid correctly refuses to bump it and establishes a TCP
tunnel using a fake CONNECT request. Unfortunately, the HTTP client
terminates with an "unknown protocol" SSL error.
Also the client_dst_passthru does not work as expected for intercepted requests
This can occur due to long lists of unit tests needing a lot of local
variable state tracking. Essentially 'function too long' after CPPUNIT
macros have been expanded.
Break the large set of request-line unit tests into groups related to
sub-parts of the request-line.
Alex Rousskov [Sat, 2 Jun 2012 00:21:53 +0000 (18:21 -0600)]
Assume [] surround an IPv6 address and strip them
Browsers such as Firefox, Chromium, and Safari prefer bare IPv6 addresses in
CNs. They generate confusing errors when they see bracketed CNs. For example:
You attempted to reach [2001:470:1:18::120], but instead you actually reached
a server identifying itself as [2001:470:1:18::120]. Chromium can say for sure
that you reached [2001:470:1:18::120], but cannot verify that that is the same
site as [2001:470:1:18::120] which you intended to reach.
Alex Rousskov [Fri, 1 Jun 2012 22:01:43 +0000 (16:01 -0600)]
Fixed several ACL-related bugs, including:
# broken when "goodGuys" matches (denies good guys)
acl_driven_option deny !goodGuys
and
# broken if badGuys fails to match or mismatch (allows bad guys)
acl_driven_option allow !badGuys
Fixing the above resulted in significant changes (and more fixes!)
detailed below.
* Revised ACLChecklist::fastCheck() and nonBlockingCheck() APIs to
clarify all possible outcomes and to specify that exceptional ACL
check outcomes (not ALLOW or DENIED) are not ignored/skipped but
result in the same exceptional final answer. I believe this is the
right behavior even if it is going to break some [already broken
IMHO] existing configurations. Skipping failed ACLs is insecure and
may lead to confusing results.
* Correctly handle cases where no rules were matched and, hence, the
keyword/action of the last seen rule (if any) has to be "reversed".
* Do not ignore non-allow/deny outcomes of rules in fastCheck().
* Move away from setting the "default" (and usually wrong) "current"
answer and then sometimes adjusting it. Set the answer only when we
know what it is. This is done using markFinished() call which now
accepts the [final] answer value and debugging reason for selecting
that answer.
* Streamline and better document ACLChecklist::matchAclList()
interface. Use it in a more consistent fashion.
* Rewrote ACLChecklist::matchAclList() implementation when it comes to
handling ACLList::matches() outcomes. Better document and restrict
supported outcomes. Assert on unsupported outcomes (for now).
* Removed ACLChecklist::lastACLResult(). It was doing nothing but
duplicating nodeMatched value as far as I could tell.
* Removed ProxyAuthNeeded class. It is an async state that does not
perform async operations and, hence, is not needed.
* Move IdentLookup::checkForAsync() connection check into
ACLIdent::match() to avoid creating an async state that is not
needed.
* Polished aclMatchExternal() and greatly simplified
ACLExternal::ExternalAclLookup() to avoid creating async state under
non-async conditions, to avoid checking for the same conditions
twice, to fix wrong debugging messages, and to simplify (and possibly
fix) the overall algorithm.
The aclMatchExternal() call now checks most of the corner cases,
discards stale cached entries, and schedules either a background
cache update or a regular external lookup as needed.
ACLExternal::ExternalAclLookup() code is now
ExternalACLLookup::Start(). It initiates an external lookup. It does
not deal with the cached entry at all. It relies on
aclMatchExternal() to check various preconditions.
Some of the old code made little sense to me, and this is the biggest
ACL-specific change in this project, with the highest probability of
new bugs or unintended side-effects.
My goal here was to prevent aclMatchExternal() from creating an async
state where none was needed because new ACLChecklist::matchAclList()
code prohibited such self-contradictory outcomes. However, I later
discovered that it is not possible to prohibit them without rewriting
how Squid DNS cache lookups are working -- ipcache_nbgethostbyname()
and similar code may call back immediately if the item is in the
cache. Since I did not want to rewrite DNS code as well, I ended up
relaxing the ACLChecklist::matchAclList() code requirements, going a
step back to where we sometimes call ACLList::matches() twice for the
same ACL node.
Thus, it is probably possible to undo most of the external_acl.cc
changes. I left them in because I think they improve the quality of
the code and possibly fix a bug or two.
* Adjusted ACLMaxUserIP::match(), ACLProxyAuth::match(), and
ACLExternal::match() to explicitly stop ACL processing when an
exceptional state is discovered instead of just setting the current
answer and proceeding as if more ACLs could be checked. On the other
hand, we now do not set the answer if the corresponding internal
matching code (e.g., AuthenticateAcl()) needs an async operation
because we do not know the answer yet.
* Fixed HttpStateData::handle1xx() and
HttpStateData::finishingBrokenPost() to correctly handle
fastCheck(void) return values. They were assuming that there are only
two possible return values (ACCESS_DENIED/ALLOWED), potentially
subjecting more messages to invasive adaptations than necessary.
Amos Jeffries [Mon, 28 May 2012 02:40:52 +0000 (20:40 -0600)]
Define PRIuSIZE for displaying size_t
This allows us to avoid casting size_t to long long for printf.
./configure script auto-detects the supposedly standard %zu macro in case
it is missing and compat/types.h defines some alternatives for systems
which do not define it.
Alex Rousskov [Wed, 23 May 2012 23:34:49 +0000 (17:34 -0600)]
Fix protocol names in AnyP::PortCfg after http_port_list revamp in r12121.
The bug manifests itself when the URIs of intercepted requests are rewritten
into "https_port://..." strings, resulting in "Invalid port '0'" errors in
urlParse, followed by HTTP 400 (Bad Request) rejection.
There are other, more subtle cases where wrong PortCfg protocol matters.
Alex Rousskov [Wed, 23 May 2012 23:23:12 +0000 (17:23 -0600)]
Fix protocol names in AnyP::PortCfg after http_port_list revamp in r12121.
The bug manifests itself when the URIs of intercepted requests are rewritten
into "https_port://..." strings, resulting in "Invalid port '0'" errors in
urlParse followed by HTTP 400 (Bad Request) rejection.
There are other, more subtle cases where wrong PortCfg protocol matters.