The AccessLogEntry objects currently are only members of the ClientHttpRequest
objects. There are cases where we need to access AccessLogEntry from server
side objects to retrieve already stored informations for the client request and use it in
server side code with format/* interface (eg use Format::Format::assemble
inside http.cc)
This patch convert AccessLogEntry class to RefCountable to allow link it with
other than the ClientHttpRequest objects.
When '-k parse' is used deprecation notices and upgrade help messages etc
need to be bumped consistently up to level-0 and this macro will help
reducing the (?:) code mistakes.
Alex Rousskov [Sat, 21 Jul 2012 01:28:42 +0000 (19:28 -0600)]
Avoid bogus "Disk space over limit" warnings when rebuidling dirty ufs index.
Subtract sizes of added-then-rejected entries while loading ufs cache index.
Before SMP changes, the ufs code was incorrectly ignoring the size of
loaded-but-not-yet-validated entries, leading to cache disk overflows.
After SMP changes, the ufs code was not subtracting sizes of
loaded-but-then-rejected entries, leading to bogus "Disk space over
limit" warnings. Now we correctly account for both kinds of entries.
Bug 3478: Partial fix: Connection-auth on intercepted connections is broken
Currenty in the case of intercepted connections each request is open a new
connection to the destination server, even if the connection is a valid pinned
connection.
This patch fixes this problem and reuses pinned connections on intercepted
requests.
Fail with an explanation instead of asserting. The assertion fails when a ufs
cache_dir's swap.state has inconsistencies AND the user starts Squid with a -S
command line option. Normally, such inconsistencies are ignored and many of
them are benign. For example, a missing cache file with an ADD record in
swap.state is such an inconsistency.
The -S option was meant to help developers troubleshoot inconsistencies by
analyzing core dumps, but (a) admins treat assertions as Squid bugs and file
bug reports and (b) in most cases, it is really difficult to find the
inconsistency when Squid asserts after detecting all of them (and leaving the
detection context).
We now explicitly tell the admin what their options are and quit instead of
asserting.
TODO: Consider adding a ufs cache_dir option that checks for and removes
inconsistencies instead of not checking at all (default) or checking and
quitting (-S). This is difficult because some valid cache entries may look
inconsistent while they are being updated and some invalid cache entries
are not visible to Squid without a full directory scan.
Alex Rousskov [Sat, 21 Jul 2012 01:14:53 +0000 (19:14 -0600)]
Bug 3525: Do not resend nibbled PUTs and avoid "mustAutoConsume" assertion.
The connected_okay flag was not set when we were reusing the connection.
The unset flag overwrote bodyNibbled() check, allowing FwdState to retry
a PUT after the failed transaction consumed some of the request body
buffer, triggering BodyPipe.cc:239: "mustAutoConsume" assertion.
We now set the connected_okay flag as soon as we get a usable connection
and do not allow it to overwrite the bodyNibbled() check (just in case).
Restore memory caching ability lost since r11969.
Honor maximum_object_size_in_memory for non-shared memory cache.
Since r11969, Squid calls trimMemory() for all entries to release unused
MemObjects memory. Unfortunately, that revision also started trimming objects
that could have been stored in memory cache later. Trimmed objects are no
longer eligible for memory caching.
It is possible that IN_MEMORY check in StoreEntry::trimMemory() was preventing
excessive trimming in the past, but after SMP changes, IN_MEMORY flag is set
only after the object is committed to a memory cache in
StoreController::handleIdleEntry(), so we cannot rely on it. For example:
StoreEntry::unlock()
StoreController::handleIdleEntry() // never get here because entry is
set IN_MEMORY status // already marked for release
This change adds StoreController::keepForLocalMemoryCache() and
MemStore::keepInLocalMemory() methods to check if an entry could be stored in
a memory cache (and, hence, should not be trimmed). The store-related part of
the trimming logic has been moved from StoreEntry::trimMemory() to
StoreController::maybeTrimMemory(). StoreEntry code now focuses on checks that
are not specific to any memory cache settings or admission policies.
These changes may resulted in Squid honoring maximum_object_size_in_memory for
non-shared memory cache. We may have been ignoring that option for non-shared
memory caches since SMP changes because the check for it was specific to a
shared memory cache.
Bug 3577: File Descriptors not properly closed in trunk r12185.
Bug 3583: Server connection stuck after TCP_REFRESH_UNMODIFIED.
These changes fix FD leaks and stuck connections under two conditions:
1) Client aborts while Squid's server-side establishes a connection
Bug 3577: When a client quits while ConnOpener is trying to open the
connection to the next hop, FwdState cancels its ConnOpener callback.
ConnOpener notices that when trying to connect again and quits before
establishing a connection. The ConnOpener cleanup code did not close the
temporary FD used for establishing the connection. It did call fd_close(),
but fd_close() does not close the FD, naturally.
ConnOpener was probably leaking the temporary FD in other error handling
cases as well. It was never closed unless the connection was successful.
2) Client aborts after Squid's server-side established a connection:
Bug 3583: When a client aborts the store entry after receiving an HTTP 304 Not
Modified reply in response to a cache refreshing IMS request, HttpStateData
notices an aborted Store entry (after writing the reply to store), but does
virtually nothing, often resulting in a stuck server connection, leaking a
descriptor. Now we abort the server-side transaction in this case.
Bug 3577: Similarly, when a client disconnects after Squid started talking to
the origin server but before Squid received a [complete] server response,
HttpStateData notices an aborted Store entry (during the next read from the
origin server), but does virtually nothing, often resulting in a stuck server
connection, leaking a descriptor. Now we abort the server-side transaction in
this case.
FwdState now also closes the server-side connection, if any, when the client
aborts the store entry and FwdState::abort() callback is called. This helps
reduce the number of concurrent server-side connections when clients abort
connections rapidly as Squid no longer has to wait for the server-side I/O to
notice that the entry is gone. The code to close the connection was temporary
removed in trunk r10057.1.51.
Alex Rousskov [Fri, 13 Jul 2012 11:53:18 +0000 (05:53 -0600)]
[request|reply]_header_* manglers fixes to handle custom headers
This patch fix the [request|reply]_header_[access|replace] configuration
parameters to support custom headers. Before this patch the user was able
to remove/replace/allow all custom headers using the "Other" as header name.
Dmitry Kurochkin [Thu, 28 Jun 2012 05:14:18 +0000 (23:14 -0600)]
Fix build with GCC 4.7 (and probably other C++11 compilers).
User-defined literals introduced by C++11 break some previously valid
code, resulting in errors when building with GCC v4.7. For example:
error: unable to find string literal operator 'operator"" PRIu64'
In particular, whitespace is now needed after a string literal and
before something that could be a valid user-defined literal. See
"User-defined literals and whitespace" section at [1] for more details.
The patch adds spaces between string literals and macros.
Amos Jeffries [Thu, 28 Jun 2012 05:12:14 +0000 (23:12 -0600)]
Cleanup: disconnect Authentication and URL-rewrite callback handlers
The authentication handlers were for some reason using RH (rewrite helper)
callback typedef. But specifying it as a fatal error if the char*
parameter was used in auth.
Assign a new callback typedef AUTHCB for use by authentication callers.
This allows auth callers to use different parameters (none) and to avoid
possibly fatal mistakes when coding new auth modules.
Alex Rousskov [Thu, 28 Jun 2012 05:11:18 +0000 (23:11 -0600)]
Account for Store disk client quota when bandwidth-limiting the server.
It is not clear why the store client type matters when
MemObject::mostBytesAllowed() is trying to find the maximum delay pool
quota for reading from the next hop HTTP server. Whether the client(s)
are reading from disk or RAM, the corresponding server-side bandwidth
ought to be limited.
This code was removed as a part of bug 3462 investigation, but it is not
needed to fix bug 3462.
Julien Pinon [Thu, 21 Jun 2012 00:39:01 +0000 (18:39 -0600)]
Bug 3462: Delay Pools and ICAP
Allow StoreEntry::bytesWanted() API to ignore delay pools. Use that
feature when shoveling adapted response body from ICAP/eCAP BodyPipe
into Store.
If we do not ignore delay pools in
ServerStateData::handleMoreAdaptedBodyAvailable() context, and our pool
quota is exhausted, we will stop reading from the adaptation BodyPipe,
but there is no code/API to notify a BodyPipe reader when the quota
becomes available again. With no reads from the pipe, the ICAP service
stops sending more adapted body data and possibly stops reading the
virgin body data from Squid, resulting in a stuck ICAP RESPMOD and the
master HTTP transaction.
We do need to call StoreEntry::bytesWanted() in this context because
otherwise Squid may run out of RAM (Squid bug #2619). The fix for that
problem inadvertently created this bug when delay pools were enabled.
Long-term, a "kick BodyPipe consumer when there is quota" code should be
added, and delay pools should be enabled for ICAP responses, but with
enough knobs for admins to disable ICAP pooling where needed. Most ICAP
environments will probably want to disable bandwidth controls because
ICAP service traffic is usually "local".
Removed StoreEntry::bytesWanted() TRY_FWD_HDR_WAIT guard that disabled
delay pool application until the final HTTP response headers are
received (HttpStateData::haveParsedReplyHeaders() is called). Nobody
could fully explain why that condition was really needed (and we
obviously want to limit bandwidth consumption at all response processing
stages, in general). The now-removed guard was bypassing delay pool
checks for virgin HTTP response (until the ICAP server responded with
adapted headers), causing bandwidth overuse.
Possibly fixes or helps with Squid bug #2606 as well.
Bug 2976: squid reports ERR_INVALID_URL for transparently captured requests when reconfiguring
During reconfigure the configured AnyP::PortCfg objects in http_port_list
may deleted so it is not safe to use them while processing Http requests.
For this reason inside prepareTransparentURL (file client_side.cc) function
the protocol was hard-coded to "http" instead of read it from the related
AnyP::PortCfg object.
But this is breaks the intercepted https traffic.
This patch:
1. Inside prepareTransparentURL read the protocol from the related
AnyP::PortCfg object
2. add_http_port() locks the new port pointer before linking it.
3. parsePortCfg() locks the new port pointer before linking it.
4. free_PortCfg() unlock the old port pointer before unlinking
it. It does not delete the old pointer.
This patch also discussed in squid-dev user mailing list in
"Re: [PATCH] Squid host rewrite for intercepted https requests"
thread.
This can occur due to long lists of unit tests needing a lot of local
variable state tracking. Essentially 'function too long' after CPPUNIT
macros have been expanded.
Break the large set of request-line unit tests into groups related to
sub-parts of the request-line.
Amos Jeffries [Mon, 4 Jun 2012 10:50:34 +0000 (04:50 -0600)]
Define PRIuSIZE for displaying size_t
This allows us to avoid casting size_t to long long for printf.
./configure script auto-detects the supposedly standard %zu macro in case
it is missing and compat/types.h defines some alternatives for systems
which do not define it.
Amos Jeffries [Mon, 4 Jun 2012 10:25:03 +0000 (04:25 -0600)]
Add support for TLSv1.1 and TLSv1.2 options and methods
When OpenSSL v1.0.1+ is being built against.
Also update the documentation for sslproxy_version which was not
mentioning what the supported version codes were.
Future work:
* make version config option(s) accept a set of named versions and
convert to codes internally.
* redesign how version and options are handled. Admin should be able to
just list the TLSv* wanted and Squid figure out the appropriate options
from there.
Amos Jeffries [Mon, 4 Jun 2012 10:23:57 +0000 (04:23 -0600)]
SourceLayout: port config and select-loop priority polishing
- renames http_port_list to AnyP::PortCfg
- de-duplicate https_port_list into AnyP::PortCfg
- shuffles related globals and defines into anyp/PortCfg.*
- renames MAXHTTPPORTS to MAXTCPLISTENPORTS to suit its actual coverage of HTTP and HTTPS ports.
- shuffled config port clone function into a method.
- rename ICP/HTCP/SNMP API functions to consistent *OpenPorts() and *ClosePorts()
NP:following applies to incoming_* and *_poll_cnt directives.
- renames *_icp_* to *_udp_*
- renames *_http_* to *_tcp_*
- shuffles duplicated struct SquidConf options into a shared structure
- shuffles related defines into comm/Loops.h
- documents options better
- various other cosmetic syntax tweaks and polish
One bug fix:
comm_dns_incoming was not being propigated in StatsHist copy/clone.
Now is. I seem to remember mention of something similar being zero before,
but can't find the bug report.
Alex Rousskov [Mon, 21 May 2012 02:57:33 +0000 (20:57 -0600)]
Bug 3466: Adaptation stuck on last single-byte body piece
Changed StoreEntry::bytesWanted(range) to return range.end when the entry can
accommodate range.end bytes. This makes it possible to use that method for
single-byte ranges. Old code returned zero for such ranges, which was
difficult to distinguish from situations where no bytes were wanted at all.
TODO: The StoreEntry::bytesWanted(range) API is left undocumented because it
seems to be slightly broken and/or inconsistent with callers and with the
DelayId::bytesWanted(min, max) API. AFAICT, we should convert
StoreEntry::bytesWanted API from range-based to min/max-based or even just
max-based.
Store Entry API does not use the lower end of the range (except for the
now-removed assertion that the range is not empty). I suspect that Store API
was meant to be used with (first, last+1) "byte position" parameters (returning
the number of bytes wanted) while the DelayId API was meant to be used with
(min, max) "number of bytes" parameters. However, StoreEntry::bytesWanted
implementation does not follow this assumption so perhaps my speculation is
wrong and there are more problems, including this change.
Amos Jeffries [Mon, 21 May 2012 02:35:41 +0000 (20:35 -0600)]
Register status 308 permanent redirect
* relay "Permanent Redirect" message on status line
* MAY cache these responses with heuristics
* accept this status as a redirect status from URL redirectors
Alex Rousskov [Wed, 11 Apr 2012 09:45:41 +0000 (03:45 -0600)]
Bug 3441: Part 3: Replace corrupted v1 swap.state with new v2 format.
A fix for bug 3408 changed the offset at which we start writing dirty
swap.state entries from StoreSwapLogHeader::record_size to StoreSwapLogHeader
size. However, the log-reading code still read the entries starting from the
old offset (which is required to remain compatible with how a clean swap.state
is written).
Wrong starting offset essentially means that the vast majority of read
swap.state entries were bogus. They could only match some real entry when 64*n
is divisible by 12 and perhaps when their random data just happened to match a
real entry. Part 2 of this bug fix (trunk r11995) started to pad the [dirty]
swap.state header to start entry writing at StoreSwapLogHeader::record_size
boundary.
Changes specific to Part 3:
Unfortunately, since old v1 logs could contain completely bogus entries as the
result of being read (at some point) from the wrong offset, we should not load
v1 logs any more (neither dirty nor clean because what looks clean now could
be based on a previously dirty and, hence, corrupted log). This forced us to
raise the swap.state format version from 1 to 2.
After this change, if a v1 swap log is detected, Squid ignores it and does a
from-directory rebuild as if no swap.state files were found.
Since we had to change swap.state format version, we also made log entry size
and composition the same across virtually all platforms; added checksums so
that a similar bug would not go unnoticed for so long (and would not result in
log corruption); and increased the size of time-related entries to avoid the
"year 2038" problem.
The swap log entries are still written to disk in host byte order.
We now also zero the [dirty] swap.state header padding to prevent random and
potentially sensitive garbage in logs.
Cache index rebuild kinds are now reported using the following three labels:
* Rebuild using a swap log created by Squid during clean shutdown: "clean log"
* Rebuild using a swap log accumulated by a running Squid: "dirty log"
* Rebuild using directory scan: "no log"
The first kind used to be reported as CLEAN and the other two as DIRTY rebuild.
This create a Comm::UdpOpenDialer class which replaces the ICP, HTCP and
SNMP start-listening dialer classes. Their code was very close to
identical anyway.
ICP and HTCP can now also use the dialer Comm::Connection parameter
instead of assuming that the callback relates to the global incoming
port variable.
projects / thirdparty / squid.git / log
