Alex Rousskov [Thu, 15 Sep 2011 17:51:23 +0000 (11:51 -0600)]
Added max-swap-rate=swaps/sec option to Rock cache_dir.
The option limits disk access to smooth out OS disk commit activity and to
avoid blocking Rock diskers (or even other processes) on I/O. Should be used
when swap demand exceeds disk performance limits but the underlying file
system does not slow down incoming I/Os until the situation gets out of
control.
TODO: Account for the I/O rate limit when estimating whether a future I/O
will complete in time (for swap-timeout).
TODO: Consider allowing the next swap in (i.e., read) through regardless of
the limit because, unlike writes, reads do not usually accumulate in OS
buffers.
Alex Rousskov [Tue, 13 Sep 2011 16:47:32 +0000 (10:47 -0600)]
SMP Caching: Core changes, IPC primitives, Shared memory cache, and Rock Store
Core changes
------------
* Added MemObject::expectedReplySize() and used it instead of object_sz.
When deciding whether an object with a known content length can be
swapped out, do not wait until the object is completely received and its
size (mem_obj->object_sz) becomes known (while asking the store to
recheck in vain with every incoming chunk). Instead, use the known
content length, if any, to make the decision.
This optimizes the common case where the complete object is eventually
received and swapped out, preventing accumulating potentially large
objects in RAM while waiting for the end of the response. Should not
affect objects with unknown content length.
Side-effect1: probably fixes several cases of unknowingly using negative
(unknown) mem_obj->object_sz in calculations. I added a few assertions
to double check some of the remaining object_sz/objectLen() uses.
Side-effect2: When expectedReplySize() is stored on disk as StoreEntry
metadata, it may help to detect truncated entries when the writer
process dies before completing the swapout.
* Removed mem->swapout.memnode in favor of mem->swapout.queue_offset.
The code used swapout.memnode pointer to keep track of the last page
that was swapped out. The code was semi-buggy because it could reset the
pointer to NULL if no new data came in before the call to doPages().
Perhaps the code relied on the assumption that the caller will never
doPages if there is no new data, but I am not sure that assumption was
correct in all cases (it could be that I broke the calling code, of course).
Moreover, the page pointer was kept without any protection from page
disappearing during asynchronous swapout. There were "Evil hack time"
comments discussing how the page might disappear.
Fortunately, we already have mem->swapout.queue_offset that can be fed
to getBlockContainingLocation to find the page that needs to be swapped
out. There is no need to keep the page pointer around. The
queue_offset-based math is the same so we are not adding any overheads
by using that offset (in fact, we are removing some minor computations).
* Added "close how?" parameter to storeClose() and friends.
The old code would follow the same path when closing swapout activity
for an aborted entry and when completing a perfectly healthy swapout. In
non-shared case, that could have been OK because the abort code would
then release the entry, removing any half-written entry from the index
and the disk (but I am not sure that release happened fast enough in
100% of cases).
When the index and disk storage is shared among workers, such
"temporary" inconsistencies result in truncated responses being
delivered by other workers to the user because once the swapout activity
is closed, other workers can start using the entry.
By adding the "close how?" parameter to closing methods we allow the
core and SwapDir-specific code to handle aborted swapouts appropriately.
Since swapin code is "read only", we do not currently distinguish
between aborted and fully satisfied readers: The readerGone enum value
applies to both cases. If needed, the SwapDir reading code can make that
distinction by analyzing how much was actually swapped in.
* Moved "can you store this entry?" code to virtual SwapDir::canStore().
The old code had some of the tests in SwapDir-specific canStore()
methods and some in storeDirSelect*() methods. This resulted in
inconsistencies, code duplication, and extra calculation overheads.
Making this call virtual allows individual cache_dir types to do custom
access controls.
The same method is used for cache_dir load reporting (if it returns
true). Load management needs more work, but the current code is no worse
than the old one in this aspect, and further improvements are outside
this change scope.
Moved common (and often rather complex!) code from store modules into
storeRebuildLoadEntry, storeRebuildParseEntry, and storeRebuildKeepEntry.
* Do not set object_sz when the entry is aborted because the true object
size (HTTP reply headers + body) is not known in this case. Setting
object_sz may fool client-side code into believing that the object is
complete.
This addresses an old RBC's complaint.
* When swapout initiation fails, mark swapout decision as
MemObject::SwapOut::swImpossible. This prevents the caller code from trying to
swap out again and again because swap_status becomes SWAPOUT_NONE.
TODO: Consider add SWAPOUT_ERROR, STORE_ERROR, and similar states. It
may solve several problems where the code sees _NONE or _OK and thinks
everything is peachy when in fact there was an error.
* Call haveParsedReplyHeaders() before entry->replaceHttpReply().
HaveParsedReplyHeaders() sets the entry public key and various flags (at
least). ReplaceHttpReply() packs reply headers, starting swapout process.
It feels natural to adjust the entry _before_ we pack/swap it, but I may be
missing some side-effects here.
The change was necessary because we started calling checkCachable() from
swapoutPossible(). If haveParsedReplyHeaders() is not called before we swap
out checks, the entry will still have the private key and will be declared
impossible to cache.
* Extracted the write-to-store step from StoreEntry::replaceHttpReply().
This allows the caller to set the reply for the entry and then update the
entry and the reply before writing them to store. For example, the server-side
haveParsedReplyHeaders() code needs to set the entry timestamps and make the
entry key public before the entry starts swapping out, but the same code also
needs access to entry->getReply() and such for timestampsSet() and similar
code to work correctly.
TODO: Calls to StoreEntry::replaceHttpReply() do not have to be modified
because replaceHttpReply() does write by default. However, it is likely that
callers other than ServerStateData::setFinalReply() should take advantage of
the new split interface because they call timestampsSet() and such after
replaceHttpReply().
* Moved SwapDir::cur_size and n_disk_objects to specific SwapDirs. Removed
updateSize(). Some cache_dirs maintain their own maps and size statistics,
making the one-size-fits-all SwapDir members inappropriate.
* A new SwapDir public method swappedOut() added. It is called from
storeSwapOutFileClosed() to notify SwapDir that an object was swapped
out.
* Change SwapDir::max_size to bytes, make it protected, use maxSize() instead.
Change SwapDir::cur_size to bytes, make it private, use currentSize() instead.
Store Config.Store.avgObjectSize in bytes to avoid repeated and error-prone
KB<->bytes conversions.
* Change Config.cacheSwap.swapDirs and StoreEntry::store() type to SwapDir.
This allows using SwapDir API without dynamic_cast.
* Always call StoreEntry::abort() instead of setting ENTRY_ABORTED manually.
* Rely on entry->abort() side-effects if ENTRY_ABORTED was set.
* Added or updated comments to better document current code.
* Added operator << for dumping StoreEntry summary into the debugging
log. Needs more work to report more info (and not report yet-unknown info).
* Fixed blocking reads that were sometimes reading from random file offsets.
Core "disk file" reading code assumed that if the globally stored disk.offset
matches the desired offset, there is no reason to seek. This was probably done
to reduce seek overhead between consecutive reads. Unfortunately, the disk
writing code did not know about that optimization and left F->disk.offset
unchanged after writing.
This may have worked OK for UFS if it never writes to the file it reads from,
but it does not work for store modules that do both kinds of I/O at different
offsets of the same disk file.
Eventually, implement this optimization correctly or remove disk.offset.
IPC primitives
--------------
To make SMP disk and memory caching non-blocking and correct, worker and
disker processes must asynchronously communicate with each other. We are
adding a collection of classes that support such communication.
At the base of the collection is the AtomicWordT template that uses GCC atomic
primitives such as __sync_add_and_fetch() to perform atomic operations on
integral values in memory shared by multiple Squid kids. AtomicWordT is used
to implement non-blocking shared locks, queues, store tables, and page pools.
To avoid blocking or very long searches, many operations are "optimistic" in
nature. For example, it is possible that an atomic store map will refuse to
allocate an entry for two processes even though a blocking implementation
would have allowed one of the processes to get the map slot. We speculate that
such conflict resolution is better than blocking locks when it comes to
caching, especially if the conflicts are rare due to large number of cache
entries, fast operations, and relatively small number of kids.
TODO: Eventually, consider breaking locks left by dead kids.
The shared memory cache keeps its own compact index of cached entries using
extended Ipc::StoreMap class (MemStoreMap). The cache also strives to keep its
Root.get() results out of the store_table except during transit.
Eventually, the non-shared/local memory cache should also be implemented
using a MemStore-like class, I think. This will allow to clearly isolate
local from shared memory cache code.
Allow the user to explicitly disable shared memory caching in SMP mode via
memory_cache_shared to squid.conf. Report whether mem_cache is shared.
Disable shared memory caching by default if atomic operations are not
supported. Prohibit shared memory caching if atomic operations are not
supported.
TODO: Better limits/separation for cache and I/O shared memory pages.
Eventually, support shared memory caching of multi-page entries.
Rock Store
----------
Rock Store uses a single [large] database-style file per cache_dir to store
cached responses and metadata. This part of the design is similar to COSS.
Rock Store does not maintain or rely on swap.state "log" for recovery.
Instead, the database is scanned in the background to load entries when Squid
starts. Rock Store maintains its own index of cached entries and avoids global
store_table. All entries must be max-size or smaller.
In SMP mode, each Rock cache_dir is given a dedicated Kid processes called
"disker". All SMP workers communicate with diskers to store misses and load
hits, using shared memory pages and atomic shared memory queues. Disker blocks
when doing disk I/O but workers do not. Any Diskers:Workers ratio is supported
so that the user can find and configure the optimal number of workers and
diskers for a given number of disks and CPU cores.
In non-SMP mode, should use good old blocking disk I/O, without any diskers,
but this has not been tested recently and probably needs more work.
Alex Rousskov [Sat, 10 Sep 2011 01:25:27 +0000 (19:25 -0600)]
Moved squid.conf global disk_io_timeout to cache_dir-local swap-timeout.
The I/O timeout option belongs to cache_dir because not all cache_dir types
support it and because different cache_dirs may need different timeout values,
especially if some of them handle very large or otherwise unusual files.
To propagate the knowledge of the option down to DiskIO/IpcIoFile I decided
to add a DiskFile::Config class and DiskFile::configure() method. At first
glance that API does not belong to DiskFile because only IpcIoFile supports
it. However, DiskFile may be a better location for it because
* Other specific DiskIO files may want to support the same configuration
API.
* Placing API in IpcIoFile would require either making Rock Store dependent
on IpcIoFile (in terms of linking and in terms of availability) or more
complex API with multiple inheritance, dynamic casting and such. We can
introduce the "more complex API" mentioned above later if needed.
Renamed "disk_io" to "swap" timeout because the option is about the whole swap
out/in delay (something an admin may care about) and not individual I/O
(something only low-level code should know about).
This patch adjusts the %la logformat code handling for intercepted connections
based on the following rules:
- If the corresponding http_port or https_port option has an explicit
listening host name or IP address, then log the IP address.
- Otherwise, log a dash character.
Also adjusts %lp logformat code handling for intercepted connections to always
log the port number from the corresponding http_port or https_port option.
Amos comments about %la formating code:
For the record these are the permutations we seek to cover...
Scenario 1: client 192.168.0.3 connects to google (74.125.237.81). Gets intercepted into Squid.
Alex Rousskov [Fri, 9 Sep 2011 16:33:54 +0000 (10:33 -0600)]
Fixed max-stale check. Entities not exceeding max-stale were marked as stale.
Since the fixed check is performed for entities already suspected of being
stale by refreshCheck(), it is difficult to describe exactly which entities
were affected by the bug. A rough description would be "entities which would
otherwise qualify for a FRESH_OVERRIDE_EXPIRES or FRESH_OVERRIDE_LASTMOD
exceptions located below the fixed check.
Other concerns about staleness checks have been discussed on squid-dev's
"max_stale broken?" email thread.
Alex Rousskov [Fri, 9 Sep 2011 16:16:43 +0000 (10:16 -0600)]
Added RunnersRegistry, an API to register and, later, run a group of actions.
Useful for keeping general initialization/cleanup management code (e.g.,
main.cc) independent from specific initialization/cleanup code (e.g.,
Store file systems or memory cache) during staged initialization and
cleaning.
Designed with Rock Store needs in mind. Currently unused. Should eventually be
used for most modules initialization and cleanup, removing main.cc dependency
on those modules and perfecting [de]initialization order.
The %I must print the server ip address, but currently displays the host name of
the server on squid error pages. This patch fixes %I to print the server ip
address in the server or "[unknown]" otherwise.
Alex Rousskov [Tue, 6 Sep 2011 20:42:29 +0000 (14:42 -0600)]
Polished cache_dir rebuild warnings.
Use DBG_IMPORTANT level, WARNING: prefix, and explain that the bad entry is
going to be ignored. While this is an improvement of the moved code, most
of these warnings are still not user-friendly as they report low-level
details that most admins cannot use.
Alex Rousskov [Tue, 6 Sep 2011 20:14:29 +0000 (14:14 -0600)]
Temporary fixed coredumps when isOpen() is called during shutdown cleanup.
For a permanent fix, we need to avoid deleting fd_table while it is still
in use by others, such as DeferredReads, possibly by allowing event loop
to run during shutdown.
Alex Rousskov [Tue, 6 Sep 2011 20:10:26 +0000 (14:10 -0600)]
Adjusted cf.data.pre to reflect cross-vectoring point adaptation plan support.
These documentation changes were somehow missed from the r11327 commit that
introduced support for dynamic adaptation plans that cover multiple vectoring
points.
Alex Rousskov [Tue, 6 Sep 2011 18:28:11 +0000 (12:28 -0600)]
Support maximum field width for string access.log fields.
Some standard command-line and some log processing tools have trouble
handling URLs or other logged fields exceeding 8KB in length. Moreover,
Squid violates its own log line format and truncates the entire log line
if, for example, the URL is 8KB long. By supporting .precision format
argument, we allow the administrator to specify logged URL size and
avoid these problems.
Limiting logged field width has no effect on traffic on the wire, with
the exception of log records if they are sent over the network, of course.
TODO: The name comes from the printf(3) "precision" format part. It may
be a good idea to rename our "precision" into max_width or similar,
especially if we do not support floating point precision logging.
TODO: Old code used chars to store user-configured field width and
precision. That does not work for URLs, headers, and other entries
longer than 256 characters. This patch changes the storage type to int.
The code should probably be polished further to remove unsigned->signed
conversions.
Alex Rousskov [Tue, 6 Sep 2011 17:57:57 +0000 (11:57 -0600)]
Send RST packet when closing an ICAP connection after a transaction error.
This helps identify abnormal transaction termination at the ICAP server. The
code is from a production-running branch and was proven useful in some
environments (especially if the ICAP service needs help correctly dealing with
prematurely terminated transfers), but it is not yet clear whether the
advantages of doing this on a regular basis outweigh the overheads. If not,
we will need a squid.conf parameter to control connection closing behavior.
Alex Rousskov [Tue, 6 Sep 2011 17:37:29 +0000 (11:37 -0600)]
Log ICAP_ERR_GONE ICAP transaction outcome when ICAP initiator disappears early.
In this case, "early" means "before the end of the ICAP transaction".
Previously, "catch all" ICAP_ERR_UNKNOWN outcome was logged into the
adaptation log in such cases, raising red flags that there is something wrong
with the ICAP service or Squid.
Bug 3243: Fix cases: raw-IPv6, case variant FQDN, internal request
* do not unwrap the Host header IPv6. URL is not unwrapped for the test.
They should be identical regardless of wrapping choice.
Also ipcache happily unwraps for the intercept resolution cases
* use matchDomainName() algorithm instead of strcmp(). It is tuned for
fast case-less domain comparisons. But does support wildcard matches
so add a check of domain string length first.
* bypass for requests about to be serviced internally. We have some nasty
code of our own re-writing the URL without updating the Host: header.
Thanks to Sergey Merzlikin and Jenny Lee for detecting these.
Alex Rousskov [Tue, 30 Aug 2011 15:45:20 +0000 (09:45 -0600)]
Use a time-based limit for the Rock Store rebuild loop.
Compared to the old counter-based limit, a time-based limit is better because
it allows us to quit right after a blocked I/O or quit sooner when I/Os become
slower. Similarly, a time-based limit often allows us to rebuild a much larger
number of entries at a time because most sequential read I/Os are very fast.
Alex Rousskov [Tue, 30 Aug 2011 03:39:39 +0000 (21:39 -0600)]
Improved disker timeout handling.
A disker timeout may happen because the disker is stuck doing an I/O or
because the worker has not received an "I/O results available" notification.
If there was no notification (e.g., because the disker got stuck before
completing UDS sending steps), it is still possible that the disker output
queue cointains completed I/Os. Try to process as many of those old queued
I/Os as possible before abandoning the rest of the I/O requests.
Reduce the number of "error: timeout" lines reported at level 1. When multiple
I/Os timeout, just report the fact of the timeout, not every timedout I/O.
Needs more work to report the actual timeout value for the longest-waiting
I/O.
Double check that we are not scheduling two timeout triggers, just in case.
Alex Rousskov [Mon, 29 Aug 2011 23:14:17 +0000 (17:14 -0600)]
Use a time-based limit for the disker I/O loop.
Compared to the old counter-based limit, a time-based limit is better because
it allows us to quit right after a blocked I/O or quit sooner when I/Os become
slower.
Alex Rousskov [Mon, 29 Aug 2011 21:02:27 +0000 (15:02 -0600)]
Initial support for breaking the "do all queued I/Os" loop.
As the old TODO explained, it is necessary to break the loop to
give IpcIo notifications and other Squid process activities to
progress when many I/Os are queued or after a slow I/O blocks
the disker process for a while. More work is needed to break
based on cumulative I/O time rather than I/O count.
Alex Rousskov [Mon, 29 Aug 2011 20:58:29 +0000 (14:58 -0600)]
Made FewToFewBiQueue::peek() check both incoming and outgoing queues
to find the oldest queued value. This may help when the delay is on
our side (i.e., we are not receiving or processing completed I/Os fast enough).
Dmitry Kurochkin [Sun, 28 Aug 2011 17:11:19 +0000 (21:11 +0400)]
Disable Rock store and shared memory cache if shared memory is not available.
The patch adds support for building on systems without POSIX shared
memory (e.g. OpenBSD). IpcIo DiskIO module and Rock store is not
enabled by default if shared memory support is not available. If IpcIo
or Rock store is explicitly enabled, but no shared memory support is
available, fail with error during configure. Fail with error at
runtime if shared memory cache is enabled but no shared memory support
is detected.
Latest squid never shows DNS errors for user. When the user tries to retrieve
web page from non-existent domain, ERR_CANNOT_FORWARD error appears,
which misleads both user and administrator.
Alex Rousskov [Fri, 26 Aug 2011 20:48:08 +0000 (14:48 -0600)]
Polished unused code.
The unused dlopen() call is actually useful to enable when lt_dlopen() reports
"file not found" errors for loadable modules that do exist but that Libtool
cannot load successfully due to undefined symbols or other errors.
This inability to correctly report a library loading error is a long-standing
Libtool bug, stemming from Libtool's desire to try and load several
differently named library files until one succeeds, losing true error
information in the process.
projects / thirdparty / squid.git / log
