Andrew Beverley [Sun, 23 Oct 2011 01:07:46 +0000 (19:07 -0600)]
ext_session_acl: version 1.2
This patch makes the following changes to the session helper:
- Removes support for Berkeley DB 1.85
- Adds support for the current Berkeley DB (db.h)
- Adds support for a DB environment (if a directory is specified as
the path then an environment is created). This gives better
synchronisation within multiple processes
- Fixes a bug with active mode where LOGIN/LOGOUT did not write to the DB
This patch finishes the conversion of ServerStateData into AsyncJob by properly
implementing the doneAll() method and by removing calls to deleteThis() or
replacing with mustStop() calls as appropriate.
The Adaptation::AccessCheck modified to schedule an AsyncJobCall when
access check finishes.
The ServerStateData and ClientHttpRequest classes modified to work with the new
Adaptation::AccessCheck.
Alex Rousskov [Fri, 14 Oct 2011 22:43:20 +0000 (16:43 -0600)]
Avoid "double to int" compiler warnings.
Could also use integer arithmetic instead of 1e3 double, but it is often safer
to use doubles when it comes to formulas because significant integer rounding
errors are difficult to spot.
Alex Rousskov [Fri, 14 Oct 2011 16:21:48 +0000 (10:21 -0600)]
SMP shared memory cache stats were not collected.
"Hot Object" stats were not reported for shared memory cache.
Mean disk object size stats were aggregated inaccurately for SMP.
Moved Store-related stats into a dedicated StoreStats class,
encapsulating memory cache-related (mem), disk cache-related (swap), and
global store (number of objects) stats. Used consistent naming scheme
and a common parent class to make memory and disk stats more alike.
Moved Store stats collection into corresponding Store classes rather
than forcing GetInfo() in stat.cc to know how to deal with all Store stats.
Alex Rousskov [Fri, 14 Oct 2011 16:00:08 +0000 (10:00 -0600)]
Account for max-swap-rate in swap-timeout handling for Rock.
Current swap-timeout code does not know about max-swap-rate. It
simply finds the longest-waiting I/O in disker queues (incoming and
outgoing) and then assumes that the new I/O will wait at least that
long. The assumption is likely to be wrong when the queue contains
lots of freshly queued requests to disker: Those requests have not
waited long yet, but a max-swap-rate limit will slow them down
shortly.
The patch changes the swap-timeout code to account for max-swap-rate
when dealing with the workers-to-disker queue: If there are N requests
pending, the new one will wait at least N/max-swap-rate seconds. Also
expected wait time is adjusted based on the queue "balance" member, in
case we have been borrowing time against future I/O already.
This patch:
- converts type of the Token::[width|precision] members from "unsigned int" to "int"
- renames the Token::[width|precision] members to Token::[widthMin/widthMax]
- removes unneeded typecastings
Alex Rousskov [Wed, 12 Oct 2011 20:56:41 +0000 (14:56 -0600)]
Allow non-shared memory caching when there are no cache_dirs.
Before this change, we destroyed unused/idle StoreEntries if nobody was voting
to keep them in store_table. That blocked non-shared memory cache from getting
new entries if there were no cache_dirs to vote for them, which is wrong. The
new code keeps unused/idle StoreEntries in store_table if nobody objects.
Alex Rousskov [Mon, 10 Oct 2011 14:39:00 +0000 (08:39 -0600)]
Fixed typos in the host_verify_strict description.
Frankly, the description is likely to still make little sense to
uninitiated because we do not explain what is "Host vs IP validation"
and what the "additional strict validation comparisons" are. There was
an attempt to explain the latter, but I think it failed. Perhaps there
are more typos that hide the intended meaning?
Amos Jeffries [Sun, 9 Oct 2011 05:44:22 +0000 (23:44 -0600)]
Add directive dns_v4_first to make IPv4 connections before IPv6 is tried.
Default off, to prefer the faster protocol.
The use-case for this is networks which are IPv6-enabled but stuck
behind slow tunnels and whose upstream is not supporting full transit
services over IP.
Properly parse HTTP list headers with embedded 8-bit characters
MSIE and maybe other browsers sometimes sends 8-bit high characters
in HTTP headers (and URLs). This was mistakenly read as CTL characters
on platforms with signed char type (i.e. x86 etc).
One visible effect of this was that HTTP Digest authentication failed
in MSIE when following a link with embedded 8-bit or UTF-8 characters.
Currently we are locking every file going to be accessed by CertificateDB code
even if it is not realy needed, because of a more general lock.
This patch:
- Replace the old FileLocker class with the pair Lock/Locker classes
- Remove most of the locks in CertificateDB with only two locks one
for main database locking and one lock for the file contain the
current serial number.
Bug 3349: Bad support for adaptation service URIs with '='
Currently using URIs which include "=" is not supported by
ecap_service/icap_service configuration parameters. Also the squid.conf
documentation about ecap_service is outdated.
This patch
- Fixes the [e|i]cap_service line parser to allow URIs with "="
- Changes the [e|i]cap_service configuration parameter to use the following syntax:
ecap_service id vectoring_point uri [name=value ...]
- Check for duplicated options
- Fixes the related documentation
- Also the older [e|i]cap_service syntax forms are supported:
ecap_service id vectoring_point [1|0] uri
ecap_service id vectoring_point [name=value ...] uri
- The "uri" options is not documented but supported.
Alex Rousskov [Thu, 6 Oct 2011 16:41:46 +0000 (10:41 -0600)]
Polished SMP caching code, primarily to stay out of the way in non-SMP mode.
Do not start useless diskers. Do not assume Rock cache_dirs are present.
Do not require IpcIo DiskIO module to build Rock store.
Check IPC I/O pages limits in Rock store only when using a disker.
Warn about Rock cache_dir disk space waste.
Warn if shared memory cache is enabled in non-SMP mode.
Fake shared memory segments if needed (e.g., we are using Rock cache_dirs with
no POSIX shared memory support) and possible (e.g., no SMP).
Alex Rousskov [Thu, 6 Oct 2011 16:38:03 +0000 (10:38 -0600)]
Added max-swap-rate=swaps/sec option to Rock cache_dir.
The option limits the rate of Rock disk access to smooth out OS disk commit
activity and to avoid blocking Rock diskers (or even other processes) on I/O.
Should be used when swap demand exceeds disk performance limits but the
underlying file system does not slow down incoming I/Os until the situation
gets out of control.
Bug 3190: Large HTTP POST stuck after early ICAP 400 error response
When an ICAP REQMOD service responds with an error to
(or the REQMOD transaction aborts while processing) a large HTTP
request, the HTTP request may get stuck because the request body
buffer gets full and nobody consumes the no-longer-needed content.
The ICAP code quits but leaves the body buffer intact in case the
client-side code wants to bypass the error. After that, nobody consumes
the request body because the buggy client side does not inform the body
pipe that there will be no other consumers, which would have triggered
a noteBodyConsumerAborted() callback and enable auto-consumption or closed
the client connection.
Remove entryLimitAllowed() TODO from RockSwapDirRr::run().
RockSwapDirRr::run() calls sd->entryLimitAllowed() when sd does not
have a map yet, which causes entryLimitAllowed() to use zero for the
lower limit. But that OK so we can remove the TODO.
In some configurations Rock cache_dir may hit the maximum entry limit.
In that case a significant disk space may be wasted. Before recent
SMP-related changes, there was code to warn the user about such
situations. The patch resurrects that code, adjusting it as needed to
match the current realities.
Check IPC I/O pages limits in Rock store only when using a disker.
Shared memory pages are used only when Rock is running with diskers.
Otherwise, Blocking IO is used and there is no need to check for IPC
I/O pages limits.
Before the change, a disker process was started for each configured
cache_dir. But not all cache_dirs need a disker: only Rock store
requires one when running in SMP mode. There was some existing code
to avoid starting useless diskers (see SwapDir::needsDiskStrand and
Config.cacheSwap.n_strands) but it was not complete. The patch fixes
this issue.
Add base class for registered runners that work with shared memory.
Shared memory runners do two main things:
* create shared memory segments
* open shared memory segments created earlier
Each runner has to decide whether it needs to create and/or open
shared memory segments. Common runners share (some) logic in making
this decisions. The purpose of this patch is to implement this logic
in a single place to allow easy reuse and avoid duplication.
"Fake" shared memory segments are enabled #if a system does not
support POSIX shared memory. Such segments use regular new to
allocate memory, it is not shared among multiple processes. The
purpose of the change is to allow code that uses Ipc::Mem::Segment to
run in non-SMP mode (that is when only a single process, except for
master, is running) on systems without POSIX shared memory support.
Note: running SHM-using features in SMP mode still requires POSIX
shared memory support, the patch does not change that.
Alex Rousskov [Mon, 3 Oct 2011 20:19:24 +0000 (14:19 -0600)]
Use atomic int for millisecond-based Balance instead of atomic int64_t.
Atomic int64_t may cause "undefined reference to __sync_fetch_and_add_8"
linking errors, due to GCC bugs. Those errors can be fixed by building with
CXXFLAGS='-march=i586' but there is no reason to introduce that complexity
as 2^31 milliseconds is 35791 hours which is still plenty for our purposes.
While the server writes the response to Store, the client side may
synchronously abort the entry. This happens, for example, when the
server receives a 304 response and handleIMSReply calls
sendClientOldEntry, which calls storeUnregister with our entry,
resulting in CheckQuickAbort.
Once server store write returns, if the server is done, it calls
FwdState::completed(). At that time, the server does not know that (and
should not care whether) the entry was aborted. Thus, we need to handle
aborted entries in FwdState::completed.
Optimized HttpHdrCc::packInto and clarified its behaviour in comments
Negative values to Cache-Control directives are now explicitly discarded
Clarified documentation
Explicitly check that integer-carrying Cache-control directives are positive.
Rename MAX_STALE_ALWAYS to MAX_STALE_ANY and changed its representation to very large positive integer.
Implemented internal HttpHdrCc::setValue call.
Removed unnecessarily explicitly-masked StringArea default constructor.
avoided temporary in HttpHdrCc name-to-id lookup
Started implementing have/set/get/clear combo for all members of HttpHdrCc
Fixed zero-pointer-dereference in http.cc
Remove SwapDir::reconfigure() arguments since they are not used.
Before the change, SwapDir::reconfigure() took index and path
arguments, but none of them was actually used: neither index nor path
can be changed during reconfigure. And both index and path are
available as SwapDir members so there is no reason to have these
arguments.
Ignore and warn about attempts to reconfigure static Rock store options.
Some Rock store options cannot be changed dynamically: path, size, and
max-size. Before the change, there were no checks during reconfigure
to prevent changing these options. This may lead to Rock cache
corruption and other bugs. The patch adds necessary checks to Rock
store code. If user tries to change an option that cannot be updated
dynamically, a warning is reported and the value is left unchanged.
Bug fix: "(ssl_crtd): Cannot add certificate to db" when updating expired cert
When ssl_crtd helper needs to add a fresh certificate to the database but
finds an expired certificate already stored, ssl_crtd deletes the expired
certificate file from disk before adding the fresh one. However, the addition
still fails because the expired certificate was not removed from database
indexes.
This fix:
- Adds code to update database indexes upon deletion of a row.
- Polishes certificates deletion code to avoid duplication.
TODO: Report failure details to Squid and make certificate-specific failures
not fatal for the ssl_crtd helper.
Alex Rousskov [Wed, 21 Sep 2011 18:05:55 +0000 (12:05 -0600)]
Temporary fix: Avoid killing Coordinator with unregistered cache mgr actions
that cause isOpen() assertions.
If a worker forwards a cache manager request to Coordinator and Coordinator
does not have that action registered, CacheManager::createRequestedAction()
throws (as it should) and Mgr::Request cleanup asserts when its half-baked
connection tries to close a not-yet-imported socket descriptor.
This workaround catches the exception, reports it, and manually closes the
socket descriptor. It also prevents an ACK response from being sent to the
worker, which triggers a worker timeout.
Mid-term TODO: Coordinator should register all actions that are known to kids.
Should Coordinator respond with an error instead of relying on a timeout?
Long-term TODO: Consider an API where cache manager responses can be
aggregated and formatted by Coordinator without knowing action-specific
details. After all, there are not so many types of action information (size,
count, rate, etc.) and most actions have simple reporting formats. Currently,
it is awkward to guarantee that Coordinator and all workers know all actions,
especially when some actions may be specific to non-worker kids such as
Coordinator and diskers.
projects / thirdparty / squid.git / log
