Amos Jeffries [Mon, 4 Jun 2012 10:23:57 +0000 (04:23 -0600)]
SourceLayout: port config and select-loop priority polishing
- renames http_port_list to AnyP::PortCfg
- de-duplicate https_port_list into AnyP::PortCfg
- shuffles related globals and defines into anyp/PortCfg.*
- renames MAXHTTPPORTS to MAXTCPLISTENPORTS to suit its actual coverage of HTTP and HTTPS ports.
- shuffled config port clone function into a method.
- rename ICP/HTCP/SNMP API functions to consistent *OpenPorts() and *ClosePorts()
NP:following applies to incoming_* and *_poll_cnt directives.
- renames *_icp_* to *_udp_*
- renames *_http_* to *_tcp_*
- shuffles duplicated struct SquidConf options into a shared structure
- shuffles related defines into comm/Loops.h
- documents options better
- various other cosmetic syntax tweaks and polish
One bug fix:
comm_dns_incoming was not being propigated in StatsHist copy/clone.
Now is. I seem to remember mention of something similar being zero before,
but can't find the bug report.
Alex Rousskov [Mon, 21 May 2012 02:57:33 +0000 (20:57 -0600)]
Bug 3466: Adaptation stuck on last single-byte body piece
Changed StoreEntry::bytesWanted(range) to return range.end when the entry can
accommodate range.end bytes. This makes it possible to use that method for
single-byte ranges. Old code returned zero for such ranges, which was
difficult to distinguish from situations where no bytes were wanted at all.
TODO: The StoreEntry::bytesWanted(range) API is left undocumented because it
seems to be slightly broken and/or inconsistent with callers and with the
DelayId::bytesWanted(min, max) API. AFAICT, we should convert
StoreEntry::bytesWanted API from range-based to min/max-based or even just
max-based.
Store Entry API does not use the lower end of the range (except for the
now-removed assertion that the range is not empty). I suspect that Store API
was meant to be used with (first, last+1) "byte position" parameters (returning
the number of bytes wanted) while the DelayId API was meant to be used with
(min, max) "number of bytes" parameters. However, StoreEntry::bytesWanted
implementation does not follow this assumption so perhaps my speculation is
wrong and there are more problems, including this change.
Amos Jeffries [Mon, 21 May 2012 02:35:41 +0000 (20:35 -0600)]
Register status 308 permanent redirect
* relay "Permanent Redirect" message on status line
* MAY cache these responses with heuristics
* accept this status as a redirect status from URL redirectors
Alex Rousskov [Wed, 11 Apr 2012 09:45:41 +0000 (03:45 -0600)]
Bug 3441: Part 3: Replace corrupted v1 swap.state with new v2 format.
A fix for bug 3408 changed the offset at which we start writing dirty
swap.state entries from StoreSwapLogHeader::record_size to StoreSwapLogHeader
size. However, the log-reading code still read the entries starting from the
old offset (which is required to remain compatible with how a clean swap.state
is written).
Wrong starting offset essentially means that the vast majority of read
swap.state entries were bogus. They could only match some real entry when 64*n
is divisible by 12 and perhaps when their random data just happened to match a
real entry. Part 2 of this bug fix (trunk r11995) started to pad the [dirty]
swap.state header to start entry writing at StoreSwapLogHeader::record_size
boundary.
Changes specific to Part 3:
Unfortunately, since old v1 logs could contain completely bogus entries as the
result of being read (at some point) from the wrong offset, we should not load
v1 logs any more (neither dirty nor clean because what looks clean now could
be based on a previously dirty and, hence, corrupted log). This forced us to
raise the swap.state format version from 1 to 2.
After this change, if a v1 swap log is detected, Squid ignores it and does a
from-directory rebuild as if no swap.state files were found.
Since we had to change swap.state format version, we also made log entry size
and composition the same across virtually all platforms; added checksums so
that a similar bug would not go unnoticed for so long (and would not result in
log corruption); and increased the size of time-related entries to avoid the
"year 2038" problem.
The swap log entries are still written to disk in host byte order.
We now also zero the [dirty] swap.state header padding to prevent random and
potentially sensitive garbage in logs.
Cache index rebuild kinds are now reported using the following three labels:
* Rebuild using a swap log created by Squid during clean shutdown: "clean log"
* Rebuild using a swap log accumulated by a running Squid: "dirty log"
* Rebuild using directory scan: "no log"
The first kind used to be reported as CLEAN and the other two as DIRTY rebuild.
This create a Comm::UdpOpenDialer class which replaces the ICP, HTCP and
SNMP start-listening dialer classes. Their code was very close to
identical anyway.
ICP and HTCP can now also use the dialer Comm::Connection parameter
instead of assuming that the callback relates to the global incoming
port variable.
Alex Rousskov [Sat, 3 Mar 2012 23:54:41 +0000 (16:54 -0700)]
Better helper-to-Squid buffer size management.
The minimum buffer size is reduced from 8KB to 4KB after a squid-dev
discussion to prevent wasting of "several hundred KB of unused permanent
memory on some installations".
We now increase the buffer if we cannot parse the helper response message.
The maximum buffer size is now 32KB. This should be enough for all known
helper responses.
We now warn if the read buffer reaches its capacity and kill the offending
helper explicitly. An increase in maximum buffer capacity to 32KB should make
such events rare.
Motivation: ssl_crtd helper may produce responses exceeding 9907 bytes in size
(and possibly much larger if multiple chained certificates need to be returned
to Squid). The old helper.cc code would fill the read buffer completely,
schedule a read for zero bytes, receive zero bytes, declare an EOF condition,
and close the stream (which kills ssl_crtd). Due to insufficient information
logged, the observable symptoms were pretty much the same as if ssl_crtd
closed the stream first, indicating a ssl_crtd bug.
HONDA Hirofumi [Sat, 3 Mar 2012 23:43:53 +0000 (16:43 -0700)]
Bug 3502: client timeout uses server-side read_timeout, not request_timeout
Also adjusts request_timeout description in squid.conf to clarify that
request_timeout applies to receiving complete HTTP request headers and not
just the first header byte or body. We reset the connection timeout to
clientLifetimeTimeout after parsing request headers.
https_port was correctly using Config.Timeout.request already.
Guy Helmer [Sat, 3 Mar 2012 23:38:55 +0000 (16:38 -0700)]
Bug 3497: Bad ssl_crtd db size file causes infinite loop
The db size file may become empty when Squid runs out of disk space. Ignoring
db size reading errors led to bogus db sizes used as looping condition. This
fix honors reading errors and also terminates the loop when no more
certificates can be removed. Both errors and removal failure are fatal to
ssl_crtd.
A positive side-effect of this fix is one less call to the relatively
expensive file-reading size()/readSize() methods under normal conditions.
I also removed "minimum db size" check because it did not seem to be in sync
with other ssl_crtd parameters such as fs block size and because its overall
purpose was unclear. The check was also removed by the original bug reporter.
TODO: Remaining problems include: ssl_crtd should not exit just because it
cannot write something to disk. A proper reporting/debugging API is missing.
- StatHist refactoring
- StatCounters untangling from global symbol table
- Added StatHist unit tests, and moved algorithm consistency checks there.
- Expanded storage for histograms to 64-bit unsigned. (bug 3381)
- Inlined StatHist constructor, destructor and assignment operator.
- added protection against self-assignment
- Implemented stubs for StatHist.cc and mem.cc
Alex Rousskov [Sat, 25 Feb 2012 04:32:57 +0000 (21:32 -0700)]
Retry requests that failed due to a persistent connection race
... instead of replying with ERR_ZERO_SIZE_OBJECT "Bad Gateway".
The ERR_ZERO_SIZE_OBJECT errors were visible to the client when the
destination had only one address because serverDestinations.shift()
made the list of destination empty and startConnectionOrFail() failed.
When FwdState starts to use a pinned connection, the connection is treated as
an idle persistent connection as far as race detection is concerned.
Currently, pinned connections cannot be reopened, repinned, and retried after
a pconn race. This will change when server-side bumped connections become
pinned.
It felt wrong that a failed serverConn may remain set while we are opening a
new connection so I set it to NULL after a squid-dev discussion indicating
that doing so should be safe.
We also now reset the local port number to zero in case it was set to the
actual source port by ConnOpener or other code working with the previous
connection to the same serverDestinations[0] address, although simple tests
worked (and showed changing source port) without this reset.
Currently the sslpassword_program configuration parameter does not work
for encrypted certificate keys on ssl-bump enabled http ports, and user
always asked to give the SSL key password.
Alex Rousskov [Sat, 25 Feb 2012 04:21:24 +0000 (21:21 -0700)]
Do not cache partially loaded entries in shared mem cache (and then serve them)
When handling a conditional request, Squid may load the beginning of a cached
object from disk, realize that the client has the same fresh copy, and respond
with 304 Not Modified. After that, Squid was checking whether the partially
loaded object should be kept in shared memory cache (if enabled). There were
no checks preventing memory caching of the partially loaded object.
Later, partially cached objects were served to clients, resulting in truncated
responses. I believe this happens because shared memory cache does not keep
all the StoreEntry data (just like a disk cache does not do that) so the fact
that only a part of the object was available was lost.
Alex Rousskov [Sat, 25 Feb 2012 04:20:25 +0000 (21:20 -0700)]
Do not swap out already swapped out objects.
I noticed that sometimes Squid would start swapping out an entry that was
recently loaded from disk and was still on disk. That wastes disk
resources (at best).
The old StoreEntry::mayStartSwapOut() code assumed that when swap_status is
not SWAPOUT_NONE it is SWAPOUT_WRITING, but SWAPOUT_WRITING is impossible
after recent StoreEntry::swapOut() modifications because mayStartSwapOut() is
only called when we are not swappingOut() already. SWAPOUT_DONE is possible.
Amos Jeffries [Sat, 25 Feb 2012 04:19:26 +0000 (21:19 -0700)]
Drop dead code in reply parsing
This code has not been used/needed in some time. It can die.
It is also no clear why it existed in the first place. The RFC is not
mentioned by number and RFC 2068/2616 only talk about tolerance for
whitespace before request lines, not replies.
Amos Jeffries [Sat, 25 Feb 2012 04:14:25 +0000 (21:14 -0700)]
SourceLayout: shuffle CommIO into DiskThreads library
While investigating the Windows port problems it became clear that the
CommIO object is not actually related to the rest of comm systems. It is
instead a dedicated higher level disk I/O pipe manager for DiskThreads.
Given that it causes build errors in comm.cc on Windows and does not
even need to be there this patch shuffles it into the DiskThreads
library. The build issues on Windows still exist but are now limited to
just that threads library and can be avoided temporarily with simple
./configure options.
This add a bit more leniency to the Host: header validation without
re-opening Squid to the cache poisoning risks involved.Resolving most
issues with websites using geo-based DNS results and/or short DNS TTL for
load balancing.
It alters host_verify_strict directive to allow requests which fail Host:
validation to continue through processing. The default remains OFF.
* blocks caching on the response to protect all other network clients
against one compromised client spreading infections.
* forces the original (untrusted) destination IP to be used instead of
any alternative Squid might find. Preventing Squid or peer DNS lookup
being the point of vulnerability for the same-origin bypass. For any
client to be vulnerable it must be vulnerable inside the browser agent
where the original TCP connection is established.
Also add a new error template ERR_CONFLICT_HOST to replace the confusing
"invalid request" message with a clear explanation of the problem and
some client workarounds.
FUTURE WORK:
* adapt processing to allow these requests to safely be passed to peers.
* adapt caching to permit safe sharing between clients making identical
requests to same sources.
Alex Rousskov [Thu, 2 Feb 2012 08:48:59 +0000 (01:48 -0700)]
Bug 3441: Part 1: Minimize cache size corruption by malformed swap.state.
If swap.state gets corrupted, a single entry with bogus size value will screw
up Squid idea of the current cache size. A newly added StoreSwapLogData sane()
method attempts to minimize the chance of corruption by ignoring log entries
with obviously bogus values.
However, without expensive size checks (-S or "Does the log entry matches the
actual cache file size?"), it is not possible to reliably detect all bogus log
entries.
If Squid gets a wrong idea of the current cache size, it may either cache too
much (and possibly run out of space) OR delete everything.
Properly initialize memory_cache_shared when not explicitly configured and
do not allocate shared memory segments if memory_cache_shared is off.
Prior to this change, shared memory was requested (during rrClaimMemoryNeeds)
before memory_cache_shared was initialized (during rrAfterConfig, unless
explicitly set in squid.conf). Moreover, MemStore::EntryLimit() calculation
ignored memory_cache_shared setting. All that resulted in an attempt to create
shared memory segments where none were needed or possible.
We now have a new Runners Registry (rrFinalizeConfig) that is dedicated to
finalizing complex configuration options, before those configurations options
are used to initialize modules, features, and services. As far as shared
memory is concerned, the initialization order is now:
* rrFinalizeConfig: finalize memory_cache_shared if needed
* rrClaimMemoryNeeds: request shared memory pages for the caches
* rrAfterConfig: create shared memory segments
The same bug 3449 also needed a fix for shared segment paths (trunk r11961).
Alex Rousskov [Fri, 27 Jan 2012 13:01:52 +0000 (06:01 -0700)]
Bug 3268: Squid cannot do anything else during ufs/diskd rebuild
Replaced 50-entries/step UFS rebuild logic with 50msec/step logic,
while also ensuring that other events can be processed between steps.
During a swap.state rebuild, progress reporting will now happen
every 4000 entries instead of every 0xFFF entries because humans do not
think in hex. Still no progress reporting for dirty rebuilds based on the
cache directory scans.
The code is based on Rock Store rebuild code. Eventually, the common rebuild
pacing logic among all stores should be merged to avoid code duplication.
Also removed RebuildState::next methods as unused. A copy-paste error?
Henrik Nordstrom [Fri, 27 Jan 2012 12:52:44 +0000 (05:52 -0700)]
Disable OpenSSL SSL/TLS bug workarounds by default
On a closer inspection the set of "harmless" SSL/TLS bug workarounds
set by SSL_OP_ALL is not all of them harmless and reduces the SSL/TLS
strength to some attacks.
To revert to the older mode the ALL option can be set explicitly, but
it's better to understand which bug is encountered and enable only that
specific workaround if needed.
Henrik Nordstrom [Fri, 27 Jan 2012 12:47:06 +0000 (05:47 -0700)]
Send DNS A and AAAA queries in parallel
This implements sending DNS A & AAAA queries in parallel by
creating "slave" idns_query requests. Current implementation
uses the A lookup as "master" and AAAA as "slave" query.
Long term this should probably be restructured to separate "lookup
state" and "query", or even better yet to defer the DNS lookups
until connect time and perform A respective AAAA as needed only and
not look up both before attemting to connect.
This also drops the dns_v4_fallback directive as it have no effect with
parallel DNS lookups. This directive should be reinstanciated in future
to enable pure IPv6 usage.
Inside StoreEntry::swapOut() the StoreEntry::trimMemory() method called to
release unused MemObjects memory. The trimMemory method must called for all
store entries, but current code blocks that call for not-swappable objects,
at least.
This patch trying to fix this bug implementing the following simple logic:
{
bool weAreOrMayBeSwappingOut =
swappingOut() || mayStartSwapout();
trimMemory(weAreOrMayBeSwappingOut);
if (!weAreOrMayBeSwappingOut)
return; // nothing else to do
}
Amos Jeffries [Fri, 27 Jan 2012 12:33:17 +0000 (05:33 -0700)]
Cache Manager migration support
* Add a little bit of XHR script to the CGI cachemgr front page which
probes each of the managed proxies for http:// and https:// capabilities
and produces web links to their internal managers.
* Reserve the template name MGR_INDEX for use by cachemgr scripts.
But do not distribute any preset template. This allows manager apps to
provide their own static template with linked scripts and objects.
* The error page system is updated to create a blanket message
indicating missing template instead of aborting Squid if a template is
not even installed.
Also, add Cross-Origin Request Support for the cache manager API
Now that tools are being implemented to access the cache manager via
http:// scheme we need to accomodate the browser XSS protection
mechanisms which limit XHR based scripts abilities.
This adds CORS headers to manager responses. Permitting XHR to view the
Server header (to detect squid version for known capabilities) and to
flag that the XHR request may need access to credentials for
authenticating with the manager.
This also closes the feature bug 3407 requesting we support the
non-standard "Origin:" header, which is used by the CORS mechanisms.
Future work:
Support the OPTIONS request used by CORS to detect requirements before
POSTing. We do not yet use POST in the Squid code so that is left until
needed.
Alex Rousskov [Sat, 21 Jan 2012 01:13:53 +0000 (18:13 -0700)]
Do not add HTTP 110 and 111 Warnings to TCP_REFRESH_UNMODIFIED responses.
The old "stale if hit" logic did not account for cases where the stored
stale response became fresh due to a successful revalidation with the
origin server.
When the stored response was stale at the time of the request, we were
adding 110 "Response is stale" and even 111 "Revalidation failed"
Warning headers to responses while logging TCP_REFRESH_UNMODIFIED, which
is considered a hit.
Amos Jeffries [Tue, 17 Jan 2012 08:34:49 +0000 (01:34 -0700)]
Cleanup: unlinkd is self-contained
This helper is very minimal. It does not use IP:: or Ipc:: symbols.
If we get linkage issues against it for them those need to be resolved
without pulling in dependencies from squid internal libraries.
Amos Jeffries [Sat, 7 Jan 2012 06:05:46 +0000 (23:05 -0700)]
SourceLayout: shuffle UserRequest into Auth:: namespace
No logic changes, symbols and debug only.
* shuffle AuthUserRequest to Auth::UserRequest
* shuffle AuthBasicUserRequest to Auth::Basic::UserRequest
* shuffle AuthDigestUserRequest to Auth::Digest::UserRequest
* shuffle AuthNegotiateUserRequest to Auth::Negotiate::UserRequest
* shuffle AuthNTLMUserRequest to Auth::NTLM::UserRequest
* polish and add documentation for several parts of the hierarchy
Amos Jeffries [Thu, 22 Dec 2011 07:34:08 +0000 (00:34 -0700)]
Bug 3370: external ACL sometimes skipping
Emit tag/user/log/message/pass details to the request in the case where
the external ACL entry has expired but within graceful revalidate period.
The result of this bug appears as incorrect matches later down the ACL
processing in any config relying on the external ACL output values.
Example; for bypassing auth login, or for filtering tagged traffic.
When the ssl_bump and authentication iare both enabled for an ssl-bumped port
all SSL enabled websites prompt the user for authentication information once
per FQDN.
This patch inherits the authentication info from the CONNECT request to the
sslbumped requests.
Bug 3448: 204 response problem in adaptation chains
When the first ICAP service in a chain respond with 204 the next service
is aborted on Must(old_request->canonical) expression inside Adaptation::Icap::ModXact::encapsulateHead method.
Squid ICAP try to set the request::canonical member of the adapted request
inside Adaptation::Icap::ModXact::prepEchoing when the 204 response received.
The adapted.header->parse(..) call some lines after will set canonical member
to NULL.
This patch calls the urlCanonical() function after parse() method
to build canonical member for the adapted request, instead of trying to copy
this member from the original request.
projects / thirdparty / squid.git / log
