cachemgr.cgi: use dynamic MemBuf for internal content generation
Using a fixed size buffer limits how big content lines can be. Modern
HTTP is fast reaching the point where such limits are problematic.
Also fixes incorrect uses of snprintf() by removing them.
Add chained certificates and signing certificate to peek-then-bumped connections.
The scenario this patch addresses is when Squid is configured with an
intermediate signing CA certificate, and clients have the root CA installed on
their machines. What happens is that the generated certificates come down with
an unknown issuer (the intermediate signing certificate), with no
intermediates, so they are rejected. By adding the configured certificate chain
as old client-first mode did, the intermediate and root certificates come down
as well, resulting in the issuer being identified and the connection being
established "securely".
This work is submitted on behalf of Bloomberg L.P.
Squid crashes on startup when the parent process exit()s after fork()ing
the kid process. Squid may also crash on shutdown after exiting main().
In both cases, the crashes are build- and environment-specific. Many
environments show no problems at all. Even disabling compiler
optimizations may prevent crashes. When crashes do happen, their
symptoms (e.g., backtrace) point to problems during destruction of
global objects, but the details point to innocent objects (e.g., PortCfg
or SSL_CTX).
In some environments, the following malloc error is printed on console
before the crash: "corrupted double-linked list".
This change replaces two StatHist globals used for SBuf statistics
collection with always-available singletons. The replaced globals could
be destructed before the last SBuf object using them, leading to memory
corruption (that would eventually crash Squid).
We should not expect the exact match because, as discovered during bug
3805 (r13947) fix, shared Segment::size() may exceed the originally
requested RAM amount due to stat() page rounding done by OSes like OS X.
Unfortunately, this rounding weakens the failed consistency check a lot.
TODO: Store the exact requested size and check against that as well.
Some servers cause an SSL handshake error with peek and splice.
The problem is related to the TLS Session Tickets extension handling. Squid
expects always a TLS Session Tickets extension, included in server hello
message, to assume that the ticket accepted and the session is a resumed
session, which is not always true.
Alex Rousskov [Tue, 12 Apr 2016 06:04:23 +0000 (18:04 +1200)]
Bug 4468: NotNode (!acl) naming: Terminate the name before strncat(name).
The fix may reduce or even eliminate garbage in logged ACL names (at
least). The bug was exposed by valgrind's "Conditional jump or move
depends on uninitialised value(s)" error.
However, pinger only drops setuid/setgid, and won't drop capabilities
after sockets are opened (when it is setuid-root, setuid(getuid()) also
drops capabilities, no code changes necessary; however, if it is only
setcap'ed, setuid() is no-op).
Fix is minimally tested, seems to work fine with both/either `setcap`
and `chmod u+s`; non-linux/non-libcap configurations should not be
affected).
Amos Jeffries [Wed, 23 Mar 2016 14:00:51 +0000 (03:00 +1300)]
RFC 7725: Add registry entry for 451 status text
While Squid does not generate these messages automatically we still have
to relay the status line text accurately, and admin may want to use it
for deny_info status.
Bug description:
- The client side and server side are finished
- On server side the Ftp::Relay::finalizeDataDownload() is called and
schedules the Ftp::Server::originDataCompletionCheckpoint
- On client side the "Ftp::Server::userDataCompletionCheckpoint" is
called. This is schedules a write to control connection and closes
data connection.
- The Ftp::Server::originDataCompletionCheckpoint is called which is
trying to write to control connection and the assertion triggered.
This bug is an corner case, where the client-side (FTP::Server) should
wait for the server side (Ftp::Client/Ftp::Relay) to finish its job before
respond to the FTP client. In this bug the existing mechanism, designed
to handle such problems, did not worked correctly and resulted to a double
write response to the client.
This patch try to fix the existing mechanism as follows:
- When Ftp::Server receives a "startWaitingForOrigin" callback, postpones
writting possible responses to the client and keeps waiting for the
stopWaitingForOrigin callback
- When the Ftp::Server receives a "stopWaitingForOrigin" callback,
resumes any postponed response.
- When the Ftp::Client starts working on a DATA-related transaction, calls the
Ftp::Server::startWaitingForOrigin callback
- When the Ftp::Client finishes its job or when its abort abnormaly, checks
whether it needs to call Ftp::Server::stopWaitingForOrigin callback.
- Also this patch try to fix the status code returned to the FTP client
taking in account the status code returned by FTP server. The
"Ftp::Server::stopWaitingForOrigin" is used to pass the returned status code
to the client side.
Squid crashes on shutdown while cleaning up idle ICAP connections.
The global Adaptation::Icap::TheConfig object is automatically
destroyed when Squid exits. Its destructor destroys Icap::ServiceRep
objects that, in turn, close all open connections in the idle
connections pool. Since this happens after comm_exit has destroyed all
Comm structures associated with those connections, Squid crases.
After certain failures, FwdState::retryOrBail() may be called twice,
once from FwdState::unregisterdServerEnd() [called from
HttpStateData::swanSong()] and once from the FwdState's own connection
close handler. This may result in two concurrent connections to the
remote server, followed by an assertion upon a connection closure.
This patch:
- After HttpStateData failures, instead of closing the squid-to-peer
connection directly (and, hence, triggering closure handlers), calls
HttpStateData::closeServer() and mustStop() for a cleaner exit with
fewer wasteful side effects and better debugging.
- Creates and remembers a FwdState close handler AsyncCall so that
comm_remove_close_handler() can cancel an already scheduled callback.
The conversion to the AsyncCall was necessary because legacy [close
handler callbacks] cannot be canceled once scheduled.
William Lima [Tue, 23 Feb 2016 15:47:10 +0000 (04:47 +1300)]
Bug 3870: assertion failed: String.cc: 'len_ + len <65536' in ESI::CustomParser
The custom ESI parser used in absence of libxml2 or libexpat parsers was
restricted to handling 64KB buffers but under some conditions could expand
to over 64KB during the parse process. Hitting this assertion.
TODO: the parser can now be redesigned to make use of Tokenizer and
CharacterSet parsing tools. But that is left for later work.
Alex Rousskov [Thu, 18 Feb 2016 04:15:33 +0000 (21:15 -0700)]
Better handling of huge response headers. Fewer "BUG 3279" messages.
When we failed to parse a response, do not store the fake half-baked
response (via a replaceHttpReply() call). Doing so leads to misleading
"BUG 3279: HTTP reply without Date" messages (at best). The fake
response is only meant for continueAfterParsingHeader().
Also removed a misleading XXX that may have caused Bug 4432 in v4.0
(trunk r14548).
In the case SSL errors detected by certificate validator helper the objects
stored in Ssl::ServerBump::sslErrors member and will never released.
This member normally points to an Ssl::CertErrors list attached to the related
SSL object which is responsible to release this list.
When the cert validator detects errors a new errors list created and attached
to the related Ssl::ServerBump::sslErrors member but the SSL objects still
hold the old one. The old list released but not the new one.
This patch also fixes the case the cbdata protected Ssl::CertErrors list,
still is used through the related Ssl::ServerBump object but it is not valid
any more, because the SSL object which hold it gone.
This patch instead of storing the Ssl::CertErrors list to Ssl::ServerBump
object stores the SSL object and increases its reference to avoid be released
Bug 4437: Fix Segfault on Certain SSL Handshake Errors
Squid after an unsuccesfull try to connect to the remote server may make two
concurrent retries to connect to the remote SSL server, calling twice the
FwdState::retryOrBail() method, which may result to unexpected behaviour.
Prevent this by just closing the connection to the remote SSL server inside
FwdState::connectedToPeer method on error and instead of calling the
FwdState::retryOrBail method, just allow comm_close handler to retry the
connection if required.
Markus Mayer [Sun, 31 Jan 2016 05:43:08 +0000 (18:43 +1300)]
Fix handling of shared memory left over by Squid crashes or bugs
A Squid instance may inherit an old shared memory segment from the
previous instance as the result of either a Squid crash or an at-exit
cleanup bug. This change fixes two problems triggered by old segments:
1. After an earlier OS X fix (bug 3805; trunk r13947), Squid stopped
initializing previously used shared memory. Uninitialzed memory
resulted in subtle bugs and crashes.
2. When called for an old Squid shared memory segment, OS X
ftruncate() fails with EINVAL, preventing Squid from starting when
the old segment is still around.
More specifically: Darwin ftruncate() calls pshm_truncate().
pshm_truncate() checks if the PSHM_ALLOCATED flag is already set on
the memory region. If the flag is set, the call fails with EINVAL.
Otherwise, pshm_truncate() sets PSHM_ALLOCATED for the region.
Since Squid must call ftruncate() to size every new segment, all
old Squid segments have that flag set, preventing ftruncate() calls
for old segments in newer Sqid instances from succeeding.
Fix invalid FTP connection handling on blocked content
FTP client gets stuck after the following chain of events:
* Client requests a file that will be blocked by ICAP.
* Squid starts downloading the file from the FTP server
and sends "150 Opening..." to the FTP client.
* Squid aborts the data connection with the FTP server
as soon as the ICAP service blocks it.
* Squid sends "451 Forbidden" to the FTP client.
* The FTP server sends "500 OOPS: setsockopt: linger" to Squid.
* Squid terminates the control connection to the FTP server.
* Squid establishes a new control connection to the FTP server
but does not authenticate itself.
* Further commands from the FTP client do not work any more.
The above and many similar problems exist because Squid handles
FTP client-to-squid and squid-to-FTP server data connections
independently from each other. In many cases, one connection does
not get notified about the problems with the other connection.
This patch:
- Add Ftp::MasterState::userDataDone to record received
the FTP client final response status code to sent (or to be send)
to the client.
- The Ftp::MasterState::waitForOriginData flag to hold status of the
squid-to-server side. If the squid-to-server side is not finishes
yet this is true.
- Send a control reply to the FTP client only after the data transfered
on both server and client sides.
- Split Client::abortTransaction to Client::abortOnData and to
Client::abortAll()
- Implement the Ftp::Relay::abortOnData() and Ftp::Relay::Abort()
(i.e., StoreEntry abort handler) to avoid closing the control
connection when the data connection is closed unexpectedly.
Alex Rousskov [Tue, 5 Jan 2016 05:36:22 +0000 (18:36 +1300)]
Reflect the [ugly] reality in external_acl_type cache=n documentation.
The patch does not change how the cache works, but may help admins
configure the cache correctly if they stumble upon the updated docs.
A typical external ACL cache with the default cache settings consumes
about 70 MB of RAM (more in many cases, e.g., if annotations are used
with external ACLs). Besides memory usage, the default cache is using
only 977 hash buckets for 262144 entries so there may be quite a bit
of linear search going on by default.
Avoid memory leaks when a certificate validator is used with SslBump
When a certificate validator was used, sslCrtvdHandleReplyWrapper delivered
validator response directly to the Ssl::PeerConnector job using job's
Ssl::CertValidationHelper::CVHCB callback. If that synchronous call happened
to be the last job call, then Ssl::PeerConnector::done() would become true
for the job, as it should, but nobody would notice that the PeerConnector
job object should be deleted, and the object would leak.
This fix converts CVHCB into an async job call to avoid direct, unprotected
job calls in this context.
Alex Rousskov [Fri, 18 Dec 2015 12:50:46 +0000 (01:50 +1300)]
Fix startup crash with a misconfigured (too-small) shared memory cache
Controller condition for allocating MemStore is slightly different from
MemStore condition for allocating MemStore::map, resulting in MemStore
without a map. Until that discrepancy is fixed, be careful when
dereferencing MemStore::map.
Fix connection retry and fallback after failed server TLS connections
FwdState should retry connect to the next ip after a Ssl::PeerConnector failure
When the Ssl::PeerConnector fails to establish an SSL connection FwdState does
not retry to connect to the next destination server ip address, but instead
returns an error.
Complete certificate chains using external intermediate certificates
... stored in sslproxy_foreign_intermediate_certs PEM file.
Many origin servers do not send complete certificate chains. Many
browsers use certificate extensions in the server certificate to
download the missing intermediate certificates automatically from
the Internet. Squid does not do that (yet?).
This patch adds the sslproxy_foreign_intermediate_certs configuration directive
to allow an admin to supply a file with intermediate certificates that
Squid may use to complete certificate chains. These intermediate
certificates are _not_ treated as trusted root certificates.
Alex Rousskov [Thu, 19 Nov 2015 05:08:41 +0000 (21:08 -0800)]
Stop using dangling pointers for eCAP-set custom HTTP reason phrases.
Squid still does not support [external] custom reason phrases and,
hence, cannot reliably support eCAP API that sets the reason phrase to
the one supplied by the adapter. This and r14398 changes fix [known]
regression bugs introduced by r12728 ("SourceLayout").
Alex Rousskov [Thu, 19 Nov 2015 05:05:49 +0000 (21:05 -0800)]
Fix status code-based HTTP reason phrase for eCAP-generated messages.
Calling .reason() on a not-yet-set theMessage.sline object resulted in
"Init" status reason phrase for all from-scratch (i.e., not cloned)
eCAP-made HTTP responses. This fix lets Squid compute the reason phrase
based on the status code, just like Squid does for forwarded responses
(IIRC).
Amos Jeffries [Thu, 19 Nov 2015 05:00:37 +0000 (21:00 -0800)]
Revert r13921: Migrate StoreEntry to using MEMPROXY_CLASS
This change has been identified as teh trigger for several object caching
errors. The real cause is not yet known, but reverting this optimisation
avoids it, so is being done for stability.
- Squid receives TLS Hello from the client (TCP connection A).
- Squid successfully negotiates an TLS connection with the origin server
(TCP connection B).
- Squid successfully negotiates an TLS connection with the client
(TCP connection A).
- Squid marks connection B as "idle" and waits an HTTP request from
connection A.
- The origin server continues talking to Squid (TCP connection B).
Squid detects a network read on an idle connection and closes TCP
connection B (and then the associated TCP connection A as well).
This patch:
- When squid detects a network read on server idle connection do an
SSL_read to:
a) see if application data received from server and abort in this case
b) detect possible TLS error, or TLS shutdown message from server
c) or ignore if only TLS protocol related packets received.
Stuart Henderson [Tue, 10 Nov 2015 09:30:19 +0000 (01:30 -0800)]
Fix SSL_get_certificate() problem detection
The autoconf check for SQUID_SSLGETCERTIFICATE_BUGGY fails on ssl library
builds which don't include SSLv3; as a result of the autoconf decision
this can end up triggering the assert(0) in Ssl::verifySslCertificate()
in ssl/support.cc (line 1712 in 3.5.11).
Amos Jeffries [Fri, 30 Oct 2015 18:26:41 +0000 (11:26 -0700)]
Add Locker friend class to SBuf for protection against memory issues
When appending or otherwise modifying an SBuf based on a SBuf& or char*
the parameter used may be pointing at the MemBlob memory buffer
indirectly without holding a separate ref-count lock to it.
If 'this' SBuf then requires reallocation for any reason the char* or
buffer pointer taken from the SBuf&, which is being manipulated may in
fact be left pointing at invalid memory.
Utilize a private Locker class to create relatively cheap ref-count locks
on the store_ MemBlob when this problem MAY occur. This Locker needs to
be used on all non-const SBuf methods accepting char* or SBuf& argument.
Alex Rousskov [Wed, 28 Oct 2015 03:47:56 +0000 (20:47 -0700)]
Connection stats, including %<lp, missing for persistent connections.
The code reusing a pconn was missing a hier.note() call, resulting in 0
values logged for %<lp (local port number of the last server or peer
connection) and probably other missing stats.
Also refactored poorly copied statistics collection code to remove
duplication and always update to-server connection stats when the actual
connection becomes available.
Positive side effect: Upon setsockopt(2) failures, the tos and nfmark
fields of a pinned connection were set to the desired (but not actually
applied) values, while persistent connection fields were left intact
(and, hence, stale). Both fields are now reset to zero on failures, for
both types of connections.
Aymeric Vincent [Tue, 27 Oct 2015 22:48:24 +0000 (15:48 -0700)]
Fix incorrect authentication headers on cache digest requests
login=NEGOTIATE can have an additional parameter specified,
like login=NEGOTIATE:xxx
One test added in rev.12714 does not take this case into account and it
will send a garbage "login:password" (== "NEGOTIATE:xxx") to its peer
when requesting a digest.
This is a workaround patch to remove the broken Authentication headers
entirely. Support for Negotiate to the peer on these digest requests is
still needed.
projects / thirdparty / squid.git / log
