<H2><A NAME="toc2">2.</A> <A HREF="#s2">Major new features since Squid-3.5</A></H2>
<UL>
-<LI><A NAME="toc2.1">2.1</A> <A HREF="#ss2.1">BLAH</A>
+<LI><A NAME="toc2.1">2.1</A> <A HREF="#ss2.1">Configurable helper queue size</A>
</UL>
<P>
<H2><A NAME="toc3">3.</A> <A HREF="#s3">Changes to squid.conf since Squid-3.5</A></H2>
<P>Most user-facing changes are reflected in squid.conf (see below).</P>
-<H2><A NAME="ss2.1">2.1</A> <A HREF="#toc2.1">BLAH</A>
+<H2><A NAME="ss2.1">2.1</A> <A HREF="#toc2.1">Configurable helper queue size</A>
</H2>
-<P>Details at
-<A HREF="http://wiki.squid-cache.org/Features/BLAH">http://wiki.squid-cache.org/Features/BLAH</A>.</P>
-
+<P>The new queue-size=N option to helpers configuration, allows users
+to configure the maximum number of queued requests to busy helpers.</P>
<H2><A NAME="s3">3.</A> <A HREF="#toc3">Changes to squid.conf since Squid-3.5</A></H2>
<P>
<DL>
+<DT><B> auth_param </B><DD>
+<P> New parameter <EM>queue-size=</EM> to set the maximum number
+of queued requests.</P>
+
+<DT><B>external_acl_type</B><DD>
+
+<DT><B></B><DD>
+<P> New parameter <EM>queue-size=</EM> to set the maximum number
+of queued requests.</P>
+
+<DT><B>url_rewrite_children</B><DD>
+
+<DT><B></B><DD>
+<P> New parameter <EM>queue-size=</EM> to set the maximum number
+of queued requests.</P>
+
+<DT><B>sslcrtd_children</B><DD>
+<P> New parameter <EM>queue-size=</EM> to set the maximum number
+of queued requests.</P>
+<DT><B>sslcrtvalidator_children</B><DD>
+<P> New parameter <EM>queue-size=</EM> to set the maximum number
+of queued requests.</P>
</DL>
</P>
Most user-facing changes are reflected in squid.conf (see below).
-<sect1>BLAH
-<p>Details at <url url="http://wiki.squid-cache.org/Features/BLAH">.
-
+<sect1>Configurable helper queue size
+<p>The new queue-size=N option to helpers configuration, allows users
+to configure the maximum number of queued requests to busy helpers.
<sect>Changes to squid.conf since Squid-3.5
<p>
<sect1>Changes to existing tags<label id="modifiedtags">
<p>
<descrip>
+ <tag> auth_param </tag>
+ <p> New parameter <em>queue-size=</em> to set the maximum number
+ of queued requests.
+
+ <tag>external_acl_type<tag>
+ <p> New parameter <em>queue-size=</em> to set the maximum number
+ of queued requests.
+
+ <tag>url_rewrite_children<tag>
+ <p> New parameter <em>queue-size=</em> to set the maximum number
+ of queued requests.
+
+ <tag>sslcrtd_children</tag>
+ <p> New parameter <em>queue-size=</em> to set the maximum number
+ of queued requests.
+ <tag>sslcrtvalidator_children</tag>
+ <p> New parameter <em>queue-size=</em> to set the maximum number
+ of queued requests.
</descrip>
<sect1>Removed tags<label id="removedtags">
For Digest there is no default, this parameter is mandatory.
For NTLM and Negotiate this parameter is ignored.
- "children" numberofchildren [startup=N] [idle=N] [concurrency=N]
+ "children" numberofchildren [startup=N] [idle=N] [concurrency=N] [queue-size=N]
The maximum number of authenticator processes to spawn. If
you start too few Squid will have to wait for them to process
Concurrency must not be set unless it's known the helper
supports the input format with channel-ID fields.
+ The queue-size= option sets the maximum number of queued
+ requests. If the queued requests exceed queue size for more
+ than 3 minutes then squid aborts its operation.
+ The default value is set to 2*numberofchildren/
+
NOTE: NTLM and Negotiate schemes do not support concurrency
in the Squid code module even though some helpers can.
Up to the value of children-max. (default 1)
concurrency=n concurrency level per process. Only used with helpers
capable of processing more than one query at a time.
+ queue-size=N The queue-size= option sets the maximum number of queued
+ requests. If the queued requests exceed queue size
+ the acl ignored.
+ The default value is set to 2*children-max.
cache=n limit the result cache size, default is 262144.
grace=n Percentage remaining of TTL where a refresh of a
cached entry should be initiated without needing to
at all times. When traffic begins to rise above what the existing
processes can handle this many more will be spawned up to the maximum
configured. A minimum setting of 1 is required.
+
+ queue-size=N
+
+ Sets the maximum number of queued requests.
+ If the queued requests exceed queue size for more than 3 minutes
+ squid aborts its operation.
+ The default value is set to 2*numberofchildren.
You must have at least one ssl_crtd process.
DOC_END
a request ID in front of the request/response. The request
ID from the request must be echoed back with the response
to that request.
+
+ queue-size=N
+
+ Sets the maximum number of queued requests.
+ If the queued requests exceed queue size for more than 3 minutes
+ squid aborts its operation.
+ The default value is set to 2*numberofchildren.
You must have at least one ssl_crt_validator process.
DOC_END
used to communicate with the helper is modified to include
an ID in front of the request/response. The ID from the request
must be echoed back with the response to that request.
+
+ queue-size=N
+
+ Sets the maximum number of queued requests.
+ If the queued requests exceed queue size and redirector_bypass
+ configuration option is set, then redirector is bypassed. Otherwise, if
+ overloading persists squid may abort its operation.
+ The default value is set to 2*numberofchildren.
DOC_END
NAME: url_rewrite_host_header redirect_rewrites_host_header
redirectors for access control, and you enable this option,
users may have access to pages they should not
be allowed to request.
+ This options sets default queue-size option of the url_rewrite_children
+ to 0.
DOC_END
NAME: url_rewrite_extras
used to communicate with the helper is modified to include
an ID in front of the request/response. The ID from the request
must be echoed back with the response to that request.
+
+ queue-size=N
+
+ Sets the maximum number of queued requests.
+ If the queued requests exceed queue size and store_id_bypass
+ configuration option is set, then storeID helper is bypassed. Otherwise,
+ if overloading persists squid may abort its operation.
+ The default value is set to 2*numberofchildren.
DOC_END
NAME: store_id_access storeurl_rewrite_access
are not critical to your caching system. If you use
helpers for critical caching components, and you enable this
option, users may not get objects from cache.
+ This options sets default queue-size option of the store_id_children
+ to 0.
DOC_END
COMMENT_START
a->children.n_idle = atoi(token + 14);
} else if (strncmp(token, "concurrency=", 12) == 0) {
a->children.concurrency = atoi(token + 12);
+ } else if (strncmp(token, "queue-size=", 11) == 0) {
+ a->children.queue_size = atoi(token + 11);
+ a->children.defaultQueueSize = false;
} else if (strncmp(token, "cache=", 6) == 0) {
a->cache_size = atoi(token + 6);
} else if (strncmp(token, "grace=", 6) == 0) {
if (a->negative_ttl == -1)
a->negative_ttl = a->ttl;
+ if (a->children.queue_size < 0)
+ a->children.queue_size = 2 * a->children.n_max;
+
/* Parse format */
external_acl_format::Pointer *p = &a->format;
if (!entry) {
debugs(82, 2, HERE << acl->def->name << "(\"" << key << "\") = lookup needed");
- if (acl->def->theHelper->stats.queue_size < (int)acl->def->theHelper->childs.n_active) {
+ if (!acl->def->theHelper->queueFull()) {
debugs(82, 2, HERE << "\"" << key << "\": queueing a call.");
if (!ch->goAsync(ExternalACLLookup::Instance()))
debugs(82, 2, "\"" << key << "\": no async support!");
} else {
/* No pending lookup found. Sumbit to helper */
- /* Check for queue overload */
-
- if (def->theHelper->stats.queue_size >= (int)def->theHelper->childs.n_running) {
- debugs(82, 7, HERE << "'" << def->name << "' queue is too long");
- assert(inBackground); // or the caller should have checked
- cbdataFree(state);
- return;
- }
-
- /* Send it off to the helper */
MemBuf buf;
buf.init();
debugs(82, 4, "externalAclLookup: looking up for '" << key << "' in '" << def->name << "'.");
- helperSubmit(def->theHelper, buf.buf, externalAclHandleReply, state);
+ if (!def->theHelper->trySubmit(buf.buf, externalAclHandleReply, state)) {
+ debugs(82, 7, HERE << "'" << def->name << "' submit to helper failed");
+ assert(inBackground); // or the caller should have checked
+ cbdataFree(state);
+ return;
+ }
dlinkAdd(state, &state->list, &def->queue);
callback(data, nilReply);
return;
}
+ hlp->prepSubmit();
+ hlp->submit(buf, callback, data);
+}
+
+bool
+helper::queueFull() const {
+ return stats.queue_size > static_cast<int>(childs.queue_size);
+}
+
+/// prepares the helper for request submission via trySubmit() or helperSubmit()
+/// currently maintains full_time and kills Squid if the helper remains full for too long
+void
+helper::prepSubmit()
+{
+ if (!queueFull())
+ full_time = 0;
+ else if (!full_time) // may happen here if reconfigure decreases capacity
+ full_time = squid_curtime;
+ else if (squid_curtime - full_time > 180)
+ fatalf("Too many queued %s requests", id_name);
+}
+
+bool
+helper::trySubmit(const char *buf, HLPCB * callback, void *data)
+{
+ prepSubmit();
+ if (queueFull()) {
+ debugs(84, DBG_IMPORTANT, id_name << " drops request due to a full queue");
+ return false; // request was ignored
+ }
+
+ submit(buf, callback, data); // will send or queue
+ return true; // request submitted or queued
+}
+
+/// dispatches or enqueues a helper requests; does not enforce queue limits
+void
+helper::submit(const char *buf, HLPCB * callback, void *data)
+{
Helper::Request *r = new Helper::Request(callback, data, buf);
helper_server *srv;
- if ((srv = GetFirstAvailable(hlp)))
+ if ((srv = GetFirstAvailable(this)))
helperDispatch(srv, r);
else
- Enqueue(hlp, r);
+ Enqueue(this, r);
debugs(84, DBG_DATA, Raw("buf", buf, strlen(buf)));
+
+ if (!queueFull()) {
+ full_time = 0;
+ } else if (!full_time) {
+ debugs(84, 3, id_name << " queue became full");
+ full_time = squid_curtime;
+ }
}
/// lastserver = "server last used as part of a reserved request sequence"
callback(data, nilReply);
return;
}
+ hlp->prepSubmit();
+ hlp->submit(buf, callback, data, lastserver);
+}
+void statefulhelper::submit(const char *buf, HLPCB * callback, void *data, helper_stateful_server * lastserver)
+{
Helper::Request *r = new Helper::Request(callback, data, buf);
if ((buf != NULL) && lastserver) {
helperStatefulDispatch(lastserver, r);
} else {
helper_stateful_server *srv;
- if ((srv = StatefulGetFirstAvailable(hlp))) {
+ if ((srv = StatefulGetFirstAvailable(this))) {
helperStatefulDispatch(srv, r);
} else
- StatefulEnqueue(hlp, r);
+ StatefulEnqueue(this, r);
}
debugs(84, DBG_DATA, "placeholder: '" << r->placeholder <<
"', " << Raw("buf", buf, (!buf?0:strlen(buf))));
+
+ if (!queueFull()) {
+ full_time = 0;
+ } else if (!full_time) {
+ debugs(84, 3, id_name << " queue became full");
+ full_time = squid_curtime;
+ }
}
/**
}
}
+/// Handles a request when all running helpers, if any, are busy.
static void
Enqueue(helper * hlp, Helper::Request * r)
{
return;
}
- if (hlp->stats.queue_size < (int)hlp->childs.n_running)
+ if (hlp->stats.queue_size < (int)hlp->childs.queue_size)
return;
if (squid_curtime - hlp->last_queue_warn < 600)
debugs(84, DBG_CRITICAL, "WARNING: All " << hlp->childs.n_active << "/" << hlp->childs.n_max << " " << hlp->id_name << " processes are busy.");
debugs(84, DBG_CRITICAL, "WARNING: " << hlp->stats.queue_size << " pending requests queued");
debugs(84, DBG_CRITICAL, "WARNING: Consider increasing the number of " << hlp->id_name << " processes in your config file.");
-
- if (hlp->stats.queue_size > (int)hlp->childs.n_running * 2)
- fatalf("Too many queued %s requests", hlp->id_name);
}
static void
return;
}
- if (hlp->stats.queue_size < (int)hlp->childs.n_running)
+ if (hlp->stats.queue_size < (int)hlp->childs.queue_size)
return;
- if (hlp->stats.queue_size > (int)hlp->childs.n_running * 2)
- fatalf("Too many queued %s requests", hlp->id_name);
-
if (squid_curtime - hlp->last_queue_warn < 600)
return;
#include "helper/forward.h"
#include "ip/Address.h"
+/**
+ * Managers a set of individual helper processes with a common queue of requests.
+ *
+ * With respect to load, a helper goes through these states (roughly):
+ * idle: no processes are working on requests (and no requests are queued);
+ * normal: some, but not all processes are working (and no requests are queued);
+ * busy: all processes are working (and some requests are possibly queued);
+ * full: all processes are working and at least 2*#processes requests are queued.
+ *
+ * A "busy" helper queues new requests and issues a WARNING every 10 minutes or so.
+ * A "full" helper either drops new requests or keeps queuing them, depending on
+ * whether the caller can handle dropped requests (trySubmit vs helperSubmit APIs).
+ * An attempt to use a "full" helper that has been "full" for 3+ minutes kills worker.
+ * Given enough load, all helpers except for external ACL will make such attempts.
+ */
class helper
{
CBDATA_CLASS(helper);
cmdline(NULL),
id_name(name),
ipc_type(0),
+ full_time(0),
last_queue_warn(0),
last_restart(0),
eom('\n') {
}
~helper();
+ ///< whether at least one more request can be successfully submitted
+ bool queueFull() const;
+
+ ///< If not full, submit request. Otherwise, either kill Squid or return false.
+ bool trySubmit(const char *buf, HLPCB * callback, void *data);
+
public:
wordlist *cmdline;
dlink_list servers;
Helper::ChildConfig childs; ///< Configuration settings for number running.
int ipc_type;
Ip::Address addr;
+ time_t full_time; ///< when a full helper became full (zero for non-full helpers)
time_t last_queue_warn;
time_t last_restart;
char eom; ///< The char which marks the end of (response) message, normally '\n'
int queue_size;
int avg_svc_time;
} stats;
+
+protected:
+ friend void helperSubmit(helper * hlp, const char *buf, HLPCB * callback, void *data);
+ void prepSubmit();
+ void submit(const char *buf, HLPCB * callback, void *data);
};
class statefulhelper : public helper
MemAllocator *datapool;
HLPSAVAIL *IsAvailable;
HLPSONEQ *OnEmptyQueue;
+
+private:
+ friend void helperStatefulSubmit(statefulhelper * hlp, const char *buf, HLPCB * callback, void *data, helper_stateful_server * lastserver);
+ void submit(const char *buf, HLPCB * callback, void *data, helper_stateful_server *lastserver);
};
/**
n_idle(1),
concurrency(0),
n_running(0),
- n_active(0)
+ n_active(0),
+ queue_size(0),
+ defaultQueueSize(true)
{}
Helper::ChildConfig::ChildConfig(const unsigned int m):
n_idle(1),
concurrency(0),
n_running(0),
- n_active(0)
+ n_active(0),
+ queue_size(2 * m),
+ defaultQueueSize(true)
{}
Helper::ChildConfig &
n_startup = rhs.n_startup;
n_idle = rhs.n_idle;
concurrency = rhs.concurrency;
+ queue_size = rhs.queue_size;
+ defaultQueueSize = rhs.defaultQueueSize;
return *this;
}
}
} else if (strncmp(token, "concurrency=", 12) == 0) {
concurrency = xatoui(token + 12);
+ } else if (strncmp(token, "queue-size=", 11) == 0) {
+ queue_size = xatoui(token + 11);
+ defaultQueueSize = false;
} else {
debugs(0, DBG_PARSE_NOTE(DBG_IMPORTANT), "ERROR: Undefined option: " << token << ".");
self_destruct();
debugs(0, DBG_CRITICAL, "WARNING OVERIDE: Capping idle=" << n_idle << " to the defined maximum (" << n_max <<")");
n_idle = n_max;
}
+
+ if (defaultQueueSize)
+ queue_size = 2 * n_max;
}
* This includes both idle and in-use children.
*/
unsigned int n_active;
+
+ /**
+ * The requests queue size. By default it is of size 2*n_max
+ */
+ unsigned int queue_size;
+
+ /**
+ * True if the default queue size is used.
+ * Needed in the cases where we need to adjust default queue_size in
+ * special configurations, for example when redirector_bypass is used.
+ */
+ bool defaultQueueSize;
};
} // namespace Helper
assert(handler);
debugs(61, 5, "redirectStart: '" << http->uri << "'");
- if (Config.onoff.redirector_bypass && redirectors->stats.queue_size) {
- /* Skip redirector if there is one request queued */
+ if (Config.onoff.redirector_bypass && redirectors->queueFull()) {
+ /* Skip redirector if the queue is full */
++redirectorBypassed;
Helper::Reply bypassReply;
bypassReply.result = Helper::Okay;
assert(handler);
debugs(61, 5, "storeIdStart: '" << http->uri << "'");
- if (Config.onoff.store_id_bypass && storeIds->stats.queue_size) {
- /* Skip StoreID Helper if there is one request queued */
+ if (Config.onoff.store_id_bypass && storeIds->queueFull()) {
+ /* Skip StoreID Helper if the queue is full */
++storeIdBypassed;
Helper::Reply bypassReply;
redirectors->cmdline = Config.Program.redirect;
+ // BACKWARD COMPATIBILITY:
+ // if redirectot_bypass is set then use queue_size=0 as default size
+ if (Config.onoff.redirector_bypass && Config.redirectChildren.defaultQueueSize)
+ Config.redirectChildren.queue_size = 0;
+
redirectors->childs.updateLimits(Config.redirectChildren);
redirectors->ipc_type = IPC_STREAM;
storeIds->cmdline = Config.Program.store_id;
+ // BACKWARD COMPATIBILITY:
+ // if store_id_bypass is set then use queue_size=0 as default size
+ if (Config.onoff.store_id_bypass && Config.storeIdChildren.defaultQueueSize)
+ Config.storeIdChildren.queue_size = 0;
+
storeIds->childs.updateLimits(Config.storeIdChildren);
storeIds->ipc_type = IPC_STREAM;
void Ssl::Helper::sslSubmit(CrtdMessage const & message, HLPCB * callback, void * data)
{
- static time_t first_warn = 0;
assert(ssl_crtd);
- if (ssl_crtd->stats.queue_size >= (int)(ssl_crtd->childs.n_running * 2)) {
- if (first_warn == 0)
- first_warn = squid_curtime;
- if (squid_curtime - first_warn > 3 * 60)
- fatal("SSL servers not responding for 3 minutes");
- debugs(34, DBG_IMPORTANT, HERE << "Queue overload, rejecting");
+ std::string msg = message.compose();
+ msg += '\n';
+ if (!ssl_crtd->trySubmit(msg.c_str(), callback, data)) {
::Helper::Reply failReply;
failReply.result = ::Helper::BrokenHelper;
failReply.notes.add("message", "error 45 Temporary network problem, please retry later");
callback(data, failReply);
return;
}
-
- first_warn = 0;
- std::string msg = message.compose();
- msg += '\n';
- helperSubmit(ssl_crtd, msg.c_str(), callback, data);
}
#endif //USE_SSL_CRTD
void Ssl::CertValidationHelper::sslSubmit(Ssl::CertValidationRequest const &request, Ssl::CertValidationHelper::CVHCB * callback, void * data)
{
- static time_t first_warn = 0;
assert(ssl_crt_validator);
- if (ssl_crt_validator->stats.queue_size >= (int)(ssl_crt_validator->childs.n_running * 2)) {
- if (first_warn == 0)
- first_warn = squid_curtime;
- if (squid_curtime - first_warn > 3 * 60)
- fatal("ssl_crtvd queue being overloaded for long time");
- debugs(83, DBG_IMPORTANT, "WARNING: ssl_crtvd queue overload, rejecting");
- Ssl::CertValidationResponse resp;
- resp.resultCode = ::Helper::BrokenHelper;
- callback(data, resp);
- return;
- }
- first_warn = 0;
-
Ssl::CertValidationMsg message(Ssl::CrtdMessage::REQUEST);
message.setCode(Ssl::CertValidationMsg::code_cert_validate);
message.composeRequest(request);
delete crtdvdData;
return;
}
- helperSubmit(ssl_crt_validator, crtdvdData->query.c_str(), sslCrtvdHandleReplyWrapper, crtdvdData);
+
+ if (!ssl_crt_validator->trySubmit(crtdvdData->query.c_str(), sslCrtvdHandleReplyWrapper, crtdvdData)) {
+ Ssl::CertValidationResponse resp;
+ resp.resultCode = ::Helper::BrokenHelper;
+ callback(data, resp);
+
+ cbdataReferenceDone(crtdvdData->data);
+ SSL_free(crtdvdData->ssl);
+ delete crtdvdData;
+ return;
+ }
}
projects / thirdparty / squid.git / commitdiff
