[MINOR] add options dontlog-normal and log-separate-errors

Some big traffic sites have trouble dealing with logs and tend to
disable them. Here are two new options to help cope with massive
logs.

  - dontlog-normal only disables logging for 100% successful
    connections, other ones will still be logged

  - log-separate-errors will cause non-100% successful connections
    to be logged at level "err" instead of level "info" so that a
    properly configured syslog daemon can send them to a different
    file for longer conservation.

diff --git a/doc/configuration.txt b/doc/configuration.txt
index 03a58e4b016314e1c3a5e24027fed4e22b5bf120..762343f670b37619feb091685316a7b4e424feef 100644 (file)
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -600,6 +600,7 @@ monitor-uri                 X          X         X         -
 [no] option checkcache      X          -         X         X
 [no] option clitcpka        X          X         X         -
 [no] option contstats       X          X         X         -
+[no] option dontlog-normal  X          X         X         -
 [no] option dontlognull     X          X         X         -
 [no] option forceclose      X          -         X         X
 option forwardfor           X          X         X         X
@@ -608,6 +609,8 @@ option originalto           X          X         X         X
 option httpchk              X          -         X         X
 [no] option httpclose       X          X         X         X
 option httplog              X          X         X         X
+[no] option log-separate-
+            errors          X          X         X         -
 [no] option logasap         X          X         X         -
 [no] option nolinger        X          X         X         X
 [no] option persist         X          -         X         X
@@ -1999,6 +2002,29 @@ option contstats
   it is not enabled by default, as it has small performance impact (~0.5%).
 
 
+option dontlog-normal
+no option dontlog-normal
+  Enable or disable logging of normal, successful connections
+  May be used in sections :   defaults | frontend | listen | backend
+                                 yes   |    yes   |   yes  |   no
+  Arguments : none
+
+  There are large sites dealing with several thousand connections per second
+  and for which logging is a major pain. Some of them are even forced to turn
+  logs off and cannot debug production issues. Setting this option ensures that
+  normal connections, those which experience no error, no timeout, no retry nor
+  redispatch, will not be logged. This leaves disk space for anomalies. In HTTP
+  mode, the response status code is checked and return codes 5xx will still be
+  logged.
+
+  It is strongly discouraged to use this option as most of the time, the key to
+  complex issues is in the normal logs which will not be logged here. If you
+  need to separate logs, see the "log-separate-errors" option instead.
+
+  See also : "log", "dontlognull", "log-separate-errors" and section 2.6 about
+             logging.
+
+
 option dontlognull
 no option dontlognull
   Enable or disable logging of null connections
@@ -2300,6 +2326,29 @@ option httplog
   See also :  section 2.6 about logging.
 
 
+option log-separate-errors
+no option log-separate-errors
+  Change log level for non-completely successful connections
+  May be used in sections :   defaults | frontend | listen | backend
+                                 yes   |    yes   |   yes  |   no
+  Arguments : none
+
+  Sometimes looking for errors in logs is not easy. This option makes haproxy
+  raise the level of logs containing potentially interesting information such
+  as errors, timeouts, retries, redispatches, or HTTP status codes 5xx. The
+  level changes from "info" to "err". This makes it possible to log them
+  separately to a different file with most syslog daemons. Be careful not to
+  remove them from the original file, otherwise you would lose ordering which
+  provides very important information.
+
+  Using this option, large sites dealing with several thousand connections per
+  second may log normal traffic to a rotating buffer and only archive smaller
+  error logs.
+
+  See also : "log", "dontlognull", "dontlog-normal" and section 2.6 about
+             logging.
+
+
 option logasap
 no option logasap
   Enable or disable early logging of HTTP requests
@@ -4979,9 +5028,10 @@ emitted earlier if "option logasap" is specified, which makes sense in most
 environments with long sessions such as remote terminals. Sessions which match
 the "monitor" rules are never logged. It is also possible not to emit logs for
 sessions for which no data were exchanged between the client and the server, by
-specifying "option dontlognull" in the frontend. A few fields may slightly vary
-depending on some configuration options, those are marked with a star ('*')
-after the field name below.
+specifying "option dontlognull" in the frontend. Successful connections will
+not be logged if "option dontlog-normal" is specified in the frontend. A few
+fields may slightly vary depending on some configuration options, those are
+marked with a star ('*') after the field name below.
 
   Example :
         frontend fnt
@@ -5141,7 +5191,8 @@ emitted at the end of the session, unless "option logasap" is specified, which
 generally only makes sense for download sites. A session which matches the
 "monitor" rules will never logged. It is also possible not to log sessions for
 which no data were sent by the client by specifying "option dontlognull" in the
-frontend.
+frontend. Successful connections will not be logged if "option dontlog-normal"
+is specified in the frontend.
 
 Most fields are shared with the TCP log, some being different. A few fields may
 slightly vary depending on some configuration options. Those ones are marked
@@ -5419,6 +5470,35 @@ with normal logs, the total time field and the number of bytes are prefixed
 with a '+' sign which means that real numbers are certainly larger.
 
 
+2.6.3.3) Raising log level upon errors
+--------------------------------------
+
+Sometimes it is more convenient to separate normal traffic from errors logs,
+for instance in order to ease error monitoring from log files. When the option
+"log-separate-errors" is used, connections which experience errors, timeouts,
+retries, redispatches or HTTP status codes 5xx will see their syslog level
+raised from "info" to "err". This will help a syslog daemon store the log in
+a separate file. It is very important to keep the errors in the normal traffic
+file too, so that log ordering is not altered. You should also be careful if
+you already have configured your syslog daemon to store all logs higher than
+"notice" in an "admin" file, because the "err" level is higher than "notice".
+
+
+2.6.3.4) Disabling logging of successful connections
+----------------------------------------------------
+
+Although this may sound strange at first, some large sites have to deal with
+multiple thousands of logs per second and are experiencing difficulties keeping
+them intact for a long time or detecting errors within them. If the option
+"dontlog-normal" is set on the frontend, all normal connections will not be
+logged. In this regard, a normal connection is defined as one without any
+error, timeout, retry nor redispatch. In HTTP, the status code is checked too,
+and a response with a status 5xx is not considered normal and will be logged
+too. Of course, doing is is really discouraged as it will remove most of the
+useful information from the logs. Do this only if you have no other
+alternative.
+
+
 2.6.4) Timing events
 --------------------
 

diff --git a/include/types/proxy.h b/include/types/proxy.h
index 924019cb71392a4a0f07656d51603b1f17588b5a..48d0a82cc4253290dfd0b93c88dd81dd3f8afcda 100644 (file)
--- a/include/types/proxy.h
+++ b/include/types/proxy.h
@@ -115,6 +115,8 @@
 #define PR_O2_SPLIC_ANY        (PR_O2_SPLIC_REQ|PR_O2_SPLIC_RTR|PR_O2_SPLIC_AUT)
 #define PR_O2_REQBUG_OK        0x00000008      /* let buggy requests pass through */
 #define PR_O2_RSPBUG_OK        0x00000010      /* let buggy responses pass through */
+#define PR_O2_NOLOGNORM        0x00000020      /* don't log normal traffic, only errors and retries */
+#define PR_O2_LOGERRORS        0x00000040      /* log errors and retries at level LOG_ERR */
 
 /* This structure is used to apply fast weighted round robin on a server group */
 struct fwrr_group {

diff --git a/src/cfgparse.c b/src/cfgparse.c
index 58e8407c6a4d003cdff26a493ab48ffbcfbe6e2a..6a4744e531f2189cd58324fcf81e3ac3ba3326d5 100644 (file)
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -125,6 +125,8 @@ static const struct cfg_opt cfg_opts2[] =
 #endif
        { "accept-invalid-http-request",  PR_O2_REQBUG_OK, PR_CAP_FE, 0 },
        { "accept-invalid-http-response", PR_O2_RSPBUG_OK, PR_CAP_BE, 0 },
+       { "dontlog-normal",               PR_O2_NOLOGNORM, PR_CAP_FE, 0 },
+       { "log-separate-errors",          PR_O2_LOGERRORS, PR_CAP_FE, 0 },
        { NULL, 0, 0, 0 }
 };
 

diff --git a/src/log.c b/src/log.c
index 976bc63dfd31c9668123da0dbfaf21f27029ae5d..bc9fd0a4b8be33df23a2b87bc2294963cad33bbd 100644 (file)
--- a/src/log.c
+++ b/src/log.c
@@ -321,10 +321,15 @@ void tcp_sess_log(struct session *s)
        struct proxy *fe = s->fe;
        struct proxy *be = s->be;
        struct proxy *prx_log;
-       int tolog;
+       int tolog, level, err;
        char *svid;
        struct tm tm;
 
+       /* if we don't want to log normal traffic, return now */
+       err = (s->flags & (SN_ERR_MASK | SN_REDISP)) || (s->conn_retries != be->conn_retries);
+       if (!err && (fe->options2 & PR_O2_NOLOGNORM))
+               return;
+
        if (s->cli_addr.ss_family == AF_INET)
                inet_ntop(AF_INET,
                          (const void *)&((struct sockaddr_in *)&s->cli_addr)->sin_addr,
@@ -353,7 +358,11 @@ void tcp_sess_log(struct session *s)
        tolog = fe->to_log;
        svid = (tolog & LW_SVID) ? (s->srv != NULL) ? s->srv->id : "<NOSRV>" : "-";
 
-       send_log(prx_log, LOG_INFO, "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
+       level = LOG_INFO;
+       if (err && (fe->options2 & PR_O2_LOGERRORS))
+               level = LOG_ERR;
+
+       send_log(prx_log, level, "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
                 " %s %s/%s %ld/%ld/%s%ld %s%lld"
                 " %c%c %d/%d/%d/%d/%s%u %ld/%ld\n",
                 pn,

diff --git a/src/proto_http.c b/src/proto_http.c
index 6ffc14e48ce11a8d114a09f78e58bfb5aeabf800..a8612a4f10c954a7572e297872d3090d1c3cbc64 100644 (file)
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -755,7 +755,7 @@ void http_sess_log(struct session *s)
        struct proxy *be = s->be;
        struct proxy *prx_log;
        struct http_txn *txn = &s->txn;
-       int tolog;
+       int tolog, level, err;
        char *uri, *h;
        char *svid;
        struct tm tm;
@@ -763,6 +763,13 @@ void http_sess_log(struct session *s)
        int t_request;
        int hdr;
 
+       /* if we don't want to log normal traffic, return now */
+       err = (s->flags & (SN_ERR_MASK | SN_REDISP)) ||
+               (s->conn_retries != be->conn_retries) ||
+               txn->status >= 500;
+       if (!err && (fe->options2 & PR_O2_NOLOGNORM))
+               return;
+
        if (fe->logfac1 < 0 && fe->logfac2 < 0)
                return;
        prx_log = fe;
@@ -830,7 +837,11 @@ void http_sess_log(struct session *s)
        if (tv_isge(&s->logs.tv_request, &s->logs.tv_accept))
                t_request = tv_ms_elapsed(&s->logs.tv_accept, &s->logs.tv_request);
 
-       send_log(prx_log, LOG_INFO,
+       level = LOG_INFO;
+       if (err && (fe->options2 & PR_O2_LOGERRORS))
+               level = LOG_ERR;
+
+       send_log(prx_log, level,
                 "%s:%d [%02d/%s/%04d:%02d:%02d:%02d.%03d]"
                 " %s %s/%s %d/%ld/%ld/%ld/%s%ld %d %s%lld"
                 " %s %s %c%c%c%c %d/%d/%d/%d/%s%u %ld/%ld%s\n",