New: Support for any xferlog files for FTP analysis.

diff --git a/docs/awstats_config.html b/docs/awstats_config.html
index d56d5b1d738e039c4e612b04befc88daa4da8859..7087811b60f6f0259a2292eb2b906c1856d99453 100644 (file)
--- a/docs/awstats_config.html
+++ b/docs/awstats_config.html
@@ -326,7 +326,7 @@ when reading it), follow the example:<br>
 <br>#   %logname          Authenticated login/user with format: alex
 <br>#   %time1            Date and time with format: [dd/mon/yyyy:hh:mm:ss +0000] or [dd/mon/yyyy:hh:mm:ss]
 <br>#   %time2            Date and time with format: yyyy-mm-dd hh-mm-ss
-<br>#   %time3            Date and time with format: Mon dd hh:mm:ss
+<br>#   %time3            Date and time with format: Mon dd hh:mm:ss or Mon dd hh:mm:ss yyyy
 <br>#   %time4            Date and time with unix timestamp format: dddddddddd
 <br>#   %methodurl        Method and URL with format: "GET /index.html HTTP/x.x"
 <br>#   %methodurlnoprot  Method and URL with format: "GET /index.html"

diff --git a/docs/awstats_faq.html b/docs/awstats_faq.html
index 0576de40228c74872cca69b44443fa22fe8c572f..06f347d776c0466931d1c84e765bb24ce26e8e13 100644 (file)
--- a/docs/awstats_faq.html
+++ b/docs/awstats_faq.html
@@ -49,7 +49,7 @@ Here, you can find the most common questions and answers about AWStats setup/usa
 <ul>
 FAQ-COM025 <a href="#NOLOG">How to use AWStats with no server log</a><br>
 FAQ-COM050 <a href="#LIMITLOG">What is the log size limit AWStats can analyze ?</a><br>
-FAQ-COM090 <a href="#FTP">Setup for FTP server log files.</a><br>
+FAQ-COM090 <a href="#FTP">Setup for FTP server log files (proftpd, vsftpd, ...).</a><br>
 FAQ-COM100 <a href="#MAIL">Setup for MAIL log files (Postfix, Sendmail, QMail, MDaemon, Exchange).</a><br>
 FAQ-COM110 <a href="#MEDIASERVER">Setup for MEDIA SERVER log files (Realmedia, Windows media, Darwin streaming server).</a><br>
 FAQ-COM120 <a href="#ROTATE">How to rotate my logs without loosing data.</a><br>
@@ -121,7 +121,7 @@ Examples of used platforms (bold means 'tested by author', others were reported
 <b>Apache 1.3.x and 2.x</b>, <b>IIS 5.0 and 6.0</b>, WebStar, WebLogic, WebSite, <b>Windows Media Server</b>, Tomcat, <b>Squid</b>,
 Sambar, Roxen, Resin, RealMedia server, Oracle9iAS, <b>Lotus Notes/Domino</b>, Darwin, IPlanet, IceCast, ZeroBrand, Zeus, Zope, Abyss...<br>
 <u>FTP servers</u><br>
-<b>ProFTP</b>, ...<br>
+<b>ProFTPd</b>, vsFTPd...<br>
 <u>Mails servers</u><br>
 <b>Postfix</b>, <b>Sendmail</b>, QMail, <b>Mdaemon</b>, www4mail, ...<br>
 <u>Perl interpreters (all Perl >= 5.005):</u><br>
@@ -144,7 +144,7 @@ Any other personalized Apache log format,<br>
 Any IIS log format (known as W3C format),<br>
 Webstar native log format,<br>
 Realmedia server, Windows Media Server, Darwin streaming server,<br>
-ProFTP server,<br>
+ProFTPd server, vsFTPd server,<br>
 Postfix, Sendmail, QMail, Mdaemon<br>
 A lot of web/wap/proxy/streaming servers log format<br>
 <br>
@@ -223,15 +223,17 @@ If you can't find it nor build it, you can try to use a LogFile parameter that l
 <br>
 
 <a name="FTP"></a><br>
-<b><u>FAQ-COM090 : SETUP FOR FTP SERVER LOG FILES</u></b><br>
+<b><u>FAQ-COM090 : SETUP FOR FTP SERVER LOG FILES (proftpd, vsftpd, ...)</u></b><br>
 <font class=CProblem>PROBLEM:</font><br>
-What do I have to do to use AWStats to analyze some FTP server log files.<br>
+What do I have to do to use AWStats to analyze some FTP server log files ?<br>
 <font class=CSolution>SOLUTION:</font><br>
 AWStats can be used with some FTP server log files.<br>
 <br>
-1- Setup your FTP log file format:<br>
+<u>With ProFTPd</u>:<br>
 <br>
-With ProFTP, modify the proftpd.conf file to add the following two lines :
+1- Setup your server log file format:<br>
+<br>
+Modify the proftpd.conf file to add the following two lines :
 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
 <!--<br>With ProFTPd 1.2.2:-->
 <i>LogFormat awstats "%t %h %u %m %f %s %b"</i> &nbsp; &nbsp; # WARNING: You must use a tab char between % tags and not a space !
@@ -252,7 +254,7 @@ Download a file by FTP and check that your new log file looks like this:<br>
 <br>
 2- Then setup AWStats to analyze the FTP log file:<br>
 <br>
-Copy config awstats.model.conf file to "awstats.proftp.conf".<br>
+Copy config file "awstats.model.conf" to "awstats.ftp.conf".<br>
 Modify this new config file:
 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
 <i>
@@ -295,11 +297,68 @@ LogFile="/var/log/xferlog"
 <br>
 Now you can use AWStats as usual (run the update process and read statistics).<br>
 <br>
+<br>
+<u>With vsFTPd, or any FTP server that log with xferlog format</u>:<br>
+<br>
+1- Check your server log file format:<br>
+<br>
+Take a look at your FTP server log file. You must have a format that match the following example to
+use this FAQ :<br>
+<table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
+<i>Wed Jan 01 19:29:35 2001 1 192.168.1.1 102 /home/file1.txt b _ o r username ftp 0 * c</i>\r
+</td></tr></table>
+<br>
+2- Then setup AWStats to analyze the FTP log file:<br>
+<br>
+If your FTP log file format looks good, copy config file "awstats.model.conf" to "awstats.ftp.conf".<br>
+Modify this new config file:
+<table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
+<i>
+LogFile="/var/log/xferlog"
+<br>LogType=F
+<br>LogFormat="%time3 %other %host %bytesd %url %other %other %method %other %logname %other %code %other %other"
+<br>LogSeparator="\s"
+<br>NotPageList=""
+<br>LevelForBrowsersDetection=0
+<br>LevelForOSDetection=0
+<br>LevelForRefererAnalyze=0
+<br>LevelForRobotsDetection=0
+<br>LevelForWormsDetection=0
+<br>LevelForSearchEnginesDetection=0
+<br>ShowLinksOnUrl=0
+<br>ShowMenu=1
+<br>ShowMonthStats=UVHB
+<br>ShowDaysOfMonthStats=HB
+<br>ShowDaysOfWeekStats=HB
+<br>ShowHoursStats=HB
+<br>ShowDomainsStats=HB
+<br>ShowHostsStats=HBL
+<br>ShowAuthenticatedUsers=HBL
+<br>ShowRobotsStats=0
+<br>ShowEMailSenders=0
+<br>ShowEMailReceivers=0
+<br>ShowSessionsStats=1
+<br>ShowPagesStats=PBEX
+<br>ShowFileTypesStats=HB
+<br>ShowFileSizesStats=0
+<br>ShowBrowsersStats=0
+<br>ShowOSStats=0
+<br>ShowOriginStats=0
+<br>ShowKeyphrasesStats=0
+<br>ShowKeywordsStats=0
+<br>ShowMiscStats=0
+<br>ShowHTTPErrorsStats=0
+<br>ShowSMTPErrorsStats=0
+</i></td></tr></table>
+<br>
+Now you can use AWStats as usual (run the update process and read statistics).<br>
+<br>
+
 
 <a name="MAIL"></a><br>
-<b><u>FAQ-COM100 : SETUP FOR MAIL LOG FILES (POSTFIX, SENDMAIL, QMAIL, MDAEMON, EXCHANGE...)</u></b><br>
+<b><u>FAQ-COM100 : SETUP FOR MAIL LOG FILES (Postfix, Sendmail, Qmail, MDaemon, Exchange...)</u></b><br>
 <font class=CProblem>PROBLEM:</font><br>
-What do I have to do to use AWStats to analyze my mail log files.<br>
+What do I have to do to use AWStats to analyze my mail log files ?<br>
 <font class=CSolution>SOLUTION:</font><br>
 <br>
 This tip works with AWStats 5.5 or higher.<br>
@@ -374,7 +433,7 @@ messy data, no id to join multiple records for same mail, etc...).
 For this reason, an "exact" log analysis is a joke with Exchange log files.
 However a little support is provided. In order to analyze Exchange traffic, you have to
 enable "Message Tracking" (see article http://support.microsoft.com/default.aspx?scid=kb;EN-US;246856).<br>
-Copy config awstats.model.conf file to "awstats.mail.conf".<br>
+Then copy config awstats.model.conf file to "awstats.mail.conf".<br>
 Modify this new config file:
 <table width="95%" border=1 cellpadding=0 cellspacing=0><tr class=CFAQ><td class=CFAQ>
 <i>
@@ -417,8 +476,8 @@ LogType=M
 Also don't forget that with Exchange, informations in a log analyses can't be exact.
 Do not send any questions or requests for using AWStats with Exchange, this is not a problem in
 AWStats and we have no time to support non opened products.<br>
-If you want to have complete and accurate information, forget using AWStats or use a more serious
-mail serveur (Postfix, Sendmail, QMail...)<br>
+If you want to have complete and accurate information with Exchange, forget using AWStats or use a
+more serious mail serveur (Postfix, Sendmail, QMail...)<br>
 <br>
 
 <a name="MEDIASERVER"></a><br>
@@ -1174,7 +1233,7 @@ You must use : <i>LogFormat="%host %logname %time1 %method %url %other %code %by
 and <i>LogSeparator="\s+"</i><br>
 
 <table bgcolor=#F4F4F4 width="100%" class=CFAQ><tr class=CFAQ><td class=CFAQ>
-<u>If you use a FTP server like <b>ProFTP</b>:</u><br>
+<u>If you use a FTP server like <b>ProFTPd</b>:</u><br>
 See <a href="#FTP">FAQ-COM090</a>.<br>
 </td></tr></table>
 

diff --git a/wwwroot/cgi-bin/awstats.model.conf b/wwwroot/cgi-bin/awstats.model.conf
index 1e1be58d0cb1462a0a74ecea3306931f97d7422c..dde8bc3d1bce51f1c1e76c1b91cd284fd52ef276 100644 (file)
--- a/wwwroot/cgi-bin/awstats.model.conf
+++ b/wwwroot/cgi-bin/awstats.model.conf
@@ -76,11 +76,11 @@ LogType=W
 #     you must use following keys to define the log format string (See FAQ
 #     for ftp, mail or exotic web log format examples):
 #   %host             Host client name or IP address
-#   %lognamequot      Authenticated login/user with format: "alex"
-#   %logname          Authenticated login/user with format: alex
+#   %lognamequot      Authenticated login/user with format: "john"
+#   %logname          Authenticated login/user with format: john
 #   %time1            Date and time with format: [dd/mon/yyyy:hh:mm:ss +0000] or [dd/mon/yyyy:hh:mm:ss]
 #   %time2            Date and time with format: yyyy-mm-dd hh:mm:ss
-#   %time3            Date and time with format: Mon dd hh:mm:ss
+#   %time3            Date and time with format: Mon dd hh:mm:ss or Mon dd hh:mm:ss yyyy
 #   %time4            Date and time with unix timestamp format: dddddddddd
 #   %methodurl        Method and URL with format: "GET /index.html HTTP/x.x"
 #   %methodurlnoprot  Method and URL with format: "GET /index.html"

diff --git a/wwwroot/cgi-bin/awstats.pl b/wwwroot/cgi-bin/awstats.pl
index 7cfee511bf9a120128244fa196c0196701637284..1715868bd991b9a109a653eaa4bed210f2a56363 100644 (file)
--- a/wwwroot/cgi-bin/awstats.pl
+++ b/wwwroot/cgi-bin/awstats.pl
@@ -3337,7 +3337,7 @@ sub Save_History {
        my $month=shift||'';
 
        my $xml=($BuildHistoryFormat eq 'xml'?1:0);
-       my ($xmlbb,$xmlbs,$xmlbe,$xmlhb,$xmlhs,$xmlhe,$xmlrb,$xmlrs,$xmlre,$xmleb,$xmlee)=();
+       my ($xmlbb,$xmlbs,$xmlbe,$xmlhb,$xmlhs,$xmlhe,$xmlrb,$xmlrs,$xmlre,$xmleb,$xmlee)=('','','','','','','','','','','');
        if ($xml) { ($xmlbb,$xmlbs,$xmlbe,$xmlhb,$xmlhs,$xmlhe,$xmlrb,$xmlrs,$xmlre,$xmleb,$xmlee)=("</comment><nu>\n",'</nu><recnb>','</recnb><table>','<tr><th>','</th><th>','</th></tr>','<tr><td>','</td><td>','</td></tr>','</table><nu>',"\n</nu></section>" ); }
        else { $xmlbs=' '; $xmlhs=' '; $xmlrs=' '; }
                        
@@ -4890,7 +4890,7 @@ sub DefinePerlParsingFormat {
                                $PerlParsingFormat .= "\\\"?([^\\\"]*)\\\"?";                   # logname can be "value", "" and - in same log (Lotus notes)
                        }
                        # Date format
-                       elsif ($f =~ /%time1$/ || $f =~ /%time1b$/) {   # [dd/mmm/yyyy:hh:mm:ss +0000] ou [dd/mmm/yyyy:hh:mm:ss],  time1b kept for backward compatibility
+                       elsif ($f =~ /%time1$/ || $f =~ /%time1b$/) {   # [dd/mmm/yyyy:hh:mm:ss +0000] or [dd/mmm/yyyy:hh:mm:ss],  time1b kept for backward compatibility
                                $pos_date = $i; $i++; push @fieldlib, 'date';
                                $pos_tz = $i; $i++; push @fieldlib, 'tz';
                                $PerlParsingFormat .= "\\[([^$LogSeparatorWithoutStar]+)( [^$LogSeparatorWithoutStar]+)?\\]";
@@ -4899,9 +4899,9 @@ sub DefinePerlParsingFormat {
                                $pos_date = $i; $i++; push @fieldlib, 'date';
                                $PerlParsingFormat .= "([^$LogSeparatorWithoutStar]+\\s[^$LogSeparatorWithoutStar]+)";  # Need \s for Exchange log files
                        }
-                       elsif ($f =~ /%time3$/) {       # mon d hh:mm:ss
+                       elsif ($f =~ /%time3$/) {       # mon d hh:mm:ss  or  mon dd hh:mm:ss yyyy  or  day mon dd hh:mm:ss  or  day mon dd hh:mm:ss yyyy
                                $pos_date = $i; $i++; push @fieldlib, 'date';
-                               $PerlParsingFormat .= "(\\w\\w\\w \\s?\\d+ \\d\\d:\\d\\d:\\d\\d)";
+                               $PerlParsingFormat .= "(?:\\w\\w\\w )?(\\w\\w\\w \\s?\\d+ \\d\\d:\\d\\d:\\d\\d(?: \\d\\d\\d\\d)?)";
                        }
                        elsif ($f =~ /%time4$/) {       # ddddddddddddd
                                $pos_date = $i; $i++; push @fieldlib, 'date';
@@ -5975,7 +5975,7 @@ if ($UpdateStats && $FrameName ne 'index' && $FrameName ne 'mainleft') {  # Updat
                }
                elsif ($dateparts[0] =~ /^....$/) { my $tmp=$dateparts[0]; $dateparts[0]=$dateparts[2]; $dateparts[2]=$tmp; }
                elsif ($field[$pos_date] =~ /^..:..:..:/) { $dateparts[2]+=2000; my $tmp=$dateparts[0]; $dateparts[0]=$dateparts[1]; $dateparts[1]=$tmp; }
-               elsif ($dateparts[0] =~ /^...$/)  { my $tmp=$dateparts[0]; $dateparts[0]=$dateparts[1]; $dateparts[1]=$tmp; $dateparts[5]=$dateparts[4]; $dateparts[4]=$dateparts[3]; $dateparts[3]=$dateparts[2]; $dateparts[2]=$nowyear; }
+               elsif ($dateparts[0] =~ /^...$/)  { my $tmp=$dateparts[0]; $dateparts[0]=$dateparts[1]; $dateparts[1]=$tmp; $tmp=$dateparts[5]; $dateparts[5]=$dateparts[4]; $dateparts[4]=$dateparts[3]; $dateparts[3]=$dateparts[2]; $dateparts[2]=$tmp||$nowyear; }
                if ($MonthNum{$dateparts[1]}) { $dateparts[1]=$MonthNum{$dateparts[1]}; }       # Change lib month in num month if necessary
 
                # Now @dateparts is (DD,MM,YYYY,HH,MM,SS) and we're going to create $timerecord=YYYYMMDDHHMMSS