+FAQ-ABO100 Which server log files or operating systems are supported ?
FAQ-ABO150 Which log format can AWStats analyze ?
FAQ-ABO200 Which languages are available ?
FAQ-ABO250 Can AWStats be integrated with PHP Nuke ?
@@ -55,6 +55,7 @@ FAQ-SET210 "Internal Error" after a long time in my browser (Se FAQ-SET220 Crash while running awstats.pl or page content only partialy loaded
FAQ-SET250 Log format setup or errors.
FAQ-SET260 Setup for FTP server log files.
+FAQ-SET270 Setup for SENDMAIL log files.
FAQ-SET280 Error "Not same number of records of...".
FAQ-SET300 Error "Couldn't open file ..."
FAQ-SET350 Empty or null statistics reported.
@@ -93,18 +94,20 @@ FAQ-SEC200 How to manage log files (and statistics) corrupted b
-FAQ-ABO100 : WHICH WEB SERVER OR OS ARE SUPPORTED ?
-AWStats can works with all web server able to write log file with a combined log format (XLF/ELF) like Apache, +FAQ-ABO100 : WHICH SERVER LOG FILES OR OS ARE SUPPORTED ?
+AWStats can works with :
+- All web server able to write log file with a combined log format (XLF/ELF) like Apache, a common log format (CLF) like Apache or Squid, an W3C log format like IIS 5.0 or higher (Some users have -reported that you can setup your log format to W3C with IIS 4.0 but you need a service pack 6), and a lot of others -Web/Wap/Proxy servers.
+reported that you can setup your log format to W3C with IIS 4.0 but you need a service pack 6)
+- Most of all others Web/Wap/Proxy servers.
+- Some FTP, syslog and mail log files.
Because AWStats is in Perl, it can works on all Operating Systems.
Examples of used platforms (bold means 'tested by author', others were reported by AWStats users to work correctly) :
|
OS: Windows NT 4.0, Windows 2000, Windows Me, Linux, Macintosh, Solaris, Aix, BeOS, ... -Web/Wap/Proxy servers -Apache, IIS 5.0, WebStar, WebLogic, WebSite, Tomcat, Squid, Roxen, Resin, IPlanet, IceCast, Zope, www4mail, ... +Web/Wap/Proxy/FTP/Mails servers +Apache, IIS 5.0, WebStar, WebLogic, WebSite, Tomcat, Squid, Sendmail, Roxen, Resin, ProFTP, IPlanet, IceCast, Zope, www4mail, ... Perl interpreters: ActivePerl 5.6, Perl for unix 5.0, mod_perl for Apache, ... |
AWStats setup knows predefined log format you can use to make AWStats config easier. However, you can define your own log format, that's the reason why AWStats can analyze nearly all web, wap -and proxy server log files. Some FTP servers are also supported.
+and proxy server log files. Some FTP servers log files, Syslog or mail logs can also be analyzed.
The only requirement is "Your log file must contain required information".
This is example of possible log format:
@@ -311,23 +314,101 @@ What do I have to do to use AWStats to analyze some FTP server log files.
SOLUTION:
AWStats was built to analyze web,wap or proxy server's log files. However it can be used with some FTP server log files.
This is example for ProFTP:
-Modify the proftpd.conf file. At the very top add two new defines.
-LogFormat awstats "%t %h %u GET %f 200 %b"
-ExtendedLog /var/log/xferlog read,write awstats
-Turn off Transfer log (optional)
-TransferLog none
+Modify the proftpd.conf file. At the very top add two new defines : + +
LogFormat awstats "%t %h %u get %f 200 %b" +
ExtendedLog /var/log/xferlog read,write awstats + +
Turn off Transfer log (optional): + +
TransferLog none + +
+Setup AWStats to analyze those FTP log files:
Copy config awstats.conf file to "awstats.proftp.conf".
-Modify this new config file:
-LogFile="/var/log/xferlog"
-LogFormat="%time1 %host %logname %method %url %code %bytesd"
-DNSLookup=0
-ShowLinksOnUrl=0
-ShowAuthenticatedUsers=1
+Modify this new config file: + +
LogFile="/var/log/xferlog" +
LogFormat="%time1 %host %logname %method %url %code %bytesd" +
DNSLookup=2 +
ShowLinksOnUrl=0 +
ShowHeader=1 +
ShowMenu=1 +
ShowMonthDayStats=1 +
ShowDaysOfWeekStats=1 +
ShowHoursStats=1 +
ShowDomainsStats=1 +
ShowHostsStats=1 +
ShowAuthenticatedUsers=1 +
ShowRobotsStats=0 +
ShowSessionsStats=1 +
ShowPagesStats=1 +
ShowCompressionStats=0 +
ShowFileTypesStats=1 +
ShowFileSizesStats=0 +
ShowBrowsersStats=0 +
ShowOSStats=0 +
ShowOriginStats=0 +
ShowKeyphrasesStats=0 +
ShowKeywordsStats=0 +
ShowHTTPErrorsStats=0 +
ShowEMailSenders=0 +
ShowEMailReceivers=0 +
+Now you can use AWStats as usual (run the update process and read statistics).
Warning: This tip doesn't work for FTPed files that contains spaces in them (they are not reported). You can still convert your log file to replace 'space' char into '_'...
+
+FAQ-SET270 : SETUP FOR SENDMAIL LOG FILES
+PROBLEM:
+What do I have to do to use AWStats to analyze my Sendmail log files.
+SOLUTION:
+First you must download SMA to use it as a post-processor for AWStats update process.
+Setup SMA to work with AWStats:
+Copy sma (or sma.exe) file and sma.conf into same directory than awstats.pl file.
+Edit sma.conf to change the CLogFormat parameter to : + +
CLogFormat "%y-%m-%d %h:%n:%s %f %t %F %T SMTP - %z %S" +
ClogSentOnly no + +
+Setup AWStats to analyze SMA processed mail log:
+Copy config awstats.conf file to "awstats.sendmail.conf".
+Modify this new config file: + +
LogFile="sma -O clog /var/log/mail/maillog |" +
LogFormat="%time2 %email %email_r %host %host_r %method %url %code %bytesd" +
DNSLookup=0 +
ShowHeader=1 +
ShowMenu=1 +
ShowMonthDayStats=1 +
ShowDaysOfWeekStats=1 +
ShowHoursStats=1 +
ShowDomainsStats=0 +
ShowHostsStats=1 +
ShowAuthenticatedUsers=0 +
ShowRobotsStats=0 +
ShowSessionsStats=0 +
ShowPagesStats=0 +
ShowCompressionStats=0 +
ShowFileTypesStats=0 +
ShowFileSizesStats=0 +
ShowBrowsersStats=0 +
ShowOSStats=0 +
ShowOriginStats=0 +
ShowKeyphrasesStats=0 +
ShowKeywordsStats=0 +
ShowHTTPErrorsStats=0 +
ShowEMailSenders=1 +
ShowEMailReceivers=1 + +
+Now you can use AWStats as usual (run the update process and read statistics).
+
+
FAQ-SET280 : ERROR "NOT SAME NUMBER OF RECORDS OF..."
PROBLEM:
diff --git a/docs/awstats_what.html b/docs/awstats_what.html index 748ee36c..1a008ea9 100644 --- a/docs/awstats_what.html +++ b/docs/awstats_what.html @@ -54,7 +54,7 @@ * HTTP errors (Page Not Found with last referrer, ...),
AWStats also supports the following features:
- * Can analyze a lot of log formats: Apache NCSA combined log files (XLF/ELF) or common (CLF), IIS log files (W3C), WebStar native log files and other web, proxy or wap servers log files). See AWStats F.A.Q. for examples.
+ * Can analyze a lot of log formats: Apache NCSA combined log files (XLF/ELF) or common (CLF), IIS log files (W3C), WebStar native log files and other web, proxy or wap servers log files (but also ftp, syslog or mail log files). See AWStats F.A.Q. for examples.
* Works from command line and from a browser as a CGI (with dynamic filters capabilities for some charts),
* Update of statistics can be made from a web browser and not only from a scheduler,
* Unlimited log file size, support split log files (load balancing system),
diff --git a/wwwroot/cgi-bin/awstats.model.conf b/wwwroot/cgi-bin/awstats.model.conf index 1b9570cb..2a34c4af 100644 --- a/wwwroot/cgi-bin/awstats.model.conf +++ b/wwwroot/cgi-bin/awstats.model.conf @@ -64,7 +64,7 @@ LogFile="/var/log/httpd/mylog.log" # %method Method with format: GET # %url URL only with format: /index.html # %query Query string (used by URLWithQuery option) -# %code HTTP return code with format: 999 +# %code Return code status (with format for web log: 999) # %bytesd Size of document in bytes # %refererquot Referer page with format: "http://from.com/from.htm" # %referer Referer page with format: http://from.com/from.htm @@ -540,7 +540,7 @@ ShowAuthenticatedUsers=0 ShowRobotsStats=1 ShowSessionsStats=1 ShowPagesStats=1 -ShowCompressionStats=0 # Show report of compression stats when using mod_gzip +ShowCompressionStats=0 # Show compression statistics when using mod_gzip ShowFileTypesStats=1 ShowFileSizesStats=0 # Not yet available ShowBrowsersStats=1 @@ -549,6 +549,8 @@ ShowOriginStats=1 ShowKeyphrasesStats=1 ShowKeywordsStats=1 ShowHTTPErrorsStats=1 +ShowEMailSenders=0 # For use when analyzing mail log files. Not yet available +ShowEMailReceivers=0 # For use when analyzing mail log files. Not yet available # This value can be used to choose maximum number of lines shown for each diff --git a/wwwroot/cgi-bin/awstats.pl b/wwwroot/cgi-bin/awstats.pl index 9eb8530b..38e80de9 100644 --- a/wwwroot/cgi-bin/awstats.pl +++ b/wwwroot/cgi-bin/awstats.pl @@ -103,6 +103,7 @@ $NbOfLinesRead $NbOfLinesDropped $NbOfLinesCorrupted $NbOfOldLines $NbOfNewLines $NewLinePhase $NbOfLinesForCorruptedLog $PurgeLogFile $ShowAuthenticatedUsers $ShowCompressionStats $ShowFileSizesStats $ShowDropped $ShowCorrupted $ShowUnknownOrigin $ShowLinksToWhoIs +$ShowEMailSenders $ShowEMailReceivers $Expires $UpdateStats $MigrateStats $URLWithQuery $UseFramesWhenCGI /; ($DNSLookup, $AllowAccessFromWebToAuthenticatedUsersOnly, $BarHeight, $BarWidth, @@ -115,8 +116,9 @@ $NbOfLinesRead, $NbOfLinesDropped, $NbOfLinesCorrupted, $NbOfOldLines, $NbOfNewL $NewLinePhase, $NbOfLinesForCorruptedLog, $PurgeLogFile, $ShowAuthenticatedUsers, $ShowCompressionStats, $ShowFileSizesStats, $ShowDropped, $ShowCorrupted, $ShowUnknownOrigin, $ShowLinksToWhoIs, +$ShowEMailSenders, $ShowEMailReceivers, $Expires, $UpdateStats, $MigrateStats, $URLWithQuery, $UseFramesWhenCGI)= -(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0); +(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0); use vars qw/ $PosTotalVisits $PosTotalUnique $PosMonthHostsKnown $PosMonthHostsUnknown /; @@ -180,11 +182,13 @@ $YearRequired, $QueryString, $SiteConfig, $StaticLinks, $URLFilter, $PageCode, $PerlParsingFormat, $SiteToAnalyze, $SiteToAnalyzeWithoutwww, $UserAgent)= ("","","","","","","","","","","","","","","","","",""); use vars qw/ -$pos_vh $pos_rc $pos_logname $pos_date $pos_method $pos_url $pos_code $pos_size +$pos_vh $pos_host $pos_logname $pos_date $pos_method $pos_url $pos_code $pos_size $pos_referer $pos_agent $pos_query $pos_gzipin $pos_gzipout $pos_gzipratio +$pos_emails $pos_emailr $pos_hostr /; -$pos_vh = $pos_rc = $pos_logname = $pos_date = $pos_method = $pos_url = $pos_code = $pos_size = -1; +$pos_vh = $pos_host = $pos_logname = $pos_date = $pos_method = $pos_url = $pos_code = $pos_size = -1; $pos_referer = $pos_agent = $pos_query = $pos_gzipin = $pos_gzipout = $pos_gzipratio = -1; +$pos_emails = $pos_emailr = $pos_hostr = -1; use vars qw/ $lowerval $FirstTime $LastTime $LastLine $LastUpdate @@ -888,6 +892,8 @@ sub Read_Config { if ($param =~ /^ShowKeywordsStats/) { $ShowKeywordsStats=$value; next; } if ($param =~ /^ShowCompressionStats/) { $ShowCompressionStats=$value; next; } if ($param =~ /^ShowHTTPErrorsStats/) { $ShowHTTPErrorsStats=$value; next; } + if ($param =~ /^ShowEMailSenders/) { $ShowEMailSenders=$value; next; } + if ($param =~ /^ShowEMailReceivers/) { $ShowEMailReceivers=$value; next; } if ($param =~ /^MaxRowsInHTMLOutput/) { $MaxRowsInHTMLOutput=$value; next; } # Not used yet if ($param =~ /^MaxNbOfDomain/) { $MaxNbOfDomain=$value; next; } if ($param =~ /^MaxNbOfHostsShown/) { $MaxNbOfHostsShown=$value; next; } @@ -1220,6 +1226,8 @@ sub Check_Config { if ($ShowKeywordsStats !~ /[0-1]/) { $ShowKeywordsStats=1; } if ($ShowCompressionStats !~ /[0-1]/) { $ShowCompressionStats=1; } if ($ShowHTTPErrorsStats !~ /[0-1]/) { $ShowHTTPErrorsStats=1; } + if ($ShowEMailSenders !~ /[0-1]/) { $ShowEMailSenders=0; } + if ($ShowEMailReceivers !~ /[0-1]/) { $ShowEMailReceivers=0; } if ($MaxNbOfDomain !~ /^\d+$/ || $MaxNbOfDomain<1) { $MaxNbOfDomain=20; } if ($MaxNbOfHostsShown !~ /^\d+$/ || $MaxNbOfHostsShown<1) { $MaxNbOfHostsShown=20; } if ($MinHitHost !~ /^\d+$/ || $MinHitHost<1) { $MinHitHost=1; } @@ -1630,7 +1638,7 @@ sub Read_History_With_Update { #if ($field[0]) { # This test must not be here for TIME section (because field[0] is "0" for hour 0) $count++; if ($SectionsToLoad{"time"}) { - if ("$MonthRequired" eq "year" || "$MonthRequired" eq "$month") { # Still required + if ($withupdate || "$MonthRequired" eq "year" || "$MonthRequired" eq "$month") { # Still required $countloaded++; if ($field[1]) { $_time_p[$field[0]]+=int($field[1]); } if ($field[2]) { $_time_h[$field[0]]+=int($field[2]); } @@ -2544,7 +2552,7 @@ sub Save_History { print HISTORYTMP "# If you remove this file, all statistics for date $year-$month will be lost/reset.\n"; print HISTORYTMP "\n"; print HISTORYTMP "# This section is not used yet\n"; - print HISTORYTMP "BEGIN_MAP 22\n"; + print HISTORYTMP "BEGIN_MAP 24\n"; print HISTORYTMP "POS_GENERAL \n"; print HISTORYTMP "POS_TIME \n"; print HISTORYTMP "POS_VISITOR \n"; @@ -2567,6 +2575,8 @@ sub Save_History { print HISTORYTMP "POS_KEYWORDS \n"; print HISTORYTMP "POS_ERRORS \n"; print HISTORYTMP "POS_SIDER_404 \n"; + print HISTORYTMP "POS_EMAILSENDER \n"; + print HISTORYTMP "POS_EMAILRECEIVER \n"; print HISTORYTMP "END_MAP\n"; } @@ -3853,23 +3863,23 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat if ($LogFormat =~ /^[1-5]$/) { # Pre-defined log format if ($LogFormat eq "1") { # Same than "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" $PerlParsingFormat="([^ ]+) [^ ]+ ([^ ]+) \\[([^ ]+) [^ ]+\\] \\\"([^ ]+) ([^ ]+) [^\\\"]+\\\" ([\\d|-]+) ([\\d|-]+) \\\"(.*)\\\" \\\"([^\\\"]*)\\\""; # referer and ua might be "" - $pos_rc=0;$pos_logname=1;$pos_date=2;$pos_method=3;$pos_url=4;$pos_code=5;$pos_size=6;$pos_referer=7;$pos_agent=8; + $pos_host=0;$pos_logname=1;$pos_date=2;$pos_method=3;$pos_url=4;$pos_code=5;$pos_size=6;$pos_referer=7;$pos_agent=8; } elsif ($LogFormat eq "2") { # Same than "date time c-ip cs-username cs-method cs-uri-stem sc-status sc-bytes cs-version cs(User-Agent) cs(Referer)" $PerlParsingFormat="([^\\s]+ [^\\s]+) ([^\\s]+) ([^\\s]+) ([^\\s]+) ([^\\s]+) ([\\d|-]+) ([\\d|-]+) [^\\s]+ ([^\\s]+) ([^\\s]+)"; - $pos_date=0;$pos_rc=1;$pos_logname=2;$pos_method=3;$pos_url=4;$pos_code=5;$pos_size=6;$pos_agent=7;$pos_referer=8; + $pos_date=0;$pos_host=1;$pos_logname=2;$pos_method=3;$pos_url=4;$pos_code=5;$pos_size=6;$pos_agent=7;$pos_referer=8; } elsif ($LogFormat eq "3") { $PerlParsingFormat="([^\\t]*\\t[^\\t]*)\\t([^\\t]*)\\t([\\d]*)\\t([^\\t]*)\\t([^\\t]*)\\t([^\\t]*)\\t[^\\t]*\\t.*:([^\\t]*)\\t([\\d]*)"; - $pos_date=0;$pos_method=1;$pos_code=2;$pos_rc=3;$pos_agent=4;$pos_referer=5;$pos_url=6;$pos_size=7; + $pos_date=0;$pos_method=1;$pos_code=2;$pos_host=3;$pos_agent=4;$pos_referer=5;$pos_url=6;$pos_size=7; } elsif ($LogFormat eq "4") { # Same than "%h %l %u %t \"%r\" %>s %b" $PerlParsingFormat="([^ ]+) [^ ]+ ([^ ]+) \\[([^ ]+) [^ ]+\\] \\\"([^ ]+) ([^ ]+) [^\\\"]+\\\" ([\\d|-]+) ([\\d|-]+)"; - $pos_rc=0;$pos_logname=1;$pos_date=2;$pos_method=3;$pos_url=4;$pos_code=5;$pos_size=6; + $pos_host=0;$pos_logname=1;$pos_date=2;$pos_method=3;$pos_url=4;$pos_code=5;$pos_size=6; } elsif ($LogFormat eq "5") { # Same than "c-ip cs-username c-agent sc-authenticated date time s-svcname s-computername cs-referred r-host r-ip r-port time-taken cs-bytes sc-bytes cs-protocol cs-transport s-operation cs-uri cs-mime-type s-object-source sc-status s-cache-info" $PerlParsingFormat="([^\\t]*)\\t([^\\t]*)\\t([^\\t]*)\\t[^\\t]*\\t([^\\t]*\\t[^\\t]*)\\t[^\\t]*\\t[^\\t]*\\t([^\\t]*)\\t[^\\t]*\\t[^\\t]*\\t[^\\t]*\\t[^\\t]*\\t[^\\t]*\\t([^\\t]*)\\t[^\\t]*\\t[^\\t]*\\t([^\\t]*)\\t([^\\t]*)\\t[^\\t]*\\t[^\\t]*\\t([^\\t]*)\\t[^\\t]*"; - $pos_rc=0;$pos_logname=1;$pos_agent=2;$pos_date=3;$pos_referer=4;$pos_size=5;$pos_method=6;$pos_url=7;$pos_code=8; + $pos_host=0;$pos_logname=1;$pos_agent=2;$pos_date=3;$pos_referer=4;$pos_size=5;$pos_method=6;$pos_url=7;$pos_code=8; } } else { # Personalized log format @@ -3925,8 +3935,12 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat $pos_vh = $i; $i++; $PerlParsingFormat .= "([^$LogSeparator]+)"; } + elsif ($f =~ /%host_r$/) { + $pos_hostr = $i; $i++; + $PerlParsingFormat .= "([^$LogSeparator]+)"; + } elsif ($f =~ /%host$/) { - $pos_rc = $i; $i++; + $pos_host = $i; $i++; $PerlParsingFormat .= "([^$LogSeparator]+)"; } elsif ($f =~ /%logname$/) { @@ -3939,9 +3953,6 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat } elsif ($f =~ /%time1$/) { $pos_date = $i; $i++; - #$pos_zone = $i; - #$i++; - #$PerlParsingFormat .= "\\[([^$LogSeparator]+) ([^$LogSeparator]+)\\]"; $PerlParsingFormat .= "\\[([^$LogSeparator]+) [^$LogSeparator]+\\]"; } elsif ($f =~ /%time2$/) { @@ -4006,16 +4017,24 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat $pos_gzipratio=$i;$i++; $PerlParsingFormat .= "([^$LogSeparator]+)"; } - elsif ($f =~ /%syslog$/) { # Added for syslog time and host stamp, fields are skipped and not analyzed + elsif ($f =~ /%syslog$/) { # Added for syslog time and host stamp, fields are skipped and not analyzed $PerlParsingFormat .= "[A-Z][a-z][a-z] .[0-9] ..:..:.. [A-Za-z]+"; } + elsif ($f =~ /%email_r$/) { + $pos_emailr = $i; $i++; + $PerlParsingFormat .= "([^$LogSeparator]+)"; + } + elsif ($f =~ /%email$/) { + $pos_emails = $i; $i++; + $PerlParsingFormat .= "([^$LogSeparator]+)"; + } else { $PerlParsingFormat .= "[^$LogSeparator]+"; } } if (! $PerlParsingFormat) { error("Error: No recognized format tag in personalized LogFormat string"); } } - if ($pos_rc < 0) { error("Error: Your personalized LogFormat does not include all fields required by AWStats (Add \%host in your LogFormat string)."); } + if ($pos_host < 0) { error("Error: Your personalized LogFormat does not include all fields required by AWStats (Add \%host in your LogFormat string)."); } if ($pos_date < 0) { error("Error: Your personalized LogFormat does not include all fields required by AWStats (Add \%time1 or \%time2 in your LogFormat string)."); } if ($pos_method < 0) { error("Error: Your personalized LogFormat does not include all fields required by AWStats (Add \%methodurl or \%method in your LogFormat string)."); } if ($pos_url < 0) { error("Error: Your personalized LogFormat does not include all fields required by AWStats (Add \%methodurl or \%url in your LogFormat string)."); } @@ -4085,8 +4104,9 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat next; } - if ($Debug) { debug(" Correct format line $NbOfLinesRead : host=\"$field[$pos_rc]\", logname=\"$field[$pos_logname]\", date=\"$field[$pos_date]\", method=\"$field[$pos_method]\", url=\"$field[$pos_url]\", code=\"$field[$pos_code]\", size=\"$field[$pos_size]\", referer=\"$field[$pos_referer]\", agent=\"$field[$pos_agent]\"",4); } + if ($Debug) { debug(" Correct format line $NbOfLinesRead : host=\"$field[$pos_host]\", logname=\"$field[$pos_logname]\", date=\"$field[$pos_date]\", method=\"$field[$pos_method]\", url=\"$field[$pos_url]\", code=\"$field[$pos_code]\", size=\"$field[$pos_size]\", referer=\"$field[$pos_referer]\", agent=\"$field[$pos_agent]\"",4); } #if ($Debug) { debug("$field[$pos_vh] - $field[$pos_gzipin] - $field[$pos_gzipout] - $field[$pos_gzipratio]\n",4); } + if ($Debug) { debug("$field[$pos_emails] - $field[$pos_emailr] - $field[$pos_hostr]\n",4); } # Check virtual host name #---------------------------------------------------------------------- @@ -4103,6 +4123,10 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat # HTTP request. Keep only GET, POST, HEAD, *OK* with Webstar but not OPTIONS $protocol=1; } + elsif ($field[$pos_method] eq 'SMTP') { + # Mail request. + $protocol=3; + } elsif ($field[$pos_method] =~ /sent/i || $field[$pos_method] =~ /get/i) { # FTP request. $protocol=2; @@ -4167,7 +4191,7 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat # Skip for some client host IP addresses, some URLs, other URLs my $qualifdrop=""; - if (@SkipHosts && &SkipHost($field[$pos_rc])) { $qualifdrop="Dropped record (host $field[$pos_rc] not qualified by SkipHosts)"; } + if (@SkipHosts && &SkipHost($field[$pos_host])) { $qualifdrop="Dropped record (host $field[$pos_host] not qualified by SkipHosts)"; } elsif (@SkipFiles && &SkipFile($field[$pos_url])) { $qualifdrop="Dropped record (URL $field[$pos_url] not qualified by SkipFiles)"; } elsif (@OnlyFiles && ! &OnlyFile($field[$pos_url])) { $qualifdrop="Dropped record (URL $field[$pos_url] not qualified by OnlyFiles)"; } if ($qualifdrop) { @@ -4192,13 +4216,13 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat $lastprocessedyearmonth=sprintf("%04i%02i",$lastprocessedyear,$lastprocessedmonth); } - # Check return code - #------------------ - if ($protocol == 1) { # HTTP record + # Check return status code + #------------------------- + if ($protocol == 1) { # HTTP record if ($ValidHTTPCodes{$field[$pos_code]}) { # Code is valid if ($field[$pos_code] == 304) { $field[$pos_size]=0; } } - else { # Code is not valid + else { # Code is not valid if ($field[$pos_code] =~ /^\d\d\d$/) { # Keep error code and next $_errors_h{$field[$pos_code]}++; if ($field[$pos_code] == 404) { $_sider404_h{$field[$pos_url]}++; $_referer404_h{$field[$pos_url]}=$field[$pos_referer]; } @@ -4211,40 +4235,50 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat } } } - - $field[$pos_agent] =~ tr/\+ /__/; # Same Agent with different writing syntax have now same name - $field[$pos_agent] =~ s/%20/_/g; # This is to support servers (like Roxen) that writes user agent with %20 in it - $UserAgent = lc($field[$pos_agent]); + elsif ($protocol == 3) { # SMTP record + if ($field[$pos_code] != 1) { # Code is not valid + $field[$pos_size]=0; + $_errors_h{$field[$pos_code]}++; + next; + } + } # Analyze: Robot #--------------- - if ($LevelForRobotsDetection) { - - my $uarobot=$TmpRobot{$UserAgent}; - if (! $uarobot) { - # If made on each record -> -1300 rows/seconds - my $foundrobot=0; - #study $UserAgent; Does not increase speed - foreach my $bot (@RobotsSearchIDOrder) { - if ($UserAgent =~ /$bot/i) { - $foundrobot=1; - $TmpRobot{$UserAgent}=$uarobot="$bot"; # Last time, we won't search if robot or not. We know it's is. - last; + if ($pos_agent >= 0) { + $field[$pos_agent] =~ tr/\+ /__/; # Same Agent with different writing syntax have now same name + $field[$pos_agent] =~ s/%20/_/g; # This is to support servers (like Roxen) that writes user agent with %20 in it + $UserAgent=lc($field[$pos_agent]); + + if ($LevelForRobotsDetection) { + + my $uarobot=$TmpRobot{$UserAgent}; + if (! $uarobot) { + # If made on each record -> -1300 rows/seconds + my $foundrobot=0; + #study $UserAgent; Does not increase speed + foreach my $bot (@RobotsSearchIDOrder) { + if ($UserAgent =~ /$bot/i) { + $foundrobot=1; + $TmpRobot{$UserAgent}=$uarobot="$bot"; # Last time, we won't search if robot or not. We know it's is. + last; + } + } + if (! $foundrobot) { # Last time, we won't search if robot or not. We know it's not. + $TmpRobot{$UserAgent}=$uarobot="-"; } } - if (! $foundrobot) { # Last time, we won't search if robot or not. We know it's not. - $TmpRobot{$UserAgent}=$uarobot="-"; + # If robot, we stop here + if ($uarobot ne "-") { + if ($Debug) { debug("UserAgent $UserAgent contains robot ID '$uarobot'",2); } + $_robot_h{$uarobot}++; $_robot_l{$uarobot}=$timerecord; + next; } + } - # If robot, we stop here - if ($uarobot ne "-") { - if ($Debug) { debug("UserAgent $UserAgent contains robot ID '$uarobot'",2); } - $_robot_h{$uarobot}++; $_robot_l{$uarobot}=$timerecord; - next; - } - } - + + # Canonize and clean target URL and referrer URL. Possible URL syntax for $field[$pos_url]: # /mydir/mypage.ext?param1=x¶m2=y#aaa # /mydir/mypage.ext#aaa @@ -4325,7 +4359,7 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat # Do DNS lookup #-------------- - my $Host=$field[$pos_rc]; + my $Host=$field[$pos_host]; my $HostResolved=''; if ($DNSLookup) { # DNS lookup is 1 or 2 if ($Host =~ /^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/) { @@ -4465,7 +4499,8 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat $_host_h{$_}++; $_host_k{$_}+=$field[$pos_size]; - if ($UserAgent) { # Made on each record -> -100 rows/seconds + + if ($pos_agent >= 0 && $UserAgent) { # Made on each record -> -100 rows/seconds if ($LevelForBrowsersDetection) { @@ -4567,7 +4602,7 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat # Analyze: Referer #----------------- my $found=0; - if ($LevelForRefererAnalyze && $field[$pos_referer]) { + if ($pos_referer >= 0 && $LevelForRefererAnalyze && $field[$pos_referer]) { # Direct ? if ($field[$pos_referer] eq "-" || $field[$pos_referer] eq "bookmarks") { # "bookmarks" is sent by Netscape, "-" by all others browsers @@ -4815,7 +4850,7 @@ if ($UpdateStats && $FrameName ne "index" && $FrameName ne "mainleft") { # Updat my $bold=($ENV{"GATEWAY_INTERFACE"}?"":""); my $unbold=($ENV{"GATEWAY_INTERFACE"}?"":""); my $br=($ENV{"GATEWAY_INTERFACE"}?"
":""); - warning("Warning: $bold$PROG$unbold has detected that some hosts names were already resolved in your logfile $bold$DNSLookupAlreadyDone$unbold.$br\nIf DNS lookup was already made by the logger (web server), you should change your setup DNSLookup=1 into DNSLookup=0 to increase $PROG speed."); + warning("Warning: $bold$PROG$unbold has detected that some hosts names were already resolved in your logfile $bold$DNSLookupAlreadyDone$unbold.$br\nIf DNS lookup was already made by the logger (web server), you should change your setup DNSLookup=$DNSLookup into DNSLookup=0 to increase $PROG speed."); } if ($DNSLookup && $NbOfNewLines) { # Save new DNS last update cache file @@ -4990,21 +5025,24 @@ EOF if (($HTMLOutput eq "main" && $FrameName ne "mainright") || $FrameName eq "mainleft") { # If main page asked my $linkpage=($FrameName eq "mainleft"?"$AWScript?${NewLinkParams}":""); $linkpage =~ s/&$//; my $targetpage=($FrameName eq "mainleft"?" target=mainright":""); + my $linetitle=0; print "
![](\"$DirIcons/other/page.png\"){kind=small}
":""); print "$Message[80]\n"; print ($frame?" ":""); print "$Message[80]\n"; print ($frame?" ":""); print "$Message[9]\n"; print ($frame?" ":""); print "$Message[80]\n"; print ($frame?" ":""); print "$Message[58]\n"; print ($frame?" ":""); print "$Message[0]\n"; print ($frame?" ":""); print "$Message[126]\n"; print ($frame?" ":""); print "$Message[127]\n"; print ($frame?" ":""); print "$Message[120]\n"; print ($frame?" ":""); print "$Message[121]\n"; print ($frame?" ":""); print "$Message[31]\n"; print ($frame?"\n"); } -- 2.47.3