# Added Storebot-Google (https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)\r
# Added Timpibot (http://www.timpi.io)\r
# Added YouBot (You.com)\r
-# \r
+#\r
# Edited description for facebookexternalhit in %RobotsHashIDLib\r
# Edited entry and description for Bytespider\r
# Edited description for CCBot\r
-# \r
+#\r
# Removed facebook (not documented by Meta - substituted by adding entries for FacebookBot and meta-... crawlers)\r
#\r
# Fixed: Missing entry for laserlikebot in %RobotsHashIDLib\r
# Fixed: Unescaped dashes (-) in entries Mediapartners-Google, Baiduspider-, Contacts-Crawler, DuckDuckBot-Https, Nimbostratus-Bot, Quick-Crawler, SemrushBot-SI, uni-leipzig.de\r
# Fixed: Unescaped dot (.) in entry bnf.fr_bot\r
# Fixed: Removed deprecated entry mojeek from %RobotsHashIDLib\r
-\r
+#\r
# 2024-07-17 Dinko Sotirov\r
-# Added Amazonbot (https://developer.amazon.com/support/amazonbot)\r
-# Added GPTBot (https://openai.com/gptbot)\r
-# Added GeedoProductSearch (http://www.geedo.com/product-search.html)\r
-\r
+# Added Amazonbot (https://developer.amazon.com/support/amazonbot)\r
+# Added GPTBot (https://openai.com/gptbot)\r
+# Added GeedoProductSearch (http://www.geedo.com/product-search.html)\r
+#\r
# 2023-07-04 RobC \r
-# Removed Dalvik as native Android UI Browser User Agent\r
-# Removed CFNetwork as native iOS and OSX Browser User Agent\r
-\r
+# Removed Dalvik as native Android UI Browser User Agent\r
+# Removed CFNetwork as native iOS and OSX Browser User Agent\r
+#\r
# 2021-05--05 RobC\r
-\r
-# Removed Baidu catchall because its picking up baidu.sogo.uc.UCBrowser which is a phone browser\r
-# Added baiduspider- catchall instead\r
-\r
-# Newly added from 2021-05-05\r
-# Adsbot\r
-# BW/\r
-# Bytespider\r
-# CheckMarkNetwork/\r
-# DuckDuckBot\r
-# # Foregenix Web Scan\r
-# IonCrawl\r
-# Linguee Bot\r
-# Neevabot\r
-# PetalBot\r
-# TkBot\r
-# vuhuvBot\r
-\r
-\r
+# Removed Baidu catchall because its picking up baidu.sogo.uc.UCBrowser which is a phone browser\r
+# Added baiduspider- catchall instead\r
+#\r
+# Newly added from 2021-05-05\r
+# Adsbot\r
+# BW/\r
+# Bytespider\r
+# CheckMarkNetwork/\r
+# DuckDuckBot\r
+# # Foregenix Web Scan\r
+# IonCrawl\r
+# Linguee Bot\r
+# Neevabot\r
+# PetalBot\r
+# TkBot\r
+# vuhuvBot\r
+#\r
# 2018-03-13 RobC \r
-# Added 36 robots and one generic ( survey ) using v 7.7 robots file as base. \r
-# Also moved robot "Obot" into generics so that it is singled out as an individual Robot. \r
+# Added 36 robots and one generic ( survey ) using v 7.7 robots file as base. \r
+# Also moved robot "Obot" into generics so that it is singled out as an individual Robot. \r
#\r
# 2016-09-02 RobC \r
-# Fixed a few errors and added a few missing bots from awstats 7.5 release.\r
+# Fixed a few errors and added a few missing bots from awstats 7.5 release.\r
#\r
# 2016-08-28 RobC \r
-# Complete re-build of this file almost from scratch.\r
-# dropped many old bots, added many new bots and reordered file.\r
-# edited and added regex expressions to stop spaces causing problems.\r
-# You should tune file by placing the most common robots crawling your site at top \r
-# in List1.\r
-#\r
-#\r
-# N.B. many bots need to be in correct order so don't chnage order without checking if\r
-# change will cause counts to be allocated to wrong bot. Not always simple.\r
+# Complete re-build of this file almost from scratch.\r
+# dropped many old bots, added many new bots and reordered file.\r
+# edited and added regex expressions to stop spaces causing problems.\r
+# You should tune file by placing the most common robots crawling your site at top \r
+# in List1.\r
#\r
+# N.B. many bots need to be in correct order so don't chnage order without checking if\r
+# change will cause counts to be allocated to wrong bot. Not always simple.\r
#\r
# 2005-08-19 Sean Carlos http://www.antezeta.com/awstats.html\r
-# added dipsie (not tested with real data).\r
-# added DomainsDB.net http://domainsdb.net/\r
-# added ia_archiver-web.archive.org (was inadvertently grouped with Alexa traffic)\r
-# added Nutch (used by looksmart (furl?))\r
-# added rssImagesBot\r
-# added Sqworm\r
-# added t\-h\-u\-n\-d\-e\-r\-s\-t\-o\-n\-e\r
-# added w3c css-validator\r
-# added documentation link to bot home pages for above and selected major bots.\r
-# In the case of international bots, choose .com page.\r
-# Included tool tip (html "title").\r
-# To do: parameterize to match both AWStats language and tooltips settings.\r
-# To do: add html links for all bots based on current documentation in source\r
-# files referenced below.\r
-# changed '\wbot[\/\-]', to '\wbot[\/\-]' (removed comma)\r
-# made minor grammar corrections to notes below\r
+# added dipsie (not tested with real data).\r
+# added DomainsDB.net http://domainsdb.net/\r
+# added ia_archiver-web.archive.org (was inadvertently grouped with Alexa traffic)\r
+# added Nutch (used by looksmart (furl?))\r
+# added rssImagesBot\r
+# added Sqworm\r
+# added t\-h\-u\-n\-d\-e\-r\-s\-t\-o\-n\-e\r
+# added w3c css-validator\r
+# added documentation link to bot home pages for above and selected major bots.\r
+# In the case of international bots, choose .com page.\r
+# Included tool tip (html "title").\r
+# To do: parameterize to match both AWStats language and tooltips settings.\r
+# To do: add html links for all bots based on current documentation in source\r
+# files referenced below.\r
+# changed '\wbot[\/\-]', to '\wbot[\/\-]' (removed comma)\r
+# made minor grammar corrections to notes below\r
# 2005-08-24 added YahooSeeker-Testing\r
-# added w3c-checklink\r
-# updated url for ask.com\r
+# added w3c-checklink\r
+# updated url for ask.com\r
# 2005-08-24 added Girafabot http://www.girafa.com/\r
# 2005-08-30 added PluckFeedCrawler http://www.pluck.com/\r
# added Gaisbot/3.0 (robot05@gais.cs.ccu.edu.tw; )\r
-# dded geniebot (wgao@genieknows.com)\r
+# added geniebot (wgao@genieknows.com)\r
# added BecomeBot link http://www.become.com/site_owners.html\r
# added topicblogs http://www.topicblogs.com/\r
# added Powermarks; seen used by referrer spam\r
# added RufusBot Rufus Web Miner http://64.124.122.252.webaroo.com/feedback.html\r
# added Seekbot (http://www.seekbot.net/bot.html)\r
# added Yahoo-MMCrawler/3.x (mms-mmcrawler-support@yahoo-inc.com)\r
-# added link for BaiDuSpider\r
+# added link for BaiDuSpider\r
# added link for Blogshares Spider\r
# added link for StackRambler http://www.rambler.ru/doc/faq.shtml\r
# added link for WISENutbot\r
# -- updates\r
# updated AskJeeves to Ask\r
# 2012-06-05 Albrecht Mueller\r
-# added Grabber from SDSC (San Diego Supercomputer Center).\r
+# added Grabber from SDSC (San Diego Supercomputer Center).\r
# 2013-09-30 Albrecht Mueller\r
-# AWStats probably cannot detect this bot as it identifies itself in\r
-# the referrer field and not in the user agent string.\r
+# AWStats probably cannot detect this bot as it identifies itself in\r
+# the referrer field and not in the user agent string.\r
#92.113.100.35 - - [29/Sep/2013:17:22:46 +0200] "GET /robots.txt HTTP/1.1" 200 516 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" "-"\r
#92.113.100.35 - - [29/Sep/2013:17:22:49 +0200] "GET /tghome.htm HTTP/1.1" 200 4445 "http://extrabot.com/help/frytygativyheku.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" "-"\r
#92.113.100.35 - - [29/Sep/2013:17:22:51 +0200] "GET / HTTP/1.1" 200 5467 "http://extrabot.com/help/frytygativyheku.htm" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20100101 Firefox/5.0" "-"\r
projects / thirdparty / AWStats.git / commitdiff
