ADSAComponent ADSAComponent (postmaster@cnds.ucd.ie) AOL-IWENG 3.0 Mozilla/2.0 (Compatible; AOL-IWENG 3.0; Win16) ASPseek 1.2.10 ASPseek/1.2.10 Almaden c01 http://www.almaden.ibm.com/cs/crawler [c01] Almaden wf55 http://www.almaden.ibm.com/cs/crawler [wf55] Amsterdam 2002 Mozilla/4.0 (Search Engine Marketing Tactics Amsterdam 2002 Information Spider) Antibot 1.1.11 antibot-V1.1.11/i586-linux-2.2 AnyBrowser Search Engine AnyBrowser.com Search Engine Appie 1.1 appie 1.1 (www.walhello.com) Ask Jeeves Mozilla/2.0 (compatible; Ask Jeeves) Ask Jeeves Teoma Mozilla/2.0 (compatible; Ask Jeeves/Teoma) AvantGo 3.2 Mozilla/3.0 (compatible; AvantGo 3.2) BaiDuSpider BaiDuSpider BaiDuSpider Baiduspider+(+http://www.baidu.com/search/spider.htm) Big Brother Big Brother (http://pauillac.inria.fr/~fpottier/) BlogBot 1.2 BlogBot/1.2 Boitho-robot 1.0 boitho.com-robot/1.0 BorderManager 3.0 Mozilla/4.0 (compatible; BorderManager 3.0) BrailleBot 1.0 BrailleBot 1.0 Bumblebee 1.0 bumblebee/1.0 (bumblebee@relevare.com; http://www.relevare.com/) Cd34 0.1 cd34/0.1 ComMOOnity LambdaMOO 1.8.1 ComMOOnity LambdaMOO/1.8.1 Cosmos 0.9 cosmos/0.9_(robot@xyleme.com) Crawl_Application Crawl_Application CrocCrawler 3.3 CrocCrawler v3.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) CrocCrawler 4.3 CrocCrawler v4.3 [en] (http://www.croccrawler.com) (X11; I; Linux 2.0.44 i686) Custo 2.0 Custo 2.0 (www.netwu.com) DeMozulator 1.0 DeMozulator 1.0 (MacOS, dMoz URL Check Agent, trebor@animeigo.com) DeepIndex DeepIndex (http://www.deepindex.com) DoCoMo 1.0 DoCoMo/1.0/N504i/c10/TB DoCoMo 1.0 DoCoMo/1.0/P504iS/c10/TB Dual Proxy Dual Proxy FAST-WebCrawler 3.2 FAST-WebCrawler/3.2 test FAST-WebCrawler 3.6 FAST-WebCrawler/3.6 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) FAST-WebCrawler 3.6 FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) FAST-WebCrawler 3.6 FAST-WebCrawler/3.6/FirstPage (crawler@fast.no; http://fast.no/support.php?c=faqs/crawler) FAST-WebCrawler 3.7 FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) FAST-WebCrawler 3.7 FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) FAST-WebCrawler 3.8 FAST-WebCrawler/3.8 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp) FAST-WebCrawler 3.x FAST-WebCrawler/3.x Multimedia FAST-WebCrawler 3.x FAST-WebCrawler/3.x Multimedia (mm dash crawler at fast dot no) FDSE robot Mozilla/4.0 (compatible: FDSE robot) Firefly 1.0 Firefly/1.0 (compatible; Mozilla 4.0; MSIE 5.5) FlickBot 1.1 Flickbot 1.1 RPT-HTTPClient/0.3-3 FlickBot 2.0 FlickBot 2.0 RPT-HTTPClient/0.3-3 Fluffy the spider Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com) FusionBot Mozilla/4.0 (compatible; MSIE 5.0; www.galaxy.com; http://www.pgts.com.au/; +http://www.galaxy.com/info/crawler.html) GAIS Robot 1.1A2 GAIS Robot/1.1A2 Gatherer 0.9 gatherer/0.9 Generic Generic GetRight 4.5 GetRight/4.5e Gigabot 1.0 Gigabot/1.0 Girafabot Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; Girafabot; girafabot at girafa dot com; http://www.girafa.com) Googlebot 2.1 Googlebot/2.1 (+http://www.googlebot.com/bot.html) Googlebot-Image 1.0 Googlebot-Image/1.0 (+http://www.googlebot.com/bot.html) Grub-client grub-client Grub-client 0.3.0 Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org) Grub-client 1.0.3 Mozilla/4.0 (compatible; grub-client-1.0.3; Crawl your own stuff with http://grub.org) Grub-client 1.0.4 Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org) Grub-client 1.0.5 Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org) Grub-client 1.0.6 Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org) Grub-client 1.0.7 Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org) Grub-client 1.07 Mozilla/4.0 (compatible; grub-client-1.07; Crawl your own stuff with http://grub.org) Grub-client 1.1.1 Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org) Grub-client 1.2.1 Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org) Grub-client 1.3.1 Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org) Grub-client 1.3.7 Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org) GulperBot 0.2.4 Crawler [en] (compatible; Crawler Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) GulperBot 0.2.4 Mozilla/5.0 [en] (compatible; Gulper Web Bot 0.2.4 www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot) HTTrack 3.0x Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98) Harvest-NG 1.0.2 Harvest-NG/1.0.2 Hget 0.3 hget/0.3 Hitwise Spider 1.0 Hitwise Spider v1.0 http://www.hitwise.com Htdig htdig Htdig 3.1.5 htdig/3.1.5 (admin@ipc-opc.lan) Htdig 3.1.5 htdig/3.1.5 (unconfigured@htdig.searchengine.maintainer) Htdig 3.1.6 htdig/3.1.6 (http://computerorgs.com) Httpcheck 1.0 Httpcheck/1.0 (Perl 5.006001) ICS 1.2.105 Mozilla/4.0 (compatible; ICS 1.2.105) IPiumBot IPiumBot laurion(dot)com Ia_archiver ia_archiver IcabotAccept lcabotAccept: */* IlTrovatore-Setaccio IlTrovatore-Setaccio (+http://www.iltrovatore.it) Imagefetch 0.1 imagefetch/0.1 libwww-perl/5.66 Indy Library Mozilla/3.0 (compatible; Indy Library) InfoSeek Sidewinder 1.0A InfoSeek Sidewinder/1.0A Ingrid 3.0 Mozilla/3.0 (INGRID/3.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints) Inktomi slurp Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi slurp Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi slurp Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html) Inktomi slurp Slurp/si-emb (slurp@inktomi.com; http://www.inktomi.com/slurp.html) InternetLinkAgent 3.1 InternetLinkAgent/3.1 Java 1.4.0 Java1.4.0 JoBo 1.3 JoBo/1.3 (http://www.matuschek.net/jobo.html) KMcrawler KMcrawler Knowledge Engine Knowledge Engine LNSpiderguy LNSpiderguy Larbin LARBIN-EXPERIMENTAL (efp@gmx.net) Larbin LARBIN-EXPERIMENTAL efp@gmx.net Larbin MSIE-5.13 (larbin@unspecified.mail) Larbin MSIE-5.13 larbin@unspecified.mail Larbin Mozilla (la2@unspecified.mail) Larbin Mozilla la2@unspecified.mail Larbin Mozilla/4.0 (efp@gmx.net) Larbin Mozilla/4.0 efp@gmx.net Larbin WinampMPEG/2.00 (larbin@unspecified.mail) Larbin WinampMPEG/2.00 larbin@unspecified.mail Larbin larbin (samualt9@bigfoot.com) Larbin larbin samualt9@bigfoot.com Larbin 2.6.2 larbin_2.6.2 (kalou@kalou.net) Larbin 2.6.2 larbin_2.6.2 (larbin2.6.2@unspecified.mail) Larbin 2.6.2 larbin_2.6.2 (larbin@correa.org) Larbin 2.6.2 larbin_2.6.2 (pimenas@systems.tuc.gr) Larbin 2.6.2 larbin_2.6.2 (sumeet_sobti@yahoo.com) Larbin 2.6.2 larbin_2.6.2 (vitalbox1@hotmail.com) Larbin 2.6.2 larbin_2.6.2 (vshelk@yahoo.com) Larbin 2.6.2 larbin_2.6.2 larbin2.6.2@unspecified.mail Larbin 2.6.2 larbin_2.6.2 larbin@correa.org Larbin 2.6.2 larbin_2.6.2 pimenas@systems.tuc.gr Larbin 2.6.2 larbin_2.6.2 sumeet_sobti@yahoo.com Larbin 2.6.2 larbin_2.6.2 vitalbox1@hotmail.com Libwww-perl Perl-Win32::Internet/0.082 Libwww-perl 5.3.2 / libwww/5.3.2 Libwww-perl 5.50 libwww-perl/5.50 Libwww-perl 5.51 libwww-perl/5.51 Libwww-perl 5.52 libwww-perl/5.52 FP/4.0 Libwww-perl 5.53 libwww-perl/5.53 Libwww-perl 5.63 libwww-perl/5.63 Libwww-perl 5.64 libwww-perl/5.64 Libwww-perl 5.65 libwww-perl/5.65 Libwww-perl 5.66 rawiswar/0.1 libwww-perl/5.66 Linkbot 3.0 Linkbot 3.0 Lite Bot 0616B Lite Bot 0616B Lwp-request 2.01 lwp-request/2.01 Lycos_Spider Lycos_Spider_(modspider) MS Data Access Microsoft Data Access Internet Publishing Provider Cache Manager MS Data Access Microsoft Data Access Internet Publishing Provider DAV MS Data Access Microsoft Data Access Internet Publishing Provider DAV 1.1 MS Data Access Microsoft Data Access Internet Publishing Provider Protocol Discovery MS FrontPage 4.0 MSFrontPage/4.0 MS FrontPage 4.0 Mozilla/2.0 (compatible; MS FrontPage 4.0) MS FrontPage 5.0 MSFrontPage/5.0 MS FrontPage 5.0 Mozilla/2.0 (compatible; MS FrontPage 5.0) MSIECrawler Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; MSIECrawler) MSIECrawler Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; Q312461; BTopenworld; MSIECrawler) MSIECrawler Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; MSIECrawler) MSIECrawler Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; MSIECrawler) MSIECrawler Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; MSIECrawler) MSIECrawler Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; matlas-2.0.2501; MSIECrawler) MSIECrawler Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler) MSProxy 2.0 MSProxy/2.0 Mercator 2.0 Mercator-2.0 MetaGer-LinkChecker MetaGer-LinkChecker Metacarta metacarta (crawler@metacarta.com) Metacarta metacarta crawler@metacarta.com Microsoft URL Control Microsoft URL Control - 5.00.3609 Microsoft URL Control Microsoft URL Control - 5.01.4319 Microsoft URL Control Microsoft URL Control - 6.00.8169 Microsoft URL Control Microsoft URL Control - 6.00.8862 Microsoft-ATL-Native 7.00 Microsoft-ATL-Native/7.00 MicrosoftPrototypeCrawler MicrosoftPrototypeCrawler (How's my crawling? mailto:newbiecrawler@hotmail.com) Moget 1.0 moget/1.0 (moget@goo.ne.jp) Moget 2.1 moget/2.1 (moget@goo.ne.jp) MysteryBot Mozilla/3.01 (compatible;) NG 1.0 NG/1.0 NPBot NPBot NPBot NPBot-1/2.0 NPBot NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html) NationalDirectory-WebSpider 1.3 NationalDirectory-WebSpider/1.3 NaverRobot 1.0 dloader(NaverRobot)/1.0 NetAnts 1.25 NetAnts/1.25 NetNose-Crawler 2.0 Mozilla/4.0 (compatible; MSIE 5.0; NetNose-Crawler 2.0; A New Search Experience: http://www.netnose.com) NetNoseCrawler v1.0 NetNoseCrawler/v1.0 NetResearchServer NetResearchServer(http://www.look.com) NetResearchServer 2.4 NetResearchServer/2.4(loopimprovements.com/robot.html) NetResearchServer 2.5 NetResearchServer/2.5(loopimprovements.com/robot.html) NetResearchServer 2.7 NetResearchServer/2.7(loopimprovements.com/robot.html) Netcraft Web Server Survey Mozilla/4.0 (compatible; Netcraft Web Server Survey) None none NutchCrawler Robot: NutchCrawler, Owner: wdavies@acm.org NutchOrg 0.03 NutchOrg/0.03-dev (Nutch; http://www.nutch.org/docs/bot.html; nutch-agent@lists.sourceforge.net) OWR_Crawler 0.1 OWR_Crawler 0.1 Obot Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot) Obot oBot Openbot 3.0 Openfind data gatherer, Openbot/3.0+(robot-response@openfind.com.tw;+http://www.openfind.com.tw/robot.html) OrangeBot OrangeBot Organica 2.24 Mozilla/4.0 (compatible; Advanced Email Extractor v2.24) PWS.Kiosk PWS.Kiosk - Content Filtering Parabot parabot (paracite@ecs.soton.ac.uk) Pavuk 0.9 pavuk/0.9pl28 i586-pc-cygwin Pavuk 0.9 pavuk/0.9pl29b i686-pc-linux-gnu Polybot 1.0 polybot 1.0 (http://cis.poly.edu/polybot/) Pompos 1.1 Pompos/1.1 http://pompos.iliad.fr Pompos 1.2 Pompos/1.2 http://pompos.iliad.fr Portal Manager 0.7 Portal Manager 0.7 Potbot 1.0 potbot 1.0 ProWebGuide Link Checker ProWebGuide Link Checker (http://www.prowebguide.com) Psbot 0.1 psbot/0.1 (+http://www.picsearch.com/bot.html) Pverify 1.2 pverify/1.2 QuepasaCreep v0.9.13 QuepasaCreep v0.9.13 RPT-HTTPClient 0.3 RPT-HTTPClient/0.3-3 Reifier reifier.org (admin@reifier.org) Reifier reifier.org admin@reifier.org Rico 0.1 rico/0.1 RobotMidareru 0.7 RobotMidareru/0.7libwww-perl/5.65 Robots.txt Validator Search Engine World Robots.txt Validator at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi Robozilla 1.0 Robozilla/1.0 SYCLIKControl LinkChecker Mozilla/5.0 (compatible; SYCLIKControl/LinkChecker;) Scooter 1.0 Scooter/1.0 Scooter 1.1 Scooter-ARS-1.1 Scooter 3.2 Scooter-3.2 Scooter 3.2 Scooter-3.2.BT Scooter 3.2 Scooter-3.2.EX Scooter 3.2 Scooter-3.2.FNR Scooter 3.2 Scooter-3.2.PDF Scooter 3.2 Scooter-3.2.SF0 Scooter 3.2 Scooter-3.2.TX.FNR Scooter 3.2 Scooter-3.2.XX0 Scooter 3.2 Scooter/3.2 Scooter 3.2 Scooter/3.2.SF0 Scooter 3.2 Scooter_x0-3.2.EX Scooter 3.3 Scooter/3.3 Scooter 3.3 Scooter/3.3.QA Scooter 3.3 Scooter/3.3.QA.pczukor Scooter 3.3 Scooter/3.3.vscooter Scooter 3.3 Scooter/3.3_SF Scrubby 2.2 Scrubby/2.2 (http://www.scrubtheweb.com/) Search Agent 1.0 Search Agent 1.0 Semanticdiscovery 0.1 semanticdiscovery/0.1 Sherlock 1.3 sherlock/1.3 httpget/1.3 Sitecheck InternetSeer.com Sitecheck sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com) Sitescooper 3.1.2 sitescooper/3.1.2 (http://sitescooper.org) libwww-perl/5.51 SlySearch 1.3 SlySearch/1.3 (http://www.slysearch.com) SlySearch 1.3 SlySearch/1.3 http://www.slysearch.com Speedy_Spider Speedy_Spider_(http://www.entireweb.com) Sqworm 2.9.85 Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux-gnu) Star Downloader Star Downloader Steeler 1.3 Steeler/1.3 (http://www.tkl.iis.u-tokyo.ac.jp/~crawler/) SuperCleaner 2.56 Mozilla/4.0 (compatible; SuperCleaner 2.56; Windows NT 5.1) Szukacz 1.5 Szukacz/1.5 Szukacz 1.5 Szukacz/1.5 (robot; www.szukacz.pl/jakdzialarobot.html; info@szukacz.pl) TclSOAP 1.0 Tcl http client package 1.0 TclSOAP 2.3 Tcl http client package 2.3 Teradex Mapper (Teradex Mapper; mapper@teradex.com; http://www.teradex.com) TulipChain 5.7 TulipChain/5.7 (http://ostermiller.org/tulipchain/) Java/1.4.0_02 (http://java.sun.com/) Windows_Me/4.90 TurnitinBot 1.4 TurnitinBot/1.4 (http://www.turnitin.com/robot/crawlerinfo.html) TurnitinBot 1.4 TurnitinBot/1.4 http://www.turnitin.com/robot/crawlerinfo.html TurnitinBot 1.5 TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html) TurnitinBot 1.5 TurnitinBot/1.5 http://www.turnitin.com/robot/crawlerinfo.html UIowaCrawler 2.0 UIowaCrawler/2.0 UdmSearch 3.1.20 UdmSearch/3.1.20 Vagabondo 2.0 Vagabondo/2.0 MT (webagent at wise-guys dot nl) Vagabondo 2.0 Vagabondo/2.0 MT (webagent@NOSPAMwise-guys.nl) Vagabondo 2.1 Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/) VoilaBot 1.6 Mozilla/4.0 (compatible; MSIE 5.0; Windows 95) VoilaBot; 1.6 VoilaBot 1.6 Mozilla/4.0_(compatible;_MSIE_5.0;_Windows_95)_VoilaBot/1.6 libwww/5.3.2 Vspider vspider W3C-checklink 2.90 W3C-checklink/2.90 libwww-perl/5.64 W3C-checklink 3.6.2.3 W3C-checklink/3.6.2.3 libwww-perl/5.64 W3C_Validator 1.183 W3C_Validator/1.183 libwww-perl/5.64 W3C_Validator 1.305.2.12 W3C_Validator/1.305.2.12 libwww-perl/5.64 WebFilter Robot 1.0 WebFilter Robot 1.0 WebSearch http://www.WebSearch.com.au/ (larbin2.6.2@unspecified.mail) WebSearch http://www.WebSearch.com.au/ - Australian Search Engine/3.1.3 (sites@websearch.com.au) WebSearch http://www.WebSearch.com.au/ - Australian Search Engine/3.1.6 (sites@websearch.com.au) WebSearch http://www.WebSearch.com.au/ larbin2.6.2@unspecified.mail WebSearch http://www.websearch.com.au (larbin2.6.2@unspecified.mail) WebSearch http://www.websearch.com.au larbin2.6.2@unspecified.mail WebSearch www.WebSearch.com.au (search@websearch.com.au) WebSearch www.WebSearch.com.au search@websearch.com.au WebSearch 2.0.1 WebSearch/2.0.1 (Dez@Blanchfield.COM.AU, http://www.WebSearch.com.au/) WebcraftBoot Thu Mar 27 18:20:34 CET 2003WebcraftBoot WebcraftBoot 1.4.1 Fri Nov 15 04:51:18 EST 2002WebcraftBoot Java/1.4.1_01 WebcraftBoot 1.4.2 Sun Apr 20 22:00:01 EDT 2003WebcraftBoot Java/1.4.2-beta WebcraftBoot 1.4.2 Tue Apr 15 22:00:03 EDT 2003WebcraftBoot Java/1.4.2-beta Webinator-indexer 2.56 Mozilla/3.0 (compatible; Webinator-indexer.cyberalert.com/2.56) Webwombat www.webwombat.com.au Wget 1.5.2 Wget/1.5.2 Wget 1.7 Wget/1.7 Wget 1.8 Wget/1.8 Wget 1.8.1 Wget/1.8.1 Wget 1.8.2 Wget/1.8.2 Willow 2.1 Willow Internet Crawler by Twotrees V2.1 Xenu Link Sleuth 1.1a Xenu's Link Sleuth 1.1a Xenu Link Sleuth 1.2b Xenu Link Sleuth 1.2b Xenu Link Sleuth 1.2d Xenu Link Sleuth 1.2d Zao 0.1 Zao/0.1 (http://www.kototoi.org/zao/) Zeus 57657 Zeus 57657 Webster Pro V2.9 Win32 ZyBorg 1.0 Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) ZyBorg 1.0 Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) unknown Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0) Fetch API Request unknown Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+.NET+CLR+1.1.4322)