Top 165 Web Crawlers (Total 254) |
|
Name
|
Hits
|
1. |
Web Crawler - ZoomInformation Bot |
2 |
2. |
Web Crawler - Zauba Crawler/1.0 (Zauba Search for Research; http://www.zauba.io/; admin@zauba.io) |
6 |
3. |
Web Crawler - yyqnnlrcewrmaitmsimi |
6 |
4. |
Web Crawler - YPBot/Raven1.1.3 (compatible; Googlebot/2.1;+http://www.yellowpages.com/about/legal/crawl) |
2 |
5. |
Web Crawler - Yahoo! Slurp/Site Explorer |
1 |
6. |
Web Crawler - Yahoo! Slurp China |
1 |
7. |
Web Crawler - Y!J-BSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/blog-search/) |
163 |
8. |
Web Crawler - Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/) |
5 |
9. |
Web Crawler - www.integromedb.org/Crawler |
20 |
10. |
Web Crawler - Willow Internet Crawler by Twotrees V2.1 |
2 |
11. |
Web Crawler - WebZIP/x.x (http://www.spidersoft.com) |
1 |
12. |
Web Crawler - WebZIP/3.5 (http://www.spidersoft.com) |
2 |
13. |
Web Crawler - WebTarantula.com Crawler |
32 |
14. |
Web Crawler - WebCrawler |
8 |
15. |
Web Crawler - WebAlta Crawler/2.0 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
463 |
16. |
Web Crawler - WebAlta Crawler/1.3.34 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
361 |
17. |
Web Crawler - WebAlta Crawler/1.3.23 (http://www.webalta.net/ru/about_webmaster.html) (Windows; U; Windows NT 5.1; ru-RU) |
245 |
18. |
Web Crawler - W3C_Validator/1.654 |
6 |
19. |
Web Crawler - W3C_Validator/1.3 http://validator.w3.org/services |
3 |
20. |
Web Crawler - Virusdie crawler/3.0 |
252 |
21. |
Web Crawler - VelenPublicWebCrawler/0.1 |
1 |
22. |
Web Crawler - VelenPublicWebCrawler (velen.io) |
19 |
23. |
Web Crawler - UXCrawlerBot |
4 |
24. |
Web Crawler - Utopia WebWasher 3.0 |
2 |
25. |
Web Crawler - User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
46 |
26. |
Web Crawler - UnisterBot (Mozilla/5.0 compatible; crawler@unister-gmbh.de) |
1 |
27. |
Web Crawler - UCMore Crawler App |
3 |
28. |
Web Crawler - TurnitinBot/3.0 (http://www.turnitin.com/robot/crawlerinfo.html) |
3156 |
29. |
Web Crawler - TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html) |
2967 |
30. |
Web Crawler - TurnitinBot (https://turnitin.com/robot/crawlerinfo.html) |
7 |
31. |
Web Crawler - Teemer (NetSeer, Inc.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
1 |
32. |
Web Crawler - Teemer (NetSeer, Inc. is a Los Angeles based Internet startup company.; http://www.netseer.com/crawler.html; crawler@netseer.com) |
2 |
33. |
Web Crawler - spotinfluence/Nutch-1.4 (Spot Influence crawler; http://spotinfluence.com; hello at spotinfluence dot com) |
1 |
34. |
Web Crawler - spiderman |
953 |
35. |
Web Crawler - Social Media Crawler using your Home URL on Twitter,Facebook,Myspace,Linkedin by ProfileCapture - contact profilecapture@gmail.com to report any problems with my crawling. http://profilecapture.com |
1 |
36. |
Web Crawler - slurp, yahoo! slurp, slurp/2.0, inktomi slurp, slurp.so/1.0 |
1 |
37. |
Web Crawler - SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com) |
9 |
38. |
Web Crawler - ShrinkTheWeb.com Crawler v1.0 (+http://www.shrinktheweb.com) |
6 |
39. |
Web Crawler - SheenBot/SheenBot-1.0.4 (Sheen web crawler) |
2 |
40. |
Web Crawler - SheenBot/SheenBot-1.0.3 (Sheen web crawler) |
3 |
41. |
Web Crawler - SheenBot/SheenBot-1.0.2 (Sheen web crawler) |
4 |
42. |
Web Crawler - SheenBot/SheenBot-1.0.1 (Sheen web crawler) |
4 |
43. |
Web Crawler - SheenBot/SheenBot-1.0.0 (Sheen web crawler) |
1 |
44. |
Web Crawler - ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com) |
542 |
45. |
Web Crawler - SapphireWebCrawler/Nutch-1.0-dev (Sapphire Web Crawler using Nutch; http://boston.lti.cs.cmu.edu/crawler/; mhoy@cs.cmu.edu) |
5 |
46. |
Web Crawler - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) |
5718 |
47. |
Web Crawler - SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html) |
3 |
48. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)/Nutch-1.13 |
2 |
49. |
Web Crawler - rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com) |
8 |
50. |
Web Crawler - rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org) |
1 |
51. |
Web Crawler - RMA/1.0 (compatible; RealMedia) |
2 |
52. |
Web Crawler - renlifangbot/1.0 (+http://search.msn.com/msnbot.htm) |
57 |
53. |
Web Crawler - REAP-crawler Nutch/Nutch-1.0-dev (Reap Project; http://reap.cs.cmu.edu/REAP-crawler/; Reap Project) |
1 |
54. |
Web Crawler - R6_FeedFetcher_(www.radian6.com/crawler) |
1 |
55. |
Web Crawler - Quick-Crawler (+https://www.scrapinghub.com/) |
4 |
56. |
Web Crawler - Python-urllib/2.7 (+http://example.com/crawler.html) |
5 |
57. |
Web Crawler - ppffpbuumkxcfHakxcmdhont wrma x |
1 |
58. |
Web Crawler - Nokia6680/1.0 ((4.04.07) SymbianOS/8.0 Series60/2.6 Profile/MIDP-2.0 Configuration/CLDC-1.1 (for mobile crawler) ) |
4 |
59. |
Web Crawler - NextGenSearchBot 1 (for information visit http://www.zoominfo.com/About/misc/NextGenSearchBot.aspx) |
348 |
60. |
Web Crawler - netEstate NE Crawler (+http://www.website-datenbank.de/) |
14797 |
61. |
Web Crawler - netEstate NE Crawler (+http://www.sengine.info/) |
7435 |
62. |
Web Crawler - MSR-ISRCCrawler |
18 |
63. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ |
19725 |
64. |
Web Crawler - msnbot/2.0b (+http://search.msn.com/msnbot.htm) |
21479 |
65. |
Web Crawler - msnbot/2.0b |
9 |
66. |
Web Crawler - msnbot/1.1 (+http://search.msn.com/msnbot.htm) |
12779 |
67. |
Web Crawler - msnbot/1.1 ( http://search.msn.com/msnbot.htm) |
5 |
68. |
Web Crawler - msnbot/1.0 (+http://search.msn.com/msnbot.htmĦħ) |
1 |
69. |
Web Crawler - msnbot/1.0 (+http://search.msn.com/msnbot.htm) |
6558 |
70. |
Web Crawler - msnbot/1.0 ( http://search.msn.com/msnbot.htm) |
5 |
71. |
Web Crawler - msnbot/0.11 ( http://search.msn.com/msnbot.htm) |
4 |
72. |
Web Crawler - msnbot/0.01 (+http://search.msn.com/msnbot.htm) |
530 |
73. |
Web Crawler - msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) |
739 |
74. |
Web Crawler - msnbot-media/1.1 ( http://search.msn.com/msnbot.htm) |
4 |
75. |
Web Crawler - msnbot-media/1.0 (+http://search.msn.com/msnbot.htm) |
3351 |
76. |
Web Crawler - Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 4.5) |
1 |
77. |
Web Crawler - Mozilla/5.0 (Windows NT 6.2; WOW64) Runet-Research-Crawler (itrack.ru/research/cmsrate; rating@itrack.ru) |
2 |
78. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0 |
7 |
79. |
Web Crawler - Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0 |
11 |
80. |
Web Crawler - Mozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html) |
5 |
81. |
Web Crawler - Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
2 |
82. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) |
89861 |
83. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/last.php) |
2 |
84. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp; http://abwc.iespana.es/forum/index.php) |
2 |
85. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) |
25933 |
86. |
Web Crawler - Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html) |
7 |
87. |
Web Crawler - Mozilla/5.0 (compatible; WebHistoryBot/1.2.1 IS NOT SE bot like Googlebot/2.1; +http://www.google.com/bot.html,Yahoo! Slurp or Bingbot) |
3 |
88. |
Web Crawler - Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io) |
6 |
89. |
Web Crawler - Mozilla/5.0 (compatible; TinEye-bot-live/1.31; +http://www.tineye.com/crawler.html) |
2 |
90. |
Web Crawler - Mozilla/5.0 (compatible; TestCrawler) |
9 |
91. |
Web Crawler - Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) |
57403 |
92. |
Web Crawler - mozilla/5.0 (compatible; SandobaCrawler/1.0; +https://www.sandoba.com/en/crawler/) |
1 |
93. |
Web Crawler - Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot) |
1 |
94. |
Web Crawler - Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler) |
1 |
95. |
Web Crawler - Mozilla/5.0 (compatible; Peew/1.0; http://www.peew.de/crawler/) |
3 |
96. |
Web Crawler - Mozilla/5.0 (compatible; PathDefender/1.0; +http://www.pathdefender.com/help/crawler) |
1 |
97. |
Web Crawler - Mozilla/5.0 (compatible; oBot/2.3.1; http://filterdb.iss.net/crawler/) |
350 |
98. |
Web Crawler - Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/) |
148 |
99. |
Web Crawler - Mozilla/5.0 (compatible; NOT Googlebot/2.1; +http://www.google.com/bot.html) |
4 |
100. |
Web Crawler - Mozilla/5.0 (compatible; Nigma.ru/3.0; crawler@nigma.ru) |
8 |
101. |
Web Crawler - Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com) |
38 |
102. |
Web Crawler - Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com) |
109 |
103. |
Web Crawler - Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler) |
388697 |
104. |
Web Crawler - Mozilla/5.0 (compatible; KaloogaBot; http://www.kalooga.com/info.html?page=crawler) |
400 |
105. |
Web Crawler - Mozilla/5.0 (compatible; heritrix/3.0.0 +http://dataclip.com/crawler.html) |
7 |
106. |
Web Crawler - Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) |
114 |
107. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html) |
25 |
108. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/vb/index.php) |
2 |
109. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/last.php) |
1 |
110. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/forum/index.php) |
6 |
111. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://abwc.iespana.es/blog/last.php) |
1 |
112. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
4212747 |
113. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html |
3 |
114. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com) |
3 |
115. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io) |
126 |
116. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html) |
21 |
117. |
Web Crawler - Mozilla/5.0 (compatible; Googlebot/2.1 +http://www.googlebot.com/bot.html) |
1 |
118. |
Web Crawler - Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/) |
17 |
119. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/pages/crawler) |
4 |
120. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +https://gimmeusa.com/crawler.html) |
3 |
121. |
Web Crawler - Mozilla/5.0 (compatible; GimmeUSAbot/1.0; +http://gimmeusa-update.com/crawler) |
2 |
122. |
Web Crawler - Mozilla/5.0 (compatible; fr-crawler/1.1) |
5099 |
123. |
Web Crawler - Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler) |
65 |
124. |
Web Crawler - Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/) |
496 |
125. |
Web Crawler - Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org) |
12142 |
126. |
Web Crawler - Mozilla/5.0 (compatible; DomainMacroCrawler/0.1; +http://domainmacro.com) |
1 |
127. |
Web Crawler - Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions. |
15 |
128. |
Web Crawler - Mozilla/5.0 (compatible; ca-crawler/1.0) |
2025 |
129. |
Web Crawler - Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) |
84502 |
130. |
Web Crawler - Mozilla/5.0 (compatible; bixolabs/1.0; +http://bixolabs.com/crawler/general; crawler@bixolabs.com) |
6 |
131. |
Web Crawler - Mozilla/5.0 (compatible; BIXOCRAWLER; +http://wiki.github.com/bixo/bixo/bixocrawler; bixo-dev@yahoogroups.com) |
3 |
132. |
Web Crawler - Mozilla/5.0 (compatible; Bender; http://sites.google.com/site/bendercrawler) |
17 |
133. |
Web Crawler - Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler) |
207728 |
134. |
Web Crawler - Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; ) |
5 |
135. |
Web Crawler - Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/webcrawler.html) Gecko/2008032620 |
5 |
136. |
Web Crawler - Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/) |
24699 |
137. |
Web Crawler - Mozilla/4.7 (compatible; OffByOne; Windows 2000) Webster Pro V3.4 |
30 |
138. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com) |
30 |
139. |
Web Crawler - Mozilla/4.0 compatible ZyBorg/1.0 (wn.zyborg@looksmart.net; http://www.WISEnutbot.com) |
1 |
140. |
Web Crawler - Mozilla/4.0 (compatible; Powermarks/3.5; Windows 95/98/2000/NT) |
7933 |
141. |
Web Crawler - Mozilla/4.0 (CMS Crawler: http://www.cmscrawler.com) |
26 |
142. |
Web Crawler - Mozilla/2.0 (compatible; NEWT ActiveX; Win32) |
1 |
143. |
Web Crawler - meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
18968 |
144. |
Web Crawler - MBCrawler/1.0 (https://monitorbacklinks.com/robot) |
2 |
145. |
Web Crawler - MBCrawler/1.0 (https://monitorbacklinks.com) |
98 |
146. |
Web Crawler - MauiBot (crawler.feedback+wc@gmail.com) |
8886 |
147. |
Web Crawler - MauiBot (crawler.feedback+dc@gmail.com) |
464 |
148. |
Web Crawler - LSSRocketCrawler/1.0 LightspeedSystems |
231 |
149. |
Web Crawler - LivelapBot/0.2 (http://site.livelap.com/crawler) |
2 |
150. |
Web Crawler - LinksCrawler 0.1beta |
3 |
151. |
Web Crawler - librabot/1.0 (+http://search.msn.com/msnbot.htm) |
52 |
152. |
Web Crawler - KSCrawler/Nutch-1.0 (http://www.kindsight.net/en/kscrawler; crawler@kindsight.net) |
2 |
153. |
Web Crawler - kalooga/KaloogaBot (Kalooga; http://www.kalooga.com/info.html?page=crawler) |
57 |
154. |
Web Crawler - Jigsaw/2.2.5 W3C_CSS_Validator_JFouffa/2.0 |
2 |
155. |
Web Crawler - Jayde Crawler. http://www.jayde.com |
3 |
156. |
Web Crawler - Jayde crawler |
3 |
157. |
Web Crawler - IP-Guide.com Crawler/1.0 (https://ip-guide.com) |
1 |
158. |
Web Crawler - ilial/Nutch-0.9 (Ilial, Inc. is a Los Angeles based Internet startup company.; http://www.ilial.com/crawler; crawl@ilial.com) |
2 |
159. |
Web Crawler - ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
160. |
Web Crawler - ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html) |
2 |
161. |
Web Crawler - ICC-Crawler/2.0 (Mozilla-compatible; ; http://kc.nict.go.jp/project1/crawl.html) |
2 |
162. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl-contact(at)ml(dot)nict(dot)go(dot)jp) |
3 |
163. |
Web Crawler - ICC-Crawler(Mozilla-compatible; http://kc.nict.go.jp/icc/crawl.html; icc-crawl(at)ml(dot)nict(dot)go(dot)jp) |
5 |
164. |
Web Crawler - ia_archiver/8.5 (Windows NT 7.2; ) |
1 |
165. |
Web Crawler - ia_archiver/8.2 (Windows NT 4.4; ) |
1 |