mstuart Posted January 6, 2014 Report Share Posted January 6, 2014 Where do others go to get their list of bots to block? Thought it might be useful to offer a list here to refine over time. I offer a starting point - any amends welcome. googlebot Googlebot-Mobile Googlebot-Image bingbot slurp msnbot Adidxbot blekkobot teoma ia_archiver GingerCrawler webmon httrack webcrawler FAST-WebCrawler FAST Enterprise Crawler convera biglotron grub.org UsineNouvelleCrawler antibot netresearchserver speedy fluffy jyxobot bibnum.bnf findlink exabot gigabot msrbot seekbot ngbot panscient yacybot AISearchBot IOI ips-agent tagoobot MJ12bot dotbot woriobot yanga buzzbot mlbot yandex purebot Linguee Bot Voyager CyberPatrol voilabot baiduspider citeseerxbot spbot twengabot postrank turnitinbot scribdbot page2rss sitebot linkdex ezooms mail\.ru discobot heritrix findthatfile europarchive.org NerdByNature.Bot sistrix crawler ahrefsbot Aboundex domaincrawler wbsearchbot summify ccbot edisterbot seznambot ec2linkfinder gslfbot aihitbot intelium_bot facebookexternalhit yeti RetrevoPageAnalyzer lb-spider sogou lssbot careerbot wotbox wocbot ichiro DuckDuckBot lssrocketcrawler drupact webcompanycrawler acoonbot openindexspider gnam gnam spide web-archive-net.com.bot backlinkcrawler coccoc integromedb content crawler spider toplistbot seokicks-robot it2media-domain-crawler ip-web-crawler.com Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.