leonardteo Posted September 19, 2014 Report Share Posted September 19, 2014 Hi guys, I'm currently using Revive Adserver to serve ads for a site. Currently, clickthroughs appear to be highly inflated. I was able to use Splunk to search through Apache logs and find the culprit being bots/crawlers clicking on all the ads on all pages. I want to avoid this in the future, and I'm looking for a list of all (or many) crawler/bot useragent strings as a flat text file with each useragent string on a new line to put into Revive Adserver to ignore. I found a number of sites like http://www.botsvsbrowsers.com/, http://www.robotstxt.org/db.html, http://www.useragentstring.com/pages/Crawlerlist/ but none of these seem to offer a simple list of strings as a flat file that I can just use. Does anyone have any suggestions? Thanks, Leonard Quote Link to comment Share on other sites More sharing options...
peebuste Posted November 19, 2014 Report Share Posted November 19, 2014 Just extract the user-agent strings from Your www logs and adopt, step-by-step. Quote Link to comment Share on other sites More sharing options...
Matt Tews Posted January 5, 2015 Report Share Posted January 5, 2015 Here are some I found. Anyone care to add to the list? BlackWidow Bot mailto:[email protected] ChinaClaw Custo DISCo Download Demon eCatch EirGrabber EmailSiphon EmailWolf Express WebPictures ExtractorPro EyeNetIE FlashGet GetRight GetWeb! Go!Zilla Go-Ahead-Got-It GrabNet Grafula HMView HTTrack Image Stripper Image Sucker Indy Library InterGET Internet Ninja JetCar JOC Web Spider larbin libghttp LeechFTP Mass Downloader MIDown tool Missigua Mister PiX Navroad NearSite NetAnts NetSpider Net Vampire NetZIP Octopus Offline Explorer Offline Navigator PageGrabber Papa Foto pavuk pcBrowser RealDownload ReGet SiteSnagger SmartDownload SuperBot SuperHTTP Surfbot tAkeOut Teleport Pro VoidEYE Web Image Collector Web Sucker WebAuto WebCopier WebFetch WebGo IS WebLeacher WebReaper WebSauger Website eXtractor Website Quester WebStripper WebWhacker WebZIP Widow WWWOFFLE Xaldon WebSpider Zeus Quote Link to comment Share on other sites More sharing options...
Fred Weiss Posted February 7, 2015 Report Share Posted February 7, 2015 Just extract the user-agent strings from Your www logs and adopt, step-by-step. Would you mind expanding on this? Which logs are you searching? What do you mean by "extract"? How would you differentiate between a legitimate bot and a malicious one? Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.