List Of Crawler Useragent Strings?

[leonardteo]

Hi guys,

 

I'm currently using Revive Adserver to serve ads for a site. Currently, clickthroughs appear to be highly inflated. I was able to use Splunk to search through Apache logs and find the culprit being bots/crawlers clicking on all the ads on all pages.

 

I want to avoid this in the future, and I'm looking for a list of all (or many) crawler/bot useragent strings as a flat text file with each useragent string on a new line to put into Revive Adserver to ignore.

 

I found a number of sites like http://www.botsvsbrowsers.com/, http://www.robotstxt.org/db.html, http://www.useragentstring.com/pages/Crawlerlist/ but none of these seem to offer a simple list of strings as a flat file that I can just use.

 

Does anyone have any suggestions?

 

Thanks,

 

Leonard

 

[peebuste]

Just extract the user-agent strings from Your www logs and adopt, step-by-step.

[Matt Tews]

Here are some I found.  Anyone care to add to the list?

 

BlackWidow

Bot mailto:[email protected]

ChinaClaw

Custo

DISCo

Download Demon

eCatch

EirGrabber

EmailSiphon

EmailWolf

Express WebPictures

ExtractorPro

EyeNetIE

FlashGet

GetRight

GetWeb!

Go!Zilla

Go-Ahead-Got-It

GrabNet

Grafula

HMView

HTTrack

Image Stripper

Image Sucker

Indy Library

InterGET

Internet Ninja

JetCar

JOC Web Spider

larbin

libghttp

LeechFTP

Mass Downloader

MIDown tool

Missigua

Mister PiX

Navroad

NearSite

NetAnts

NetSpider

Net Vampire

NetZIP

Octopus

Offline Explorer

Offline Navigator

PageGrabber

Papa Foto

pavuk

pcBrowser

RealDownload

ReGet

SiteSnagger

SmartDownload

SuperBot

SuperHTTP

Surfbot

tAkeOut

Teleport Pro

VoidEYE

Web Image Collector

Web Sucker

WebAuto

WebCopier

WebFetch

WebGo IS

WebLeacher

WebReaper

WebSauger

Website eXtractor

Website Quester

WebStripper

WebWhacker

WebZIP

Widow

WWWOFFLE

Xaldon WebSpider

Zeus

[Fred Weiss]

Just extract the user-agent strings from Your www logs and adopt, step-by-step.

Would you mind expanding on this? Which logs are you searching? What do you mean by "extract"? How would you differentiate between a legitimate bot and a malicious one?