Jump to content

List Of Crawler Useragent Strings?


leonardteo

Recommended Posts

Hi guys,
 
I'm currently using Revive Adserver to serve ads for a site. Currently, clickthroughs appear to be highly inflated. I was able to use Splunk to search through Apache logs and find the culprit being bots/crawlers clicking on all the ads on all pages.
 
I want to avoid this in the future, and I'm looking for a list of all (or many) crawler/bot useragent strings as a flat text file with each useragent string on a new line to put into Revive Adserver to ignore.
 
I found a number of sites like http://www.botsvsbrowsers.com/, http://www.robotstxt.org/db.html, http://www.useragentstring.com/pages/Crawlerlist/ but none of these seem to offer a simple list of strings as a flat file that I can just use.
 
Does anyone have any suggestions?
 
Thanks,
 
Leonard
 
Link to comment
Share on other sites

  • 2 months later...
  • 1 month later...

Here are some I found.  Anyone care to add to the list?

 

BlackWidow

Bot mailto:[email protected]

ChinaClaw

Custo

DISCo

Download Demon

eCatch

EirGrabber

EmailSiphon

EmailWolf

Express WebPictures

ExtractorPro

EyeNetIE

FlashGet

GetRight

GetWeb!

Go!Zilla

Go-Ahead-Got-It

GrabNet

Grafula

HMView

HTTrack

Image Stripper

Image Sucker

Indy Library

InterGET

Internet Ninja

JetCar

JOC Web Spider

larbin

libghttp

LeechFTP

Mass Downloader

MIDown tool

Missigua

Mister PiX

Navroad

NearSite

NetAnts

NetSpider

Net Vampire

NetZIP

Octopus

Offline Explorer

Offline Navigator

PageGrabber

Papa Foto

pavuk

pcBrowser

RealDownload

ReGet

SiteSnagger

SmartDownload

SuperBot

SuperHTTP

Surfbot

tAkeOut

Teleport Pro

VoidEYE

Web Image Collector

Web Sucker

WebAuto

WebCopier

WebFetch

WebGo IS

WebLeacher

WebReaper

WebSauger

Website eXtractor

Website Quester

WebStripper

WebWhacker

WebZIP

Widow

WWWOFFLE

Xaldon WebSpider

Zeus

Link to comment
Share on other sites

  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...