Jump to content

Recommended Posts

Posted
Hi guys,
 
I'm currently using Revive Adserver to serve ads for a site. Currently, clickthroughs appear to be highly inflated. I was able to use Splunk to search through Apache logs and find the culprit being bots/crawlers clicking on all the ads on all pages.
 
I want to avoid this in the future, and I'm looking for a list of all (or many) crawler/bot useragent strings as a flat text file with each useragent string on a new line to put into Revive Adserver to ignore.
 
I found a number of sites like http://www.botsvsbrowsers.com/, http://www.robotstxt.org/db.html, http://www.useragentstring.com/pages/Crawlerlist/ but none of these seem to offer a simple list of strings as a flat file that I can just use.
 
Does anyone have any suggestions?
 
Thanks,
 
Leonard
 
  • 2 months later...
  • 1 month later...
Posted

Here are some I found.  Anyone care to add to the list?

 

BlackWidow

Bot mailto:[email protected]

ChinaClaw

Custo

DISCo

Download Demon

eCatch

EirGrabber

EmailSiphon

EmailWolf

Express WebPictures

ExtractorPro

EyeNetIE

FlashGet

GetRight

GetWeb!

Go!Zilla

Go-Ahead-Got-It

GrabNet

Grafula

HMView

HTTrack

Image Stripper

Image Sucker

Indy Library

InterGET

Internet Ninja

JetCar

JOC Web Spider

larbin

libghttp

LeechFTP

Mass Downloader

MIDown tool

Missigua

Mister PiX

Navroad

NearSite

NetAnts

NetSpider

Net Vampire

NetZIP

Octopus

Offline Explorer

Offline Navigator

PageGrabber

Papa Foto

pavuk

pcBrowser

RealDownload

ReGet

SiteSnagger

SmartDownload

SuperBot

SuperHTTP

Surfbot

tAkeOut

Teleport Pro

VoidEYE

Web Image Collector

Web Sucker

WebAuto

WebCopier

WebFetch

WebGo IS

WebLeacher

WebReaper

WebSauger

Website eXtractor

Website Quester

WebStripper

WebWhacker

WebZIP

Widow

WWWOFFLE

Xaldon WebSpider

Zeus

  • 1 month later...
Posted

Just extract the user-agent strings from Your www logs and adopt, step-by-step.

Would you mind expanding on this? Which logs are you searching? What do you mean by "extract"? How would you differentiate between a legitimate bot and a malicious one?

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...