Comments

1. kd says

   September 30, 2012 at 11:15 pm

   Interesting. Based on my own statistics, I see 90% bot traffic on my own portfolio of domains. I see 10% of traffic being humans and legitimate, the rest is Google Bot and so many other bots combined.

2. Grim says

   October 1, 2012 at 5:39 am

   If you can access your .htaccess file through FTP, copying and pasting the lines below into it will block many of the most common (bad) bots. Just remember to make a backup of your original .htaccess file before modifying it.

   [code]

   RewriteEngine On
   RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
   RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Curl [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
   RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
   RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EmailCollector [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
   RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
   RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
   RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
   RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
   RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
   RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
   RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
   RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
   RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
   RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
   RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
   RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
   RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
   RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
   RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
   RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
   RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
   RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
   RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
   RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
   RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
   RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
   RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
   RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
   RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
   RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
   RewriteCond %{HTTP_USER_AGENT} ^Zeus
   RewriteRule ^.* – [F,L]

   # Deny based on User-Agent
   SetEnvIfNoCase User-Agent “libwww-perl” bad_bot
   SetEnvIfNoCase User-Agent “psycheclone” bad_bot
   #
   # Allow universal access to robots.txt and custom 403 error page
   SetEnvIf Request_URI “robots\.txt$” allow_all
   SetEnvIf Request_URI “custom-403-page\.html$” allow_all
   #
   Order Deny,Allow
   Allow from env=allow_all
   Deny from env=bad_bot

   [/code]

3. Grim says

   October 1, 2012 at 5:42 am

   As a correction to the above, don’t include the lines [code] and [/code] in your .htaccess file.

4. Danny Pryor says

   October 1, 2012 at 6:23 am

   It would be interesting to see how they quantified their data. You know, 42.7% of all statistics are false.

   Anyway, I suppose this would be a problem for smaller companies that don’t know about things like Google Analytics, Omniture, Quantcast or the like, but for anyone else that uses these services, you’re probably pretty safe.

   Just my ever-so-humble opinion.

5. John Berryhill says

   October 1, 2012 at 12:01 pm

   Meanwhile, back at the ranch, UDRP panelists STILL can’t figure this out:

   Bacchus Gate Corporation d/b/a International Wine Accessories v. CKV and Port Media, Inc.
   Case No. D2008-0321

   http://www.wipo.int/amc/en/domains/decisions/html/2008/d2008-0321.html

   The Respondent’s secrecy appears not to end even there. When one of the panelists in this case attempted to investigate using the Way-Back Machine what pages have historically appeared at the Domain Name since its registration, he discovered that the Respondent had set up a “robot.txt” file on the website so as to prevent the Way-Back Machine from recording or revealing that information3. There sometimes may be good reasons why a legitimate commercial entity may wish to set up a “robot.txt” file in this fashion, but the Panel has some difficulty in understanding why it is that the operator of a website that provides sponsored search results would ever wish to do so. The natural inference in these circumstances is that the Respondent has something to hide and again this is suggestive of bad faith.

   ——–

   Those evil bot-blocking cybersquatters – just who do they think they are.

   If you try to block bot traffic, you are obviously a cybersquatter.

   If you don’t block bot traffic, you are ripping off advertisers, and you are obviously a cybersquatter.

   If you respond to a sales inquiry, you are a cybersquatter.

   If you don’t accept a purchase price offered in an inquiry, you are a cybersquatter.

   They’ll stone you when you’re trying to be so good
   They’ll stone you just like they said they would
   They’ll stone you when you’re trying to go home
   They’ll stone you when you’re there all alone
   But I would not feel so all alone
   Everybody must get stoned.