Products

Solutions

Resources

Partners

Community

Blog

About

QA

Ideas Test

New Community Website

Ordinarily, you'd be at the right spot, but we've recently launched a brand new community website... For the community, by the community.

Yay... Take Me to the Community!

Welcome to the DNN Community Forums, your preferred source of online community support for all things related to DNN.
In order to participate you must be a registered DNNizen

HomeHomeOur CommunityOur CommunityGeneral Discuss...General Discuss...Is there a way of telling if a user is a search engine spider?Is there a way of telling if a user is a search engine spider?
Previous
 
Next
New Post
5/8/2007 7:59 AM
 

Hi,

My Error log is filling up with loads of "Can't load private UserProfile/Event/Group" where a search engine spider tries to go to a URL with an item ID that is private. A real user would never see these items, unless they hack the URL, so I throw an error (so I can keep an eye on this) but lately the spiders are taking over and keep going to these URLS (of course they are non-authenticated) so it throws the error.

I would like to do a check in the code and throw the error if it is a user and handle it differently if it is a spider. Is there any way of parsing the referrer with a regex expression to determine this (does anyone have a RegEx mask or a better idea of handling this?)

It wont be foolproof of course, but should reduce error log bloat...


Entrepreneur

PokerDIY Tournament Manager - PokerDIY Tournament Manager<
PokerDIY Game Finder - Mobile Apps powered by DNN
PokerDIY - Connecting Poker Players

 
New Post
5/8/2007 8:14 AM
 

 

Hi Rodney,

You can usually tell it is a spider by the User Agent field.

The Free PageBlaster allows you to block abusive user agents using Regex.  They will receive a 403 "not authorized" response returned before they are able to consume any resources.

 


DotNetNuke Modules from Snapsis.com
 
New Post
5/15/2007 2:22 PM
 

Hi John,

 

The problem is that all the pages are and should be seen by all users - I do want spiders to index them, BUT for some reason some of them are not passing in a querystring param (it's not just going directly to the page, it will have the variable name (ie. UserID) but no value) - so this is logging an error (as it should).

I don't want to tell them not to go to it at all (or I do IF there is no valid querystring param in the URL - is it possible to do that with a RegEx expression on PageBlaster? It's not really a problem as such, it just means I get a lot of log errors. I guess the spiders will index the DNN error on that URL.


Entrepreneur

PokerDIY Tournament Manager - PokerDIY Tournament Manager<
PokerDIY Game Finder - Mobile Apps powered by DNN
PokerDIY - Connecting Poker Players

 
New Post
5/15/2007 2:32 PM
 

Maybe you should handle it in your code that expects UserId to have a value.  Maybe if you default UserId to -1 the problem would be handled.

No, PageBlaster doesn't have blocking of pages based on Url.  I'll have to think about if that would actually have much validity, but maybe I could add it.

Actually, now that I am thinkng about it, I guess it could be used for some validation of the Url like the UrlScan in the IIS Lockdown tool.


DotNetNuke Modules from Snapsis.com
 
Previous
 
Next
HomeHomeOur CommunityOur CommunityGeneral Discuss...General Discuss...Is there a way of telling if a user is a search engine spider?Is there a way of telling if a user is a search engine spider?


These Forums are dedicated to discussion of DNN Platform and Evoq Solutions.

For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

  1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
  2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
  3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
  4. No Flaming or Trolling.
  5. No Profanity, Racism, or Prejudice.
  6. Site Moderators have the final word on approving / removing a thread or post or comment.
  7. English language posting only, please.
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out