Products

Solutions

Resources

Partners

Community

Blog

About

QA

Ideas Test

New Community Website

Ordinarily, you'd be at the right spot, but we've recently launched a brand new community website... For the community, by the community.

Yay... Take Me to the Community!

Welcome to the DNN Community Forums, your preferred source of online community support for all things related to DNN.
In order to participate you must be a registered DNNizen

HomeHomeOur CommunityOur CommunityGeneral Discuss...General Discuss...Custom Browser UserAgent causes errorCustom Browser UserAgent causes error
Previous
 
Next
New Post
6/11/2009 5:18 PM
 

Hi,

We have a custom crawler that has a custom UserAgent and when it hits the site it generates an error http://localhost/Default.aspx?tabid=36&error=Value+cannot+be+null.%0D%0AParameter+name%3a+String&content=0

I've added a browser file to the App_Browser directory (OurCrawler.Broswer) with the following:

<browsers>
 <browser id="OurCrawler" parentID="Default">
     <sampleHeaders>
       <header name="User-Agent" value="OurCrawler" />
     </sampleHeaders>
 <identification>
  <userAgent match="OurCrawler" />
 </identification>
 <capabilities>
       <capability name="crawler" value="true" />
       <capability name="browser" value="OurCrawler" />
       <capability name="crawler" value="true" />
       <capability name="isMobileDevice" value="false" />
 </capabilities>
 </browser>
</browsers>

Even after installing the browser file I still get the error...  what do I need to do to get the site to pick up the crawler.

thanks, -Jason

 
New Post
6/11/2009 8:09 PM
 

The error happens because there is no version available for the crawler. you need to add the following nodes to capabilities:

      <capability name="majorversion" value="0" />
      <capability name="minorversion" value=".0" />
      <capability name="version" value="0.0" />

(or other applicable versions...)

A better solution would be to add a version for your crawler in its user agent. Ideally, if your crawlers user agent adheres to the regex in the general crawler definition, you would not need to create your own capabilities file. The default definition is this:

  <browser id="DefaultCrawler" parentID="Default">
    <identification>
      <userAgent nonMatch="(?i-s:crawler|bot|spider|larbin|search|indexer|archiver|Nutch)" />
      <capability name="isMobileDevice" nonMatch="(?i-s:true)" />      
      <userAgent match="(?'browser'[^/]*)/(?'version'(?'major'\d+)(?'minor'\.\d+)\w*)|(?'browser'^.*)[ /](?'version'(?'major'\d+)(?'minor'\.\d+)\w*)" />     
    </identification>
    <capabilities>
      <capability name="crawler" value="true" />
      <capability name="browser" value="${browser}" />
      <capability name="majorversion" value="${major}" />
      <capability name="minorversion" value="${minor}" />
      <capability name="version" value="${version}" />
      <capability name="tagWriter" value="System.Web.UI.HtmlTextWriter" />
    </capabilities>
  </browser>

 


Erik van Ballegoij, Former DNN Corp. Employee and DNN Expert

DNN Blog | Twitter: @erikvb | LinkedIn: Erik van Ballegoij on LinkedIn

 
Previous
 
Next
HomeHomeOur CommunityOur CommunityGeneral Discuss...General Discuss...Custom Browser UserAgent causes errorCustom Browser UserAgent causes error


These Forums are dedicated to discussion of DNN Platform and Evoq Solutions.

For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

  1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
  2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
  3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
  4. No Flaming or Trolling.
  5. No Profanity, Racism, or Prejudice.
  6. Site Moderators have the final word on approving / removing a thread or post or comment.
  7. English language posting only, please.
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out