New Community Website

Ordinarily, you'd be at the right spot, but we've recently launched a brand new community website... For the community, by the community.

Yay... Take Me to the Community!

Welcome to the DNN Community Forums, your preferred source of online community support for all things related to DNN.
In order to participate you must be a registered DNNizen

HomeHomeDevelopment and...Development and...Building ExtensionsBuilding ExtensionsModulesModulesMissing Credentials in WebCrawlerMissing Credentials in WebCrawler
Previous
 
Next
New Post
9/28/2011 2:11 PM
 
I am nearing completion of my WebCrawler Module. This module gets a list of pages from the PortalConroller and TabController, parsing through each one and checking for broken links, oversized images and missing image alt tags.

When running request.getresponse, any page requiring a secure connection returns an error. I've tried setting request.Credentials to CredentialCache.DefaultCredentials, but this does not help.

Is there a way to use the User's credentials? This seems to be the exact functionality I'm looking for.

This last SSL issue seems to be the last thing separating me from a fully working WebCrawler.
 
New Post
9/28/2011 6:39 PM
 
i would check what the requests look like in a http proxy such as http fiddler to see if there is a redirect going on e.g. if the request is HTTP initially but the page is marked as secure and force secure is enabled a 302 redirect will happen to the page to change it to HTTPS -if your crawler is making HTTP requests they'll not get any content and hence the error.

Also, you say youre using DefaultCredentials but have you wired up the ServerCertificateValidationCallback -as otherwise the certificate will appear as invalid and not render content - http://stackoverflow.com/questions/70... is a good example of how to do this.

Thanks,
Cathal

Buy the new Professional DNN7: Open Source .NET CMS Platform book Amazon US
 
New Post
9/30/2011 11:16 AM
 
Thank you for your response- I've given it a shot, and it does solve the error my crawler was throwing earlier.

Now, however, the response returned is a redirect to the login page; I've tried this with DefaultCredentials and with the user entering in their own credentials, which is then passed as a new NetworkCredential.

In both cases, the response was the login page.


Is there a way to get the page content by any other means? As in... through the tabcontroller? All I need is for my webcrawler to get the content of each page listed within the website; Past that, I will only be checking the size and status of images and outside links.
 
New Post
9/30/2011 10:53 PM
 
sure, you can use the TabController, or the syndicated results using the search engine API (e.g. those via ISearchable) - as to your issue, are you aware that httpwebrequest does not send cookies by default? This means that even if you're logged in it appears to the webserver that you're unauthenticated - to resolve this you can use a cookiecontainer - http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx . You can also pass through credentials to a page (eg. http://odetocode.com/Articles/162.aspx ), but please be aware that as this will be seen as a cross-site request, recent DotNetNuke versions (from 5.5.0) will catch it and deny it as a potential cross-site request forgery.

Buy the new Professional DNN7: Open Source .NET CMS Platform book Amazon US
 
New Post
9/30/2011 10:56 PM
 
oh, and I should have mentioned that NetworkCredential is only intended for basic, digest, ntlm and kerberos authentication (http://msdn.microsoft.com/en-us/library/system.net.networkcredential.aspx) and not for forms based authentication.

Buy the new Professional DNN7: Open Source .NET CMS Platform book Amazon US
 
Previous
 
Next
HomeHomeDevelopment and...Development and...Building ExtensionsBuilding ExtensionsModulesModulesMissing Credentials in WebCrawlerMissing Credentials in WebCrawler


These Forums are dedicated to discussion of DNN Platform and Evoq Solutions.

For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

  1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
  2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
  3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
  4. No Flaming or Trolling.
  5. No Profanity, Racism, or Prejudice.
  6. Site Moderators have the final word on approving / removing a thread or post or comment.
  7. English language posting only, please.