i would check what the requests look like in a http proxy such as http fiddler to see if there is a redirect going on e.g. if the request is HTTP initially but the page is marked as secure and force secure is enabled a 302 redirect will happen to the page to change it to HTTPS -if your crawler is making HTTP requests they'll not get any content and hence the error.

Also, you say youre using DefaultCredentials but have you wired up the ServerCertificateValidationCallback -as otherwise the certificate will appear as invalid and not render content - http://stackoverflow.com/questions/70... is a good example of how to do this.

Thanks,
Cathal

---

Buy the new Professional DNN7: Open Source .NET CMS Platform book Amazon US

sure, you can use the TabController, or the syndicated results using the search engine API (e.g. those via ISearchable) - as to your issue, are you aware that httpwebrequest does not send cookies by default? This means that even if you're logged in it appears to the webserver that you're unauthenticated - to resolve this you can use a cookiecontainer - http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.cookiecontainer.aspx . You can also pass through credentials to a page (eg. http://odetocode.com/Articles/162.aspx ), but please be aware that as this will be seen as a cross-site request, recent DotNetNuke versions (from 5.5.0) will catch it and deny it as a potential cross-site request forgery.

---

Buy the new Professional DNN7: Open Source .NET CMS Platform book Amazon US