I've been seeing a few errors in one of my portals due to (mostly) tiny spiders. Mostly I didn't care about them, but today I'm trying to trim the exceptions getting thrown on this specific portal. I'm curious if anyone has created custom browser caps entries for these spiders yet...

UserAgent: Mozilla/5.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
UserAgent: Mozilla/5.0 (compatible; Charlotte/1.1; http://www.searchme.com/support/)
UserAgent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
UserAgent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
UserAgent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
UserAgent: ip2 larbin2.6.3@unspecified.mail

Method: DotNetNuke.UI.WebControls.MenuNodeWriter.Render or DotNetNuke.UI.Skins.Skin.ProcessSlaveModule or System.Number.StringToNumber
 

I'm digging through the spiders browsercaps included in dnn5, but they're not the easiest thing to read... So, I was wondering if anyone else likes search engines to see their content and fixed this already (I already fixed the bug with the name of the app_browsers folder), I would love to see what you changed... if not expect a fix to be posted in the next 24 hours...

These files are very easy to read, after you figure out how all the data relates to each other, and really easy to test with Joe Brinkman's Test/Sample UserAgent/BrowserCap Application... 

Anyways... If anyone wants it... I've added a BrowserCaps file to my download section that will fix errors with spiders.

I've got a portal that used to use the DNN Gallery, but now it uses my Photo Gallery control. Now these stupid spiders are getting killed when crawling my site on Library\UI\Skins\Skin.vb:424. The old gallery is currently in the trash / recycling bin.

Looks like it's having trouble getting the modules controls, so I added a check for nothing and a redirect to an access denied message at line 422.

If objModuleControl Is Nothing Then
   'we have a problem....
    Response.Redirect(AccessDeniedURL("This content has been deleted"), True)
Else

Now, when I request mydomain.com/Default.aspx?tabid=1442&ctl=Viewer&mid=5322, I get redirected to a login control, after typing a valid login i'll get the content has been deleted warning. So, did my quick fix expose another bug? shouldn't the Access Denied control work for a non-authenticated user?

At first I tried to add a check if the module was deleted, but for some reason it's always returning false in the ProcessSlaveModule() function, that's how I got to the above fix...