Products

Solutions

Resources

Partners

Community

Blog

About

QA

Ideas Test

New Community Website

Ordinarily, you'd be at the right spot, but we've recently launched a brand new community website... For the community, by the community.

Yay... Take Me to the Community!

Welcome to the DNN Community Forums, your preferred source of online community support for all things related to DNN.
In order to participate you must be a registered DNNizen

HomeHomeUsing DNN Platf...Using DNN Platf...Administration ...Administration ...CanCan't get crawler to work (intranet)
Previous
 
Next
New Post
7/29/2009 12:39 PM
 

We've been trying to get ANYTHING to work on our intranet portal in terms of a decent search but so far can't get anything to crawl it.  Our portal is set up in IIS to be available to Anonymous users except for a single WindowsLogin.aspx page which has Windows Authentication enabled on it so we can detect their AD/Windows user and automatically log them in based on that.  All the pages on the portal are otherwise locked down and require a logged in user to view them.

We have tried Google Mini and specified some Windows credentials to use, we have attempted Microsoft Search Server Express, WrenSoft Zoom, and others.  We have used OpenSearch and it actually works for indexing the site but the search results are horrendous and frankly not really all that useful.

We'd really prefer to get either the Google Mini or Search Server Express to work so if anybody out there has had success in setting up such an environment I'd love to hear what I'm missing.  As an aside, the Search Server Express also seems to have difficulty crawling a "normal" DNN site; the log shows that it is stripping off the "default.aspx" portion on the end of the URL resulting in an incomplete URL. 

Any help greatly appreciated!


-- Jon Seeley
DotNetNuke Modules
Custom DotNetNuke and .NET Development
http://www.seeleyware.com
 
New Post
8/5/2009 4:44 PM
 

Even though nobody responded I figured I'd follow-up so anybody else with the same problem can benefit.

So Google Mini (in our setup) is a no-go.  It just won't cut the cheese so to speak.  That leaves Microsoft Search Server 2008 and WrenSoft Zoom since OpenSearch "worked" but had awful results (not to mention it totally nuked the server while indexing).

I managed to get WrenSoft to work by logging in to our portal manually in IE and then saving the authentication ("Remember Me").  Next, I had to leave the browser Window open and then go back into WrenSoft, tell it to use my manual login page and also to use the cookies from IE and my computer.  With that setup, I was able to get Zoom to index our portal.  Worked just fine, but a manual process and a pain in the bum.  Search results were better than OpenSearch but still not perfect.  Still, it is nice being able to add things into the HTML on the site and have it flat out skip it in the indexer (ie, have it skip the whole menu).

What I was pleased to get working was Search Express.  I ended up writing my own ultra basic login module and slapping it on a page with no skin.  I'm unsure if the regular login module would work since everytime I tried it through the Search Express authentication form DNN would crash (pretty sure it has to do with something "missing" when using that control).  Next I set up a rule in Search Express and told it to use forms Authentication, pointed it to the URL containing my ultra basic module, and then proceeded to set the credentials.  The key here is that when it successfully logs in you have to wait for it to close the Window for you (up to 30 seconds).  *DO NOT CLOSE THE WINDOW YOURSELF*.  That is what killed me... I was closing it myself assuming that was what I needed to do and it wouldn't ever set the creds for me.  Doh!

That's pretty much it in a nutshell.  Additionally it is VERY helpful to use URLMaster.  It cleans up the URLs and makes them much more SEO-friendly.

The Search Express is sweet though because now I can create my own skin object for the search, plug it to a module, and then use a web-service to populate the search on our DNN portal.  We could use the Search Express search page itself and reskin it to look like the portal, but frankly the webservice way will be much more effective.  Woot!


-- Jon Seeley
DotNetNuke Modules
Custom DotNetNuke and .NET Development
http://www.seeleyware.com
 
New Post
8/5/2009 5:27 PM
 

Thanks Jon - that was enlightening,.



Alex Shirley


 
Previous
 
Next
HomeHomeUsing DNN Platf...Using DNN Platf...Administration ...Administration ...CanCan't get crawler to work (intranet)


These Forums are dedicated to discussion of DNN Platform and Evoq Solutions.

For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

  1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
  2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
  3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
  4. No Flaming or Trolling.
  5. No Profanity, Racism, or Prejudice.
  6. Site Moderators have the final word on approving / removing a thread or post or comment.
  7. English language posting only, please.
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out