Hi Folks,
I hope all is well.
I'd like to share what I know about search engine optimization and see if I'm on the right track and if other people have some ideas.
Basically I do these things:
- Add a browserCaps section to my web.config file so that known spiders will be recognized as such. (listed below)
- Add a robots.txt file to the root of my dnn install (listed below)
- Provide Keywords and Descriptions and "Submit to Google" the pages I want indexed
- Use Friendly URLs
Also, I can check my site to make sure the SolPart menu it properly displaying as a site map if I set my FireFox browser to spoof as a spider.
So I'm wondering...
- Do I really need to provide the browserCaps section in my web.config?
- Other than the steps I'm taking, do other folks have any recommendations?
Thanks,
Wylie
Here’s how to set you FireFox browser to spoof a GoogleBot.
*************************************************************
- Open a FireFox browser and enter about:config as a URL address.
- You will see all the configuration settings for your browser
- Now click the right mouse button to get the context menu
- Select "String" > "New".
- Enter the preference name "general.useragent.override", without the quotes.
- Then enter the string value “Googlebot/2.1 (+http://www.googlebot.com/bot.html)”
When you are finished browsing the web as a googlebot, you may re-open the config screen, find the general.useragent.override preference, right click on it and “reset”.
BrowserCaps - place within <system.web> tags in your web.config
*********************************************************
<browserCaps>
<filter>
<!-- SEARCH ENGINES GROUP -->
<!-- check Google (Yahoo uses this as well) -->
<case match="^Googlebot(\-Image)?/(?'version'(?'major'\d+)(?'minor'\.\d+)).*">
browser=Google
version=${version}
majorversion=${major}
minorversion=${minor}
crawler=true
</case>
<!-- check Alta Vista (Scooter) -->
<case match="^Scooter(/|-)(?'version'(?'major'\d+)(?'minor'\.\d+)).*">
browser=AltaVista
version=${version}
majorversion=${major}
minorversion=${minor}
crawler=true
</case>
<!-- check Alta Vista (Mercator) -->
<case match="Mercator">
browser=AltaVista
crawler=true
</case>
<!-- check Slurp (Yahoo uses this as well) -->
<case match="Slurp">
browser=Slurp
crawler=true
</case>
<!-- check MSN -->
<case match="MSNBOT">
browser=MSN
crawler=true
</case>
<!-- check Northern Light -->
<case match="^Gulliver/(?'version'(?'major'\d+)(?'minor'\.\d+)).*">
browser=NorthernLight
version=${version}
majorversion=${major}
minorversion=${minor}
crawler=true
</case>
<!-- check Excite -->
<case match="ArchitextSpider">
browser=Excite
crawler=true
</case>
<!-- Lycos -->
<case match="Lycos_Spider">
browser=Lycos
crawler=true
</case>
<!-- Ask Jeeves -->
<case match="Ask Jeeves">
browser=AskJeaves
crawler=true
</case>
<!-- check Fast -->
<case match="^FAST-WebCrawler/(?'version'(?'major'\d+)(?'minor'\.\d+)).*">
browser=Fast
version=${version}
majorversion=${major}
minorversion=${minor}
crawler=true
</case>
<!-- IBM Research Web Crawler -->
<case match="http\:\/\/www\.almaden.ibm.com\/cs\/crawler">
browser=IBMResearchWebCrawler
crawler=true
</case>
</filter>
</browserCaps>
Robots.txt
*********************************************
User-agent: *
Disallow: /admin/
Disallow: /app_globalresources/
Disallow: /bin/
Disallow: /Components/
Disallow: /config/
Disallow: /controls/
Disallow: /DesktopModules/
Disallow: /images/
Disallow: /install/
Disallow: /js/
Disallow: /Portals/
Disallow: /Providers/