Products

Evoq Content Overview Content Creation Workflow Asset Management Mobile Responsive Personalization Content Analytics SEO Integrations Security Website PerformanceEvoq Engage Overview Community Management  Dashboard  Analytics  Member Profile Gamification Advocate Marketing Community Engagement  Ideas  Answers  Discussions  Groups  Wikis  Events Mobile ReadyDNN Support PackagesEvoq: CMS FeaturesEvoq OnDemandCompare DNN's CMS Products

Solutions

Content Management SystemMulti-Channel PublishingDigital MarketingOnline Community ManagementIntranet Software SolutionOur CustomersPrime

Resources

Test DrivesSchedule A DemoDocumentation CenterWebinarsWhite PapersProduct ManualsRequest PricingData SheetsCase StudiesEvoq Preferred Products Content Management Ecommerce Forms Identity Management and Authentication Site Management Social Themes

Partners

DNN Partners Partner Directory Become a DNN Partner Partner Program Overview  Hosting  Implementation  ISV  Training  Competencies

Community

Participate Ask a Question Forums Community Showcase DNN MVP  Program Overview  Lifetime MVPs  Honorary MVPs Subscribe to DNN Digest Advisory Groups  Awareness Advisory Group  Developer Advisory Group  Partner Advisory Group  Technology Advisory GroupLearn Documentation Wiki Community Blog Video Library Project History Development RoadmapDownload DNN Store Language Packs Manuals Nightly BuildsSecurity Policies Security Center

Blog

About

News Press Releases News Coverage Press KitCareersContact DNN

QA

Ideas Test

New Community Website

Ordinarily, you'd be at the right spot, but we've recently launched a brand new community website... For the community, by the community.

Yay... Take Me to the Community!

Welcome to the DNN Community Forums, your preferred source of online community support for all things related to DNN.
In order to participate you must be a registered DNNizen
HomeHomeOur CommunityOur CommunityGeneral Discuss...General Discuss...bots ignore robots.txt if META TAGS say OKbots ignore robots.txt if META TAGS say OK
Previous
 
Next
New Post
12/12/2008 9:36 AM
 
lou brown

Joined: 6/10/2004
Posts: 26

I put a "robots.txt" file, while denied all bots, but they kept coming even though I explicitly denied them in META Tags. Its because the default DotNetNuke META tag that's coded into the default.aspx page allows all bots.

 

So the answer to my own question is the bots will keep coming until all avenues deny them access.
 
New Post
12/12/2008 10:52 PM
 
Tom Kraak


Joined: 12/26/2002
Posts: 458

Regardless of what the robots meta tags says, I've never had anything indexed with a robots.txt containing:

User-agent: *
Disallow: /
Tom Kraak
SEO Analyst
R2integrated
 
New Post
12/13/2008 5:57 AM
 
Donald Hurst

www.internetpengar.com
Joined: 12/13/2008
Posts: 2

All bots should follow the robots.txt. Otherwise you can easily ban specific bots in your .htaccess file.
 
New Post
12/14/2008 8:12 AM
 
Erik van Ballegoij


Erik van Ballegoij's Avatar

erikvanballegoij.com
Joined: 4/7/2004
Posts: 4445
since DNN only runs on windows server / IIS, .htaccess is not available... (just in case someone wonders why .htaccess does not work...) You could use the DNN request filter though to block specific ip addresses
Erik van Ballegoij, Former DNN Corp. Employee and DNN Expert

DNN Blog | Twitter: @erikvb | LinkedIn: Erik van Ballegoij on LinkedIn
 
New Post
12/14/2008 5:33 PM
 
Tom Kraak


Joined: 12/26/2002
Posts: 458

If you are lucky enough to run an IIS extension such as isapi_rewrite, you will have a .htaccess file.

Tom Kraak
SEO Analyst
R2integrated
 
 Page 1 of 1
Previous
 
Next
HomeHomeOur CommunityOur CommunityGeneral Discuss...General Discuss...bots ignore robots.txt if META TAGS say OKbots ignore robots.txt if META TAGS say OK


These Forums are dedicated to discussion of DNN Platform and Evoq Solutions.

For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

  1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
  2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
  3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
  4. No Flaming or Trolling.
  5. No Profanity, Racism, or Prejudice.
  6. Site Moderators have the final word on approving / removing a thread or post or comment.
  7. English language posting only, please.
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out