Hi All, am having a bit of trouble with this and hope to get some help. I have a robots.txt in the root of my domain, but Google says it can't find it - it's tried only once, so I'll be patient and give this problem another few days.

But, I'm trying to get the syntax of it right to disallow crawling to all of the /ctl/ pages, i.e. terms, privacy and login. Have tried the syntax used in DNN's own file at http://www.dotnetnuke.com/robots.txt of Disallow: /*/ctl/ in the Google robots.txt testing/diagnostics thing but it still finds a path to the test /ctl/ url I entered.

Has anyone solved this? My domain, in case it matters, is set up as below:

default.htm - redirects to /prod
/prod - IIS virtual folder with separate DNN install and db
/test  - IIS virtual folder with separate DNN install and db

Thanks and kind regards, JP

Many thanks Carlos. I actually had a typo in the Google testing thing and the /prod/*/ctl/ does actually pass the test (i.e. disallows access).

Yep, my set up is one domain with two sub-dirs that are set up as virtual directories in IIS - each with a completely separate DNN installation and SQL database - seems to work well.

I have included Disallow: /test/ in the robots.txt, so hopefully those pages won't be indexed - have yet to see if this works - although it does pass the google robots.txt test.