I recently created a dot net nule site for a friend, we created a basic site, applied a skin , everything seemed fine. We even applied a google sitemap generator that we could submit to google. When google spidered the site it seemed to get the first few links correct then it seemed to focus on the login and privacy links that appear off our main content pages ... it did this many times ... without actually finishing spidering all the actual main content pages themselves. We also spent quite a lot of time creating relevant meta tag descriptions and keywords for each content page.

What I would like to know, is there any way of controlling the meta tag directives on a per page basis that would allow me to discourage a spider from following that page ... ie the robots/follow/nofollow directive ... in particular I would want the spider to be discouraged from spidering all the various login and privacy "virtual" pages that are created from each main page I actually create ... I dont want to make these pages hidden to normal browsing.

Any help appreciated :)

Hi Sam,

I was thinking of doing this, but I was unsure if I had to create an entry for each page I didnt want the spider to visit , thus making a very manual process, or if there was some kind of wildcard entry I could use to catch every occurance of a privacy / login version of a page ? How do you do this ?

rgds,
Cameron.