Hello,

I have multiple parent portals with different domains.

My strategy is to add only Allow statements into the robots.txt so that only useful URLs will be crawled. But due to the fact that the robots.txt has relative URLs, it might by confusing for Google Bot. Here is an example:

• Portal0 - www.domain0.com            (here I have this url: www.domain0.com/Product000.aspx)
• Portal1 - www.domain1.com (here I have this url: www.domain1.com/Product111.aspx)

My robots txt will look like this:

Allow: /Product000.aspx
Allow: /Product111.aspx

The problem is that the second line will cause an error  when the Google bot crawls the Portal0 as that line belongs to the Portal1 and does not exist under www.domain0.com.

Is this a problem at all? Can I do like this or do you have any experiences on similar situations?

Thanks