(Cross post from my Google+ profile)

If you haven't yet, install http://www.iis.net/download/URLRewrite. If you don't have access to the server, ask your host to install.  (Honestly, it should be a default on all Windows Servers.)

Once installed, add the following to the <system.webServer> section of the web.config file...

<rewrite>
<rules>
 <rule name="robots">
<!-- Matches the requests for /robots.txt -->
   <match url="^robots\.txt" />
   <conditions>
<!-- Matches all domains. -->
    <add input="{HTTP_HOST}" pattern="(.*)" />
   </conditions>
<!-- Rewrites to the domain specific file. -->
   <action type="Rewrite" url="robots.{C:0}.txt" appendQueryString="false" />
  </rule>
 </rules>
</rewrite>

The rewrite engine looks for requests to the /robot.txt then grabs the domain from the condition to change the request to the pattern: robots.www.mydomain.com.txt

I chose this pattern as it groups the files together in the file browser.  As this is a rewrite rule, the search engine spiders don’t know the different.  As far as they know, they get properly served, individual files for each domain.

If you don’t want to create a file for each portal, then change the condition to:

<add input="{HTTP_HOST}" pattern="^([a-zA-Z0-9]*\.)?(domain1\.org|domain2\.com)" />

Now you can post different files for:
Robots.txt  generic or “not-matched”
Robots.domain1.com.txt
Robots.dev.domain1.com.txt
Robots.www.domain2.org.txt

http://www.ifinity.com.au/2012/03/21/Different_robots_files_for_different_DotNetNuke_Portals

Compliant spiders like Googlebot or Bingbot start indexing a site by going to /robots.txt first.  The file indicates where they can and can't go on the site, if at all.

That means they will only request the sitemap et al after they parse this file and site access is established.

I have an amendment too.  There are a couple ways to do this: 1) a global rule that requires a robots.txt file for every domain or 2) specific domain files and a fallback for others.  In either case, you have to create specific content or rule for each domain.

Global:

• Match: < add input="{HTTP_HOST}" pattern="(.*)" / > 
• Will match all domains and always request a special robots file

Specific domains and fallback:

• Match: < add input="{HTTP_HOST}" pattern="([a-zA-Z0-9]\.)(domain1\.com|domain2\.org)" / >
• Will only match domain1.com, domain2.com and their sub-domains. The rest will get the normal  /robot.txt

Both write to:

< action type="Rewrite" url="robots.{C:0}.txt" appendQueryString="false" / >

{C:0} will contain the whole string of the condition.