The extension are mapped to a handler, .htm is handled by ISAPI, .asp by asp.dll and .aspx by aspnet_isapi.dll

Your real problems are caused by the way IIS 5.x handles web.config files. IIS uses the root web.config then the child web.config which is not what you might expect in a virtual directory. Since the processing is additive settings may confict with each other. Considering the following...

(0) .NET Framework      c:\...\machine.config      
(1) Root \                    c:\inetpub\wwwroot\web.config      
(2) Virtual Alias \site1\   c:\inetpub\wwwroot\site1\web.config
(3) Virtual Alias \abc\     c:\dotnetnuke\web.config

http://localhost/ uses (0) then (1)
http://localhost/site1 uses (0) then (1) then (2) 
http://localhost/abc uses (0) then (1) then (3)

If you have dotnetnuke in the root (1) then any other web applications in virtual sites will include your dotnetnuke web.config which is probably not what you intended.

The easiest solution with IIS 5 is to leave the root empty then all of your virtual directories will use the web.config as intended. The side effect is that your different sites need a different suffix (which may be an advantage). You can handle many different ASP.NET applications (even multiple DotNetNuke) this way and they won't interfere with each other.

A more difficult resolution is to add a <clear /> to every section that conflicts with the parent web.config.

With Windows 2003 (IIS 6) you can simply host your virtual folder under a new application pool.