This was tricky. You can add <location path="." inheritInChildApplications="false"> around the system.webServer and system.web so basically:

<location path="." inheritInChildApplications="false">
  <system.web>
            <compilation debug="true" />
            <customErrors mode="Off" />
            <httpRuntime maxRequestLength="32768" />
            <httpModules>
                <clear />
            </httpModules>
        </system.web>

        <system.webServer>
            <modules>
                <remove name="UrlRewrite" />
            </modules>
        </system.webServer>
</location>

Then convert the imagemanager to an application if you use IIS 7. The problem is that it wont share session data between .NET nuke and the imagemanager application. So a double login would be needed. So this is far from ideal. The problem is that dotnet nuke adds lots of httphandlers etc that can't be disabled on a directory level a.f.a.i.k.