Well I figured it out.  Apparantly there's an HTTP module or something in DNN that once you're on a module url, it goes to that page no matter what the page on the ending page is.  For ex:

http://localhost:4312/DotNetNukeCustom/Products/tabid/56/ctl/Module/ModuleId/385/Default.aspx

You can do: http://localhost:4312/DotNetNukeCustom/Products/tabid/56/ctl/Module/ModuleId/385/AnyValueHere and it will still go to the same page.

So basically I just had to get the relative path to the virtual root and make a call to it and it works perfectly now (I set a JS variable with the path):

var ajaxCallbackURL = "'" + VirtualPathUtility.ToAbsolute("~/") +"ServerFolderBrowser.aspx'"; //returns /DotNetNukeCustom/ServerFolderBrowser.aspx

I have the following http handler setup in the web.config:

<add verb="*" path="ServerFolderBrowser.aspx" validate="false" type="SoutheastSolutions.Web.ServerFolderBrowserHTTPHandler"/>