Hi Paul,
Yes this is possible. DNN isn't really doing anything outside the scope of any other .NET Web Application in existence, it just takes advantage of a lot of its features. So in order to use the membership provider within DNN I would do the following
1) Let DNN install a database as it normally would by actually following all the steps to setting up a DNN site
2) Take most of your web.config (I believe this is release.config or debug.config in the source download) and copy all the <sections> relating to dotnetnuke as well as the actual <dotnetnuke> section and all its contents in this file. Move this information to your new web.config. Also, make sure you copy the machinevalidationkey you created your database with to your new web.config.
3) copy the core DLL files (this will be the DotNetNuke.dll, plus you'll need some of the referenced files like WebUtility I think and some of the sql data providers (dotnetnuke.sqldataprovider.dll I think as well as all the memberhsip, profile, caching etc providers... these won't affect your current website, just communication when using the DNN framework.). Some trial and error will get this perfect.
4) reference the DotNetNuke dll in your new web app. call membership/user functions from there, should be working. I'm stating alot of this blindly but I've done it before and had it work. It all really comes down to making sure you have your config file correct, DNN heavily relies on it (which makes swapping certain parts out *really* easy)
Here is a post from ASP.NET where I used it in a unit testing scenario. I seem to remember everything working... but this was back in early 2005...scary to think that was 3 years ago...
http://forums.asp.net/p/799690/848227.aspx#848227