Mike,
The DotNetNuke solution is a collection of projects that make up the whole. Specifically there is the main one you are trying to run (DotNetNuke.Library) as well as supportive projects (HttpModules, Modules, Providers).
The project that fails to open is the actual website, this is what you'd run if it was working on your machine. By default it is configured to go to IIS (which, you don't have) and it looks for 'http://localhost/DotNetNuke_Community' which is the binding you'd need configured in IIS. There is good news though...
Factoid: To know the application type, right-click on 'DotNetNuke.Library' and choose properties. You'll see that the 'Application type' says 'Class Library' and under 'Compile' you can see that it outputs to the bin\ folder. This would only matter to you if you were modifying the core DotNetNuke project. If you want to do custom module development you actually never need to update the core DotNetNuke projects.
What you need to do, is tell Visual Studio that you haven't got IIS and want it to use Cassini. Also, to avoid the error about trying to directly starting a class you'll need to change the default startup project. Here are some tips:
1. Right-click on the 'http://localhost/DotNetNuke_Community' website project and go to 'properties'. You'll notice under 'Start Options' Visual Studio wants to know what web server to run this under. If it is set to 'Use default Web server' and 'ASP.NET' for debuggers then you're off to a good start.
2. For sake of getting the connection between the 'DotNetNuke.Library' project and this website, I highly recommend browsing the 'References' section to see how all the classes are used in this site.
3. Get out of the properties menu and right-click on the 'http://localhost/DotNetNuke_Community' website. Choose 'Set as StartUp Project'. This will make your hitting the 'play' button or 'Debug > Start Debugging' launch THIS project and not the class library project. To confirm you did this correctly try hitting F5 and you should be prompted to 'modify the web.config to have debug=true'. Say yes to this and you're good to go. (This is how Cassini is forced, you can't set the port they are randomly assigned and change very frequently.)
Word of advice though, get an OS with IIS. Ideally Vista or the Windows 7 professional editions. Having unlimited websites you can configure allows you to have many different instances of DotNetNuke going and be able to quickly attach to processes for debugging. I find it faster (no stop/start of the project), more robust (see: many instances) and let's not forget the "look how cool I am, I can attach to the IIS worker process" factor. ;)
Cheers, Dylan