You can try to use method called ValidateUser in DotNetNuke.Entities.Users.UserController

The return value is an UserInfo object that you could use to determine whether or not the username and password is valid, or you could also use the last ref parameter of the method that could give additional information such as when the user required to change their password, etc.

However, I'm not quite sure whether you could use it directly from your windows application by referencing it directly, because I think it has something to do with the data source and other provider initialization. Anyone could confirm this?

But what you could do is expose a web service from your DNN installation which expose operations that you wish, and consume it from your windows application. You might want to take a look at IWeb module developed by Michael, and some tutorials about IWeb.