It's actually not a DotNetNuke question, but a matter of email server and DNS server configuration.  There are quite a few factors involved, and of course the specific procedures depend on the platforms you are using for DNS and for email.  I like what Graham Orndorff wrote: 

"When I was recently asked "How do I configure my mail server?" I replied, "That is sort of like asking me 'How do I write my thesis?'"

This quote is from his article here. 

So, setting up a mail server, and the associated DNS records such as mx, and other relevant topics such as avoiding blacklists --- there's so much, it's really quite outside the scope of the DotNetNuke forums.  I'd suggest you look into what platform you're using and then search for documentation related to configuration of that particular platform.