Your website performance depends on:

1. Most important - hosting provider. According to my experience, your DNN application MUST always have at least 356MB RAM to perform on.
2. Remove un-neccesary services on your server.
3. Use a Keep Alive service to make sure your application doesn't sleep too often.
4. Configure DNN's performance:
   1. Use GZip (core compression). I don't use Page Blaster because it affects dynamic modules.
   2. Set to Heavy Caching, use Memory caching for Module & Disk caching for Page State Persistence
   3. Set server cacheability to "Public"
   4. Set scheduler method --> Timer
   5. Shutdown un-necessary  schedule tasks.
   6. Use a light-weight skin

Ok, these are just my personal experience. I'm not completely sure that it'll work for you.

HTH