Its unfortunate that this thread has gone cold. I have experienced similiar problems with DNN 4.5.3 on a dual core 4GB server which I reported about a week ago. The web process was assigned 1.5 GB of memory which it still managed to consume under relatively low usage (about 10 users and 500 pages). I do know that there was a lot of editing happening at the time as most of the pages were in the process of being constructed. 

My DNN setup makes use of a modified TextHTML module (simple modifications that I doubt are causing the problems) and no other modules apart from administration modules. I am currently using process recycling ever four hours to limit the problems that "out of memory errors" cause.

I have seen similiar errors reported else where in these forums. I have noticed that most of the reports refer to the retrieval of the navigation nodes (GetNavigationNodes) in the stack trace. This may just be a commonly used method but could at least be a starting point.