We have developed an application using DNN 7.3.3 community edition and gone for live. We have hosted on a Windows Server 2012 R2 having 4cores 4GB RAM. The memory is growing to 100% in peak time (max of 60 to 80 users accessing the site). Can we know the reason why it is consuming 100% memory (RAM) even for only 80 users? No other applications running on this system and database is on other server and the configuration is 8GB ram.

I had an identical problem two years back (on a dedicated Win2012 and SQL 2012 Web),  tried every possible scenario.

But the best approach would be to  conduct  a thorough analysis/examination of the dump file by creating a dumb file of the IIS Worker Process (w3wp.exe) (see below)

Sent it to an experienced  System Administrator, who will be to locate the culprit method that is causing the infinite loop  and raking up the RAM!

Mine was a customized third party module (by an experienced developer!), that was never tried and and tested before. I was then able to go  to the  vendor who customized the module and had it corrected in less than an hour.

From my experience, infinite loops rarely (if any) occur with core  or tried and tested third party modules.