A dotnetnuke portal built for a client is crashing several times a day. When it crashes the application process reloads, which causes an outage of several minutes whilst the worker processes restart. This is often (but not always) accompanied by a prolonged period of 100% cpu usage on the relevant webserver.
I'm getting to my wits end with this, so any suggestions would be gratefully received. The site is a single portal built for a client as an internal Intranet. The maximum number of users would be about 800 or so, but we've had crashes with as few as four users on the site.
The site was originally built in a development environment on a Windows 2003 R2 server with SQL 2005 Personal Edition. The site was then moved to the production environment - two dedicated Quad Core Xeon machines each running Windows 2003 R2 with the data files stored on a DFS drive (Windows 2003 Storage Edition I believe) and the database on a separate dedicated db machine running SQL 2005 Enterprise. Machines have 4gb ram and DotNetNuke is the only application running on the machines. IIS is set to run the app in a dedicated application pool. After problems with caching and sessions we've turned off web gardening.
Originally built on DotNetNuke 4.3.4, we upgraded to 4.5.3 early in the development. We've stayed there as recent changes to the authentication provider made it difficult to upgrade to the 4.6 or 4.7 trees until recently. We are now considering upgrading again if it will help with our problems. I've had to slightly customise the main dotnetnuke code (mainly in the area of Active Directory authentication) and have customised the Events and Documents modules. We're also running some third party modules - notably NukeNews, ScrollControllerPro, bring2mind RSS and SimpleGallery. We've written two modules ourselves - one to display a list of the top 10 most popular documents and one to query an LDAP server to produce an online address book.
We never had a crash on the development environment - we first started getting crashes on the live environment after we'd put the site live and the first batch of people (about 50 individuals) were using the site. Since then we've never been able to get it stable. I've tried running the site in a single server environment, serving the content from the local disk instead of the DFS share, turning off all Active Directory authentication and precompiling the content pages, all with no success.
The logs (both dnn and the system event log) don't show any errors - they just show the usual log messages when the application process restarts. I wouldn't mind the application process restarting if it didn't take so long - during the 2 minutes or so that it takes for the app to restart users do notice the outage.
It's not the usual IIS application idle timeout or any other inactivity setting as far as we're aware - I've changed all timeouts to 24 hours. At one point we even set the worker processes to recycle automatically at 2am every morning to clean out any stale references or memory issues - again with no joy.
I know other people are running in similar environments, so any ideas? Also, if anyone out there lives in London (UK), and is familiar with dotnetnuke (particularly if your expertise also covers serving dnn sites off DFS shares or using a web farm) then we may be willing to hire someone in to assist us with this problem, as the client is starting to get a bit irritated (we've been having these problems for two months or so and we're on the verge of abandoning DotNetNuke and putting it on our 'never use this software again' list).
Andrew