Products

Solutions

Resources

Partners

Community

Blog

About

QA

Ideas Test

New Community Website

Ordinarily, you'd be at the right spot, but we've recently launched a brand new community website... For the community, by the community.

Yay... Take Me to the Community!

Welcome to the DNN Community Forums, your preferred source of online community support for all things related to DNN.
In order to participate you must be a registered DNNizen

HomeHomeUsing DNN Platf...Using DNN Platf...Administration ...Administration ...Thousands of System out of Memory Errors on Web FarmThousands of System out of Memory Errors on Web Farm
Previous
 
Next
New Post
9/16/2008 11:46 AM
 

I have a single Portal on a web farm that will have nearly 4000 browsers in the next few weeks. Each day I have 10,000 or more errors in my Event Log that state:


DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke.SqlDataProvider
ExceptionGUID: ee7ba3ba-e18d-4c7d-a68e-e12c2523a027
InnerException: Exception of type 'System.OutOfMemoryException' was thrown.
FileName:
FileLineNumber: 0
FileColumnNumber: 0
Method: System.Threading.Thread.StartInternal
StackTrace:
Message: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)at System.Threading.Thread.Start()at DotNetNuke.Common.Initialize.RunSchedule(HttpRequest request) in \\ServerName\SiteName\Library\Components\Shared\Initialize.vb:line 319


I get so many of these that  I have to truncate the table each day in order to view the Event Viewer. Otherwise, I see an error page or it simply doesn’t load. When I take a look at the servers I see no major utilization of memory. The error does point to Initialize.vb line 319 which is: RequestScheduleThread.Start()

 

Let me give some background on the application that could possibly lead me to the reason and hopefully a solution.

 

I have a 2-server web farm where I host my DNN site. I am using DFS for ensuring that all documents and files are identical on both servers. (So the web.configs are the same) I used the Web Farm documentation to set it up. I, however, was unable to properly complete 2 items in this document:

 

Step 3:  Next, the machine keys in web.config must match across all web servers in the

farm. Open web.config on one machine and find the following XML node:

“configuration/appSettings”. There are two application settings defined here for

the machine keys:

MachineValidationKey – ensure all servers in the farm have the same value

for this key

MachineDecryptionKey – ensure all servers in the farm have the same value

for this key

 

I did not do anything to the servers in the web farm as the Engineers at my company were hesitant to change anything. I indeed found this requirement when researching using Web Farms with ASP.NET applications, but all we did was make sure the web.configs read the same.  

 

Step 5: Alternative Configuration (using file replication services)

 “$root\Providers\CachingProviders\BroadcastPollingCachingProvider\Providers\

SQLDataProvider\ 03.01.00.SqlDataProvider”

 Log in to your DotNetNuke installation as a SuperUser, go to the Host menu and

select “SQL”.

 Paste the contents of your clipboard into the textbox on the SQL page.

 Select the “Run as Script” checkbox so it is checked.

 

This file did not exist in the location listed. I did find a 03.01.00.SqlDataProvider file in another location (Website\Providers\DataProviders\SqlDataProvider). I attempted to run the SQL on the site, but to no avail. I get the longest SQL Error known to man. (This is after changing the {dbo} and {objectqualifier})

 

Also, the only other error message I see is DNN not able to access a cache file because it’s being “utilized by another process”

 

Does anyone have any insight for me? I desperately need some help to halt this slew of error messages. Yesterday I truncated the table and this morning I already had 18,530.

 

Thanks in advance!!

 

 

 

 
New Post
9/17/2008 10:18 AM
 

Does anyone have any solutions?  I am seeing the same kind of thing on my intranet site...  Any help would be appreciated!!!

 
New Post
9/17/2008 10:56 AM
 

Hi everyone,

I have very little experience in operating DNN in a webfarm environment, but since no one has stepped in I thought I'd get you both started.

Couple things to think about:

1. Brian, is your intranet site also running under a webfarm setup?  It would be nice to remove that from the equation, as I suspect it is unlikely to be the source of the issue.

2. Are you running a 32- or 64-bit OS?  Do either of you have more than 2gb of RAM?

3. Were there any changes (upgrades or otherwise) that might have caused this problem to manifest?

4. Perhaps obviously, have you checked your available RAM to ensure that you aren't in fact low on resources?  Have you played with perfmon to evaluate resource usage?  How many active threads are being spawned in the appDomain?

5. Are you running under a custom identify or AppPool?  Is the AppPool configured in any way that might cause this issue (automatic recycle after x megs of actual/virtual memory are allocated, etc)?

Note that this is likely to be a system and/or .NET issue and therefore not strictly DNN-related, and these sorts of issues can be very difficult to lock down. 

Hope this gets you started in your search for a solution!

Brandon

 


Brandon Haynes
BrandonHaynes.org
 
New Post
9/17/2008 12:14 PM
 

Thanks for the response, Brandon!!

2. Each server in the web farm has 4GB of RAM. 32-bit OS.
3. I have been using the same version since I placed it in production - 4.8.4. I am about to upgrade to 4.9 on the dev site, so hopefully I'll move on to the new version in prod soon.
4. I have checked the RAM on each server and I have not seen a spike in utlization for RAM. Each server seems to have about 1000 threads and 50 processes. I don't have access to perfmon at the moment (I'll go work with an Engineer to look at this.) 
5. I do have a specific App Pool for my app:

  • Recycle Worker Processes (in minutes) : 1740       
  • Request queue limit : limit the kernel request queue (number of requests) : 1000
  • Enable pinging: Ping Worker Process every 30 seconds

    I was thinking about adding a few Recycling rules, but I'm not sure if it will help:

  • Recycle worker process after 35000 requests
  • Recycle worker process after consuming too much memory: 500 MB (too little, too much?)

I haven't experienced this issue with a similar set up with any other DNN sites. The only discernable difference is the web farm...

 
New Post
9/23/2008 10:32 AM
 

As I mentioned above I am getting an issue with a cache file. I am seeing this error now very frequently rather than the System out of memory. What's curious is that the path listed leads to ServerName1 but the server listed at the end is ServerName2. Does that mean potential issues with the web farm set up?

System.IO.IOException: The process cannot access the file '\Website\Portals\_default\Cache\412517EFE3EAAF5BD4873D1097CA25CC.resources' because it is being used by another process.at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)at System.IO.StreamWriter.CreateFile(String path, Boolean append)at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize)at System.IO.StreamWriter..ctor(String path, Boolean append)at System.IO.File.CreateText(String path)at DotNetNuke.Services.Cache.FileBasedCachingProvider.FBCachingProvider.CreateCacheFile(String FileName) in \\ServerName1\Library\Providers\CachingProviders\FileBasedCachingProvider\FileBasedCachingProvider.vb:line 287at DotNetNuke.Services.Cache.FileBasedCachingProvider.FBCachingProvider.Insert(String CacheKey, Object objObject, CacheDependency objDependency, DateTime AbsoluteExpiration, TimeSpan SlidingExpiration, Boolean PersistAppRestart) in \\ServerName1\Library\Providers\CachingProviders\FileBasedCachingProvider\FileBasedCachingProvider.vb:line 176at DotNetNuke.Common.Utilities.DataCache.SetCache(String CacheKey, Object objObject, TimeSpan SlidingExpiration, Boolean PersistAppRestart) in \\ServerName1\Library\Components\Providers\Caching\DataCache.vb:line 246at DotNetNuke.Security.Permissions.TabPermissionController.GetTabPermissionsDictionary(Int32 PortalId) in \\ServerName1\Library\Components\Security\Permissions\TabPermissionController.vb:line 181at DotNetNuke.Security.Permissions.TabPermissionController.GetTabPermissionsCollectionByTabID(Int32 TabID, Int32 PortalId) in \\ServerName1\Library\Components\Security\Permissions\TabPermissionController.vb:line 233at DotNetNuke.Entities.Tabs.TabController.FillTabInfo(IDataReader dr, Boolean CheckForOpenDataReader, Boolean CheckForLegacyFields) in \\ServerName1\Library\Components\Tabs\TabController.vb:line 108at DotNetNuke.Entities.Tabs.TabController.FillTabInfoDictionary(IDataReader dr) in \\ServerName1\Library\Components\Tabs\TabController.vb:line 173
Source:
Server Name: ServerName2

 
Previous
 
Next
HomeHomeUsing DNN Platf...Using DNN Platf...Administration ...Administration ...Thousands of System out of Memory Errors on Web FarmThousands of System out of Memory Errors on Web Farm


These Forums are dedicated to discussion of DNN Platform and Evoq Solutions.

For the benefit of the community and to protect the integrity of the ecosystem, please observe the following posting guidelines:

  1. No Advertising. This includes promotion of commercial and non-commercial products or services which are not directly related to DNN.
  2. No vendor trolling / poaching. If someone posts about a vendor issue, allow the vendor or other customers to respond. Any post that looks like trolling / poaching will be removed.
  3. Discussion or promotion of DNN Platform product releases under a different brand name are strictly prohibited.
  4. No Flaming or Trolling.
  5. No Profanity, Racism, or Prejudice.
  6. Site Moderators have the final word on approving / removing a thread or post or comment.
  7. English language posting only, please.
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out
What is Liquid Content?
Find Out