I run a DNN installation including custom modules. The modules rely on background operation run as scheduled tasks.

Sometimes (less than once per day) the scheduler fails to acquire the internal locks incurring in timeout expiration.

Most of the times the only effect is the logging of the error and nothing else happens. Sometimes (such as the case for which i report the event log here down)  the failure happen while removing a "terminating" task from the queue. In such a case the tasks execution is reported as successfully completed in task log (with a duration time of slightly longer of the 45 seconds of the timeout to acquire the lock), but the tusk is not removed by the queue. Looking at the tasks plan the task results running with an improper start date (1900....). From that point on, the task remains stuck as is not run any more. The only way to recover is to restart the application.

 Here there is an example of the events logged (these events have been logged one after the other at the same time):

AssemblyVersion: 5.4.4
PortalID: -1
PortalName: 
UserID: -1
UserName: 
ActiveTabID: -1
ActiveTabName: 
RawURL: 
AbsoluteURL: 
AbsoluteURLReferrer: 
UserAgent: 
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke.SqlDataProvider
ExceptionGUID: 00765487-c621-4e67-bbe5-8fc4b85fdfd8
InnerException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4)
FileName: 
FileLineNumber: 0
FileColumnNumber: 0
Method: System.Threading.ReaderWriterLock.AcquireWriterLockInternal
StackTrace: 
Message: System.ApplicationException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4) at System.Threading.ReaderWriterLock.AcquireWriterLockInternal(Int32 millisecondsTimeout) at DotNetNuke.Services.Scheduling.DNNScheduling.Scheduler.CoreScheduler.RemoveFromScheduleInProgress(ScheduleItem objScheduleItem)
Source: 


AssemblyVersion: 5.4.4
PortalID: -1
PortalName: 
UserID: -1
UserName: 
ActiveTabID: -1
ActiveTabName: 
RawURL: 
AbsoluteURL: 
AbsoluteURLReferrer: 
UserAgent: 
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke.SqlDataProvider
ExceptionGUID: 8efe0d06-cc81-4347-8c4c-7e40017dad90
InnerException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4)
FileName: 
FileLineNumber: 0
FileColumnNumber: 0
Method: System.Threading.ReaderWriterLock.AcquireReaderLockInternal
StackTrace: 
Message: System.ApplicationException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4) at System.Threading.ReaderWriterLock.AcquireReaderLockInternal(Int32 millisecondsTimeout) at DotNetNuke.Services.Scheduling.DNNScheduling.Scheduler.CoreScheduler.get_ScheduleInProgress()
Source: 

AssemblyVersion: 5.4.4
PortalID: -1
PortalName: 
UserID: -1
UserName: 
ActiveTabID: -1
ActiveTabName: 
RawURL: 
AbsoluteURL: 
AbsoluteURLReferrer: 
UserAgent: 
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke.SqlDataProvider
ExceptionGUID: b9ad4a56-6146-427e-a742-bebb317e1604
InnerException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4)
FileName: 
FileLineNumber: 0
FileColumnNumber: 0
Method: System.Threading.ReaderWriterLock.AcquireWriterLockInternal
StackTrace: 
Message: System.ApplicationException: This operation returned because the timeout period expired. (Exception from HRESULT: 0x800705B4) at System.Threading.ReaderWriterLock.AcquireWriterLockInternal(Int32 millisecondsTimeout) at DotNetNuke.Services.Scheduling.DNNScheduling.Scheduler.CoreScheduler.AddToScheduleQueue(ScheduleHistoryItem objScheduleHistoryItem)
Source: 

The tasks scheduler is configured to run on timer and the site is kept alive by an external polling process, so the scheduler is maintained operational all the time.

The custom tasks sometimes last long (they repeatedly check for actions to perform and if, any do their work) and spawn parallel threads, but it seams that the problem does not relate with the tasks duration or multi threading, it never occurred while running these long tasks.

I checked the size of the scheduler log table and it is not big (about 7000 records), DNN runs on a dedicated multicore server, the database and server load is always low, apart when the application tasks are run.

I didn't found evidence of problems in the DB logs.

Apparently there is some situation in witch the scheduling providers ends in a deadlock.

The problem started after upgrading DNN to 5.4.4 from 5.1.4. and is independent on the net version (experienced both on 3.5 and 4.0).

Any suggestion?