The issue is resolved. Luckily, as the db was not impacted, I was able to view in the log files the 'erosion' that occurred that preceded the need to rebuild [inability to log in to any portal].
A summary of those log files and brief discussion is below if anyone is curious, and in case it proves meaningful to someone.
Thanks for the help,
Bob H.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
After months of error-free logs, on 10/24 thru 10/31 there were no less than 300 General Exception/Page Load Exception errors logged. At first they were innocuous to the admin and went unreported and unnoticed. The first 5 in a period of 40-50 seconds were essentially identical but had different GUIDs:
AssemblyVersion: 7.0.6
PortalID: 13
PortalName: "Taste of the Laurel Highlands" - SAJSO
UserID: -1
UserName:
ActiveTabID: 534
ActiveTabName: Venue
RawURL: /asp/tolh/venue.aspx
AbsoluteURL: /asp/Default.aspx
AbsoluteURLReferrer:
UserAgent: Mozilla/4.0 (compatible; Synapse)
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke
ExceptionGUID: 4129898d-5651-4c11-979d-4a1f4fd4323d
InnerException: The state information is invalid for this page and might be corrupted.
FileName:
FileLineNumber: 0
FileColumnNumber: 0
Method: System.Convert.FromBase64_Decode
StackTrace:
Message: DotNetNuke.Services.Exceptions.PageLoadException: The state information is invalid for this page and might be corrupted. ---> System.Web.HttpException: The state information is invalid for this page and might be corrupted. ---> System.Web.UI.ViewStateException: Invalid viewstate. Client IP: 86.57.191.22 Port: 15055 Referer: Path: /asp/Default.aspx User-Agent: Mozilla/4.0 (compatible; Synapse) ViewState: /wepdwujndmwnzq0ndu3d2qwcgypfgiebfrlehqfetwhre9dvflqrsbodg1sifbvqkxjqyails8vvzndly9eveqgwehutuwgms4wifryyw5zaxrpb25hbc8vru4iicjodhrwoi8vd3d3lnczlm9yzy9uui94ahrtbdevrfrel3hodg1sms10cmfuc2l0aw9uywwuzhrkij5kagipfgifaavdihhtbdpsyw5npsjlbi1vuyigbgfuzz0izw4tvvmiihhtbg5zpsjodhrwoi8vd3d3lnczlm9yzy8xotk5l3hodg1simqcba9kfg4cbg8wah4hvmlzawjszwhkagcpfgieb2nvbnrlbnqfsgfuyxn0zsbvzib0agugtgf1cmvsiehpz2hsyw5kcybpcybhigz1bmryywlzzxigzxzlbnqgd2hpy2ggymvuzwzpdhmgdghliepvag5zdg93biwgueegu3ltcghvbnkgt3j2agvzdhjhliagvghlifnvbwvyc2v0ief1eglsawfyesbjzwxlynjhdgvzigl0cya1mhroigfubml2zxjzyxj5igluidiwmtmgyw5kidiwmtquzaiidxychwify0pvag5zdg93bibqzw5uc3lsdmfuawegt3jjagvzdhjhief1eglsawfyesbtb21lcnnldcbqqsb0yxn0zsbvzib0agugtgf1cmvsiehpz2hsyw5kcyxeb3rozxrodwtllerotmqccq8wah8cbrldb3b5cmlnahqgynkgugfkcmugr3jhbmrlzaikdxychwifc0rvde5lde51a2ugzaildxychwifjyjuyxn0zsbvzib0agugtgf1cmvsiehpz2hsyw5kcyiglsbtquptt... ---> System.FormatException: Invalid length for a Base-64 char array or string. at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength) at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength) at System.Convert.FromBase64String(String s) at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString, Purpose purpose) at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter2.Deserialize(String serializedState, Purpose purpose) at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter2 formatter, String serializedState, Purpose purpose) at System.Web.UI.HiddenFieldPageStatePersister.Load() --- End of inner exception stack trace --- --- End of inner exception stack trace --- at System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidati ) at System.Web.UI.HiddenFieldPageStatePersister.Load() at System.Web.UI.Page.LoadPageStateFromPersistenceMedium() at System.Web.UI.Page.LoadAllState() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) --- End of inner exception stack trace ---
Source:
Server Name: P3NW8SHG349
Six hours later, 3 different ones occurred, but they were all 3 the same except GUIDs:
AssemblyVersion: 7.0.6
PortalID: 0
PortalName: Indian Lake Info Network
UserID: -1
UserName:
ActiveTabID: 40
ActiveTabName: Home
RawURL: /asp/Default.aspx?tabid=40&error=An%20unexpected%20error%20has%20occurred
AbsoluteURL: /asp/Default.aspx
AbsoluteURLReferrer: http://www.indianlake-pa.net/asp/Default.aspx?tabid=40&error=An%20unexpected%20error%20has%20occurred
UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke
ExceptionGUID: 8af44269-2235-440e-a8fa-5cc169d7d42f
InnerException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.
FileName:
FileLineNumber: 0
FileColumnNumber: 0
Method:
StackTrace:
Message: DotNetNuke.Services.Exceptions.PageLoadException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.UI.ViewStateException: Invalid viewstate. Client IP: 46.246.56.7 Port: 59911 Referer: http://www.indianlake-pa.net/asp/Default.aspx?tabid=40&error=An%20unexpected%20error%20has%20occurred Path: /asp/Default.aspx User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:12.0) Gecko/20100101 Firefox/12.0 ViewState: /wEPDwUJNDMwNzQ0NDU3D2QWCGYPFgIeBFRleHQFejwhRE9DVFlQRSBodG1sIFBVQkxJQyAiLS8vVzNDLy9EVEQgWEhUTUwgMS4wIFRyYW5zaXRpb25hbC8vRU4iICJodHRwOi8vd3d3LnczLm9yZy9UUi94aHRtbDEvRFREL3hodG1sMS10cmFuc2l0aW9uYWwuZHRkIj4gZAICDxYCHwAFQyB4bWw6bGFuZz0iZW4tVVMiIGxhbmc9ImVuLVVTIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCJkAgQPZBYOAgYPFgIeB1Zpc2libGVoZAIHDxYEHgdjb250ZW50ZB8BaGQCCA8WAh8CBQ8sRG90TmV0TnVrZSxETk5kAgkPFgIfAgUPYnkgUGFkcmUgR3JhbmRlZAIKDxYCHwIFC0RvdE5ldE51a2UgZAILDxYCHwIFGEluZGlhbiBMYWtlIEluZm8gTmV0d29ya2QCDg8WAh8CBQ1JTkRFWCwgRk9MTE9XZAIGD2QWAgIBD2QWAgIHD2QWAmYPZBY0AgQPZBYIZg8WAh8ABR48ZGl2IGNsYXNzPSJsYW5ndWFnZS1vYmplY3QiID5kAgIPEA8WAh8BaGRkFgBkAgQPFgIfAWhkAgYPFgIfAAUGPC9kaXY+ZAIID2QWAmYPDxYEHgdUb29sVGlwBRhJbmRpYW4gTGFrZSBJbmZvIE5ldHdvcmseC0... --- End of inner exception stack trace --- at System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidati ) at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString, Purpose purpose) at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter2.Deserialize(String serializedState, Purpose purpose) at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter2 formatter, String serializedState, Purpose purpose) at System.Web.UI.HiddenFieldPageStatePersister.Load() at System.Web.UI.Page.LoadPageStateFromPersistenceMedium() at System.Web.UI.Page.LoadAllState() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) --- End of inner exception stack trace ---
Source:
Server Name: P3NW8SHG349
The following day, 10/25, three more occurred over the period of 3 seconds:
AssemblyVersion: 7.0.6
PortalID: 0
PortalName: Indian Lake Info Network
UserID: -1
UserName:
ActiveTabID: 40
ActiveTabName: Home
RawURL: /asp/Default.aspx?tabid=40&error=An%20unexpected%20error%20has%20occurred
AbsoluteURL: /asp/Default.aspx
AbsoluteURLReferrer: http://indianlake-pa.net/asp/Default.aspx?tabid=40&error=An%20unexpected%20error%20has%20occurred
UserAgent: Opera/9.80 (Windows NT 6.1; Win64; x64) Presto/2.12.388 Version/12.10
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke
ExceptionGUID: 5d772c4b-70ad-43e5-a36a-a07421d39034
InnerException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.
FileName:
FileLineNumber: 0
FileColumnNumber: 0
Method:
StackTrace:
Message: DotNetNuke.Services.Exceptions.PageLoadException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.UI.ViewStateException: Invalid viewstate. Client IP: 128.68.240.156 Port: 50997 Referer: http://indianlake-pa.net/asp/Default.aspx?tabid=40&error=An%20unexpected%20error%20has%20occurred Path: /asp/Default.aspx User-Agent: Opera/9.80 (Windows NT 6.1; Win64; x64) Presto/2.12.388 Version/12.10 ViewState: /wEPDwUJNDMwNzQ0NDU3D2QWCGYPFgIeBFRleHQFejwhRE9DVFlQRSBodG1sIFBVQkxJQyAiLS8vVzNDLy9EVEQgWEhUTUwgMS4wIFRyYW5zaXRpb25hbC8vRU4iICJodHRwOi8vd3d3LnczLm9yZy9UUi94aHRtbDEvRFREL3hodG1sMS10cmFuc2l0aW9uYWwuZHRkIj4gZAICDxYCHwAFQyB4bWw6bGFuZz0iZW4tVVMiIGxhbmc9ImVuLVVTIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCJkAgQPZBYOAgYPFgIeB1Zpc2libGVoZAIHDxYEHgdjb250ZW50ZB8BaGQCCA8WAh8CBQ8sRG90TmV0TnVrZSxETk5kAgkPFgIfAgUPYnkgUGFkcmUgR3JhbmRlZAIKDxYCHwIFC0RvdE5ldE51a2UgZAILDxYCHwIFGEluZGlhbiBMYWtlIEluZm8gTmV0d29ya2QCDg8WAh8CBQ1JTkRFWCwgRk9MTE9XZAIGD2QWAgIBD2QWAgIHD2QWAmYPZBY0AgQPZBYIZg8WAh8ABR48ZGl2IGNsYXNzPSJsYW5ndWFnZS1vYmplY3QiID5kAgIPEA8WAh8BaGRkFgBkAgQPFgIfAWhkAgYPFgIfAAUGPC9kaXY+ZAIID2QWAmYPDxYEHgdUb29sVGlwBRhJbmRpYW4gTGFrZSBJbmZvIE5ldHdvcmseC05hdm... --- End of inner exception stack trace --- at System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidati ) at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString, Purpose purpose) at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter2.Deserialize(String serializedState, Purpose purpose) at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter2 formatter, String serializedState, Purpose purpose) at System.Web.UI.HiddenFieldPageStatePersister.Load() at System.Web.UI.Page.LoadPageStateFromPersistenceMedium() at System.Web.UI.Page.LoadAllState() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) --- End of inner exception stack trace ---
Source:
Server Name: P3NW8SHG349
Then 1, 1, 1, and another group of 3 that same day at increasing frequencies...
9 on 10/26
5 on 10/27
7 on 10/28
27 on 10/29
12 on 10/30
ALL of these seemed to involve (only) default.aspx and were internal (UserI). They went unreported and unnoticed.
There were 45 errors part way through 10/31 [rebuild begun] with default.aspx, login.aspx and register.aspx all listed as the offending source. Logins became impossible and the issues reported. Below was typical from the logs:
AssemblyVersion: 7.0.6
PortalID: 0
PortalName: Indian Lake Info Network
UserID: -1
UserName:
ActiveTabID: 40
ActiveTabName: Home
RawURL: /asp/register.aspx?returnurl=http%3a%2f%2fwww.indianlake-pa.net%2fasp%2fUserProfile.aspx
AbsoluteURL: /asp/Default.aspx
AbsoluteURLReferrer: http://www.indianlake-pa.net/asp/register.aspx?returnurl=http%3a%2f%2fwww.indianlake-pa.net%2fasp%2fUserProfile.aspx
UserAgent: Opera/9.80 (Windows NT 5.1; U; YB/3.5.1; ru) Presto/2.10.229 Version/11.64
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke
ExceptionGUID: 9fc4b396-ab6c-4dd8-aab2-759822916109
InnerException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.
FileName:
FileLineNumber: 0
FileColumnNumber: 0
Method:
StackTrace:
Message: DotNetNuke.Services.Exceptions.PageLoadException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.HttpException: Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster. ---> System.Web.UI.ViewStateException: Invalid viewstate. Client IP: 46.246.62.42 Port: 56166 Referer: http://www.indianlake-pa.net/asp/register.aspx?returnurl=http%3a%2f%2fwww.indianlake-pa.net%2fasp%2fUserProfile.aspx Path: /asp/Default.aspx User-Agent: Opera/9.80 (Windows NT 5.1; U; YB/3.5.1; ru) Presto/2.10.229 Version/11.64 ViewState: /wEPDwUJNDMwNzQ0NDU3D2QWCGYPFgIeBFRleHQFejwhRE9DVFlQRSBodG1sIFBVQkxJQyAiLS8vVzNDLy9EVEQgWEhUTUwgMS4wIFRyYW5zaXRpb25hbC8vRU4iICJodHRwOi8vd3d3LnczLm9yZy9UUi94aHRtbDEvRFREL3hodG1sMS10cmFuc2l0aW9uYWwuZHRkIj4gZAICDxYCHwAFQyB4bWw6bGFuZz0iZW4tVVMiIGxhbmc9ImVuLVVTIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCJkAgQPZBYOAgYPFgIeB1Zpc2libGVoZAIHDxYEHgdjb250ZW50ZB8BaGQCCA8WAh8CBQ8sRG90TmV0TnVrZSxETk5kAgkPFgIfAgUPYnkgUGFkcmUgR3JhbmRlZAIKDxYCHwIFC0RvdE5ldE51a2UgZAILDxYCHwIFGEluZGlhbiBMYWtlIEluZm8gTmV0d29ya2QCDg8WAh8CBRFOT0lOREVYLCBOT0ZPTExPV2QCBg9kFgICAQ9kFgICBw9kFgJmD2QWNAIED2QWCGYPFgIfAAUePGRpdiBjbGFzcz0ibGFuZ3VhZ2Utb2JqZWN0IiA+ZAICDxAPFgIfAWhkZBYAZAIEDxYCHwFoZAIGDxYCHwAFBjwvZGl2PmQCCA9kFgJmDw8WBB4HVG9vbFRpcAUYSW5kaWFuIExh... --- End of inner exception stack trace --- at System.Web.UI.ViewStateException.ThrowError(Exception inner, String persistedState, String errorPageMessage, Boolean macValidati ) at System.Web.UI.ObjectStateFormatter.Deserialize(String inputString, Purpose purpose) at System.Web.UI.ObjectStateFormatter.System.Web.UI.IStateFormatter2.Deserialize(String serializedState, Purpose purpose) at System.Web.UI.Util.DeserializeWithAssert(IStateFormatter2 formatter, String serializedState, Purpose purpose) at System.Web.UI.HiddenFieldPageStatePersister.Load() at System.Web.UI.Page.LoadPageStateFromPersistenceMedium() at System.Web.UI.Page.LoadAllState() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) --- End of inner exception stack trace ---
Source:
Server Name: P3NW8SHG349
In retrospect I could have gone to the db directly and read the logs but at that point I wasn't sure if the db or the instance was corrupt.
Rebuild from another operating/backup instance was successful. Today (11/1) I got the following error every 10-20 minutes...
AssemblyVersion: 7.0.6
PortalID: -1
PortalName:
UserID: -1
UserName:
ActiveTabID: -1
ActiveTabName:
RawURL:
AbsoluteURL:
AbsoluteURLReferrer:
UserAgent:
DefaultDataProvider: DotNetNuke.Data.SqlDataProvider, DotNetNuke
ExceptionGUID: b45acaba-23ae-4454-bf90-fbd964902881
InnerException: Error Creating BusinessControllerClass 'DotNetNuke.Modules.ActiveForums.TopicsController, DotNetNuke.Modules.ActiveForums' of module(Active Forums) id=(513) in tab(142) and portal(0)
FileName:
FileLineNumber: 0
FileColumnNumber: 0
Method: System.Activator.CreateInstance
StackTrace:
Message: System.Exception: Error Creating BusinessControllerClass 'DotNetNuke.Modules.ActiveForums.TopicsController, DotNetNuke.Modules.ActiveForums' of module(Active Forums) id=(513) in tab(142) and portal(0) ---> System.ArgumentNullException: Value cannot be null. Parameter name: type at System.Activator.CreateInstance(Type type, Boolean nonPublic) at System.Activator.CreateInstance(Type type) at DotNetNuke.Services.Search.ModuleIndexer.GetModuleList(Int32 PortalID) --- End of inner exception stack trace --- at DotNetNuke.Services.Search.ModuleIndexer.GetModuleList(Int32 PortalID)
Source:
Server Name: P3NW8SHG349
Reinstalled AF 5.0.3. Logs are clear... 11/2/2013