I really went through the wringer yesterday with a failed upgrade from a 6.x release to 7.3.4.  I am looking for suggestions of things to examine to build confidence that the site is healthy now and perhaps some discussion on what can be done to improve the upgrade experience.

I tested my upgrade several times in my own local development environment.  There were a few minor issues that were easily fixed with a little Google-fu, for example the App_Code XML contents needing to be cleared out, and some issues with mimetypes  and assembly reference version mapping in my web.config.  No big deal, I was ready to push the upgrade button in production, or so I thought.

What got me was the SQL2008 dependency.  My local environment is on SQL2014.  Our site is hosted at PowerDNN and has been for years, since about 2008.  It was still on SQL2005.  After the first few SQL dataprovider migration scripts completed everything started erroring out and I knew I was in for some work.

My successful experience testing the upgrade locally made me overconfident.  I did not have a fresh backup.  I backed up the database a few days ago (Nov 27) because that is a fairly painless process, but had not bothered taking a recent download of the file system because it is a couple GB, about 18,000 files and takes a while to ftp down.  My bad.

I had PowerDNN migrate our account to newer infrastructure with SQL2008.  That took 5 hours of downtime to complete, which was disappointing.  Then I tried and failed to resume the upgrade after doing a fresh unzip of the upgrade zip.  The upgrade page was just blank, it never even tried to do anything.  So I pulled the trigger on asking PowerDNN to restore their backups of my site from Nov 29 (the night before).  That did not go so well.  They took over 12 hours to complete this task, twice restoring incorrect backups.  First they restored a backup from April 2013 (!) and the second time a backup from Nov 13 2014.  I've had some good conversations with them about this -- they fully recognize this is not up to their standards for service excellence.

After the second incorrect backup restore, at about midnight after working on this since 8:00 AM, I started to really scratch my head and think of ways to get this thing upgraded on my own.  I wasn't sure PowerDNN would ever be able to restore a backup from Nov 29 and I wasn't excited about being up all night waiting for the third attempt to restore a backup.  I thought since what failed was the SQL provider migration scripts, maybe the file system was OK-ish and the upgrade might run if I could just get it a clean db from before any upgrade tasks ran.  I had a db backup from Nov 27, and given it was Thanksgiving weekend there were minimal content changes on the site between Nov 27 and Nov 30.  I checked the Blog_Entries and HtmlText tables which are the bread and butter of content on our site and found only one update made between those days that wasn't in my Nov 27 db backup.

This worked fine.  I think.  No errors reported and all of my post-upgrade maintenance proceeded with no indication of any problems in the upgraded system. 

Question 1:  hearing all of this do have any further suggestions on how to examine the system to evaluate its health and build confidence that the upgrade was a success, given the slightly unorthodox state it was upgraded from?  Of course I've been keeping an eye on the Event Viewer and it is pretty clean, and I'm doing a lot to exercise various features of the site to be sure they are working, but are there other places to look for a canary in the coalmine?

Question 2:  What can be done to improve the upgrade experience?  I can't help but wonder why the DNN upgrade process can't perform a few basic checks to determine what database version you are on.  It would have been nice if the upgrade just recognized it was running on SQL2005 and stopped before changing anything.  I've opened a feature request in Jira to this end.  I know the mantra is "backup before upgrading", and I've seen good advice that holds you should backup before installing any extensions as well.  It seems like more could be done to help prevent folks from shooting themselves in the foot.  My bad for getting into this situation without a fresh backup, but I think there is an opportunity to improve the software here too.

They do nightly backups.  I've had the database backup pulled before and it took about 15 minutes.  For some reason pulling the database plus file system takes hours.  

The rough part of this was the mistakes in pulling old backups, and the lack of awareness that they had pulled the wrong data.  I was quite surprised they even had a backup still from April 2013.  I was given to understand they only kept the last 5 days.  They eventually did get the right backup on the third try, over 12 hours after the initial request.