Chris hit the nail on the head for the proper process, but keep in mind there is a high potential that it will NOT be an easy upgrade, as if the core files that you have modified have been changed by the core, you must re-work your changes into the core.
I have completed a number of upgrades with customized core installations, typically the following process has worked for me, but again, it might be different for you.
What was done?
First I identify what was changed, if you do not have a comprehensive, line-by-line listing of the changes I would do something similar to the following.
1.) Install an out of the box installation of DotNetNue for the CURRENT version.
2.) Check that version into a source control system
3.) Modify the bindings of your source control to point to your existing system
4.) Your source control system can then compare files, and you will need to find ALL differences.
Upgrade?
Now that you know what was done, do an ugrade like you normally would, keep in mind that ALL changes will be lost at this time.
Re-Apply
Now that you have upgraded, re-apply, or re-work all of your core modifications into the framework.
I hope that at least helps a bit.