I had run into the same errors when testing the DotNetNuke Gallery module against 5.4.0 before its release. In the case of the FCKEditorProvider, both of the pop-up pages fckStyles.aspx and fckCSS.aspx do not include html Form tags but do in their codebehind files inherit from the core class DotNetNuke.Framework.PageBase. Whenever .aspx pages inheriting from this base class are loaded, an attempt is made to insert an AJAX ScriptManager into the page's controls. In versions of DotNetNuke prior to 5.4.0, a Try/Catch block had been placed around the code (in DotNetNuke.Framework.AJAX.vb) for adding the ScriptManager so that should the code fail, any exception would be gracefully ignored. In 5.4.0, the Try/Catch block had been removed with the understanding that since MS AJAX was always installed and enabled, the code could not fail. However, it will fail when there is no Form tag on the aspx page.
For the Gallery module, I had to re-design its pop-up pages so that they did not inherit from DotNetNuke.Framework.PageBase. The FCKEditorProvider will have to undergo a similar change. However, with the backporting of the Telerik RADEditorProvider from DotNetNuke Professional Edition in version 5.4.0, I'm not sure what the plans are for continuing support of the FCKEditorProvider.
As a fix for now, you may want to change the htmlEditor provider defaultProvider from FckEditorProvider to TelerikEditorProvider which is the default on new installs of 5.4.0 but not on upgrades. As I havn't yet done a test upgrade, I'm not sure if the TelerikEditorProvider is installed during an upgrade or if it has to be installed manually from Host-->Extensions. I have experimented briefly with the Telerik editor and so far really like it - will take a while to get used to, though.