You can look at the portalsettings to see if popups are enabled globally

if (PortalSettings.EnablePopUps)
{
}

And you could look at module info to see if popups are active for the current module.
But this is a little more complex as the Supports popup information is stored separately for each control in you module
- do if you have a View control and an Edit control for example each has its own settings.

ModuleController mc = new ModuleController();
ModuleInfo moduleInfo = mc.GetTabModule(TabModuleId);
String controlKey = "View"; // get the info for the View control of the current tabmoduleid.
ModuleControlInfo moduleControlInfo = ModuleControlController.GetModuleControlByControlKey (controlKey, moduleInfo.ModuleDefID);
if( moduleControlInfo.SupportsPopUps)
{
}

BUT having said all this - the only time you would need to be changing what a button looks like is when the module in question is being rendered.

It then that you need to do different things - its when a module IS being displayed in a popup that you want things to work differently.
And as such - you can test that by simply looking at the querystring at runtime and testing for popup=true. Thats the only point where you should
ever really care what action a button needs to take.

Westa