Sakis:
I didn't pay much attention to the IE7 emphasis initially, sorry about that. You are correct, if Firefox works okay, then the issue is with IE7 and there have been several documented since its inception. First of all, I assume you have your patches up to date right? Then, and please bear with me, you may want to add your site to the Trusted Sites in the Security section. Also, turn off the Phishing Filter, this has been documented to slow down access to some sites.
I would also check the caching settings for IE (you may have done this already), it is not intuitive but this is under Tools, General tab, Browsing History, Settings. Make sure that "Automatically" is selected. By the way, have you tried IE7 from other machines?
Hope some of this works, I'm running out of ideas. In any case, you may want to search in the Microsoft's knowledgebase (or Google) because there have been documented issues with the performance of IE7.
Carlos