Allow me to tell you what I discovered the root of the problem.
I created a mirror web portal (dnn 6.2.4 .net 4 with sql 2012 web edition db) to a server of one of our clients in Jordan.
Initially it was put on a temporary url. To our surprised for a few days the speed was super fast, although the bandwidth and the hardware was lower than my server with the host (4 core xeon, 4 GB Ram, Win 2012 Std with sql 2012 web)
We used red-gate and sql profilers and saw nothing serious to warrant a red flag.
Then we decided to go a step further and change the dns setting of our active domain to the local testing server.
For the first 6 hours or so, it maintained the same performance, then we started to experience the same issues of high cpu.
It behaved like this: up to 25% hovers around it then jumped 50% hovered around 50% jumped to 75% hovered around 75% then jumped to 100% stuck and the site become non responsive and generated server error messages, We manually killed the w3wp.exe when it reached 100% to keep the site alive
Initially we suspected a malicious spider or crawler. We know spiders consume very little band-width hence difficult to detect.
The site is as such
google indexed pages: currently over 25,000.
daily average visitors unique based on smarter tools stats (IIS raw logs) over 1,000 unique visitor with pages view average 3 per visitor (obviously not including real spiders) Google analytics figures much less.
The portal is not considered high traffic, but very targeted and focused
From the dump log we discovered some troublesome urls, they all had /en-gb/ or language=en-gb
a year or so i had en-gb language pack installed on the site, then removed it.
We checked the db and discovered reference to /en-gb/ without any fallback, it appeared to have been uninstalled but not fully. And google still has some urls with /en-gb/ or language=en-gb indexed.
Since the problems started i installed the Arabic language pack to launch an Arabic version of my portal
Hence all the urls now are language enabled (either /en-us/ or /ar-jo/) In dotnetnuke every language package has a fallback language,
We tried an exercise the minute we reproduced a url with /en-gb/, the site went into a spine and unrecoverable loop, and saw the surge of cpu to 100%.
We enabled request filters on the IIS to deny /en-gb/, it was magic for 6 hours or so, we tested it with /en-gb/ and it was handled
The problem reappeared again but this time with less severity but the same behavior (25 - 50 - 75 - 100) at 100% the cpu stayed for 10 to 15 mins non responsive. while in this case the site is back to normal after that, with no server error messages.
On IIS 8, one can throttle down the cpu at a defined %, but the root of the problem is no solved.
I suspect low bandwidth malicious crawlers, obviously these can be filtered out by branded hardware firewalls but their cost is out of reach for me. Although I am a member of Microsoft BizSpark program and have access to most MS products (MSDN) but could not find a MS product to protect web servers, yet