Yes, I have some luck with that and working fine for me.
If you are using AJAXControlToolkit ver 1.x then it should be straight forward, what is the actual error message you are getting? Log as a host or check event log, it will give you right error message. See steps from below for ver1.x, step3 does not required.
But if you are using AJAXControlToolkit ver 3.x then there is some issue with DNN and below is the work arround:
Steps:
1. Download AjaxControlToolkit dlls and put them into the DNN bin folder. you can download a copy of it from the microsoft site.
2. Open your DNN module project and add reference to the above AjaxControlToolkit.dll and build your project (depends on your project setup, your dll will be copied to DNN bin folder). I have noticed following changes in main web.config file
<dependentAssembly>
<assemblyIdentity name="System.Web.Extensions" publicKeyToken="31bf3856ad364e35" />
<bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0" />
</dependentAssembly>
<dependentAssembly>
<assemblyIdentity name="System.Web.Extensions.Design" publicKeyToken="31bf3856ad364e35" />
<bindingRedirect oldVersion="1.0.0.0-1.1.0.0" newVersion="3.5.0.0" />
</dependentAssembly>
And do your normal work like, add usercontrol or extender from toolbar to your module ascx page and write your code behind code ......
3. As DNN is using its own ScriptManager rather than the ToolkitScriptManager, so the work arround for it (not a best option, unless dynamically you can do it from your module code, have not found enough time to play with this):
- open DNN default.aspx page and add following code
<asp:ToolkitScriptManager ID="ToolkitScriptManager1" runat="server">
</asp:ToolkitScriptManager>
just after <dnn:Form id="Form" runat="server" ENCTYPE="multipart/form-data" >
4. Delete any web.config from you module folder, which is under "DesktopModules".
5. Open DNN website and load your module, it should work.
Hope this helps.