Web Based Stack Dump Tool for ASP.NET Using Mdbg?

Question

There is a great presentation by Dan Farino, Chief Systems Architect at MySpace.com, showcasing a web-based stack dump tool that catalogues all threads running in a given process (what they're doing, how long they've been executing, etc.)

Their techniques are also summarized on highscalability.com:

• PerfCollector.
  Centralized collection of performance data via UDP. More reliable than Windows and allows any client to connect and see stats.
• Web Based Stack Dump Tool.
  Can right-click on a problem server and get stack dump of the .Net managed threads. Used to have to RDC into system and attach a debugger and 1/2 later get an answer. Slow, nonscalable, and tedious. Not just a stack dump, gives a lot of context about what the thread is doing. Troubleshooting is easier because you can see 90 threads are blocked on a database so the database may be down.
• Web Base Heap Dump Tool.
  Dumps all memory allocations. Very useful for developers. Save hours of doing it by hand. • Profiler. Traces a request from start to finish and produces a report. See URL, methods, status, everything that will help you identify a slow request. Looks at lock contentions, are a lot of exceptions being thrown, anything that might be interesting. Very light weight. It's running on one box in every VIP (group of 100 servers) in production. Samples 1 thread every 10 seconds. Always tracing in background.

The question is: what tools are required to build a web-based stack dump tool for ASP.NET? For convenience, let's assume that an *.aspx hosted in the target AppDomain, able to output all managed call stacks in that process, is sufficient.

There are a few posts that cover the use of Mdbg (debugger for managed code written entirely in C#/IL that started shipping with CLR 2 SDK) and the mdbgcore assembly typically found in C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bin:

• http://dotnetdebug.net/2005/11/09/exceptiondbg-v01-debug-your-exceptions/
• http://blogs.msdn.com/jmstall/archive/tags/MDbg/default.aspx
• http://blogs.msdn.com/vijaysk/archive/2009/11/04/asp-net-debugger-extension-for-iis-7.aspx

Would a solution simply reference this assembly to produce the desired output? What impact would a "list all managed call-stacks" operation have on the running process that's servicing production traffic?

Answer 1

I believe the profiling API of .Net are the way to go.

Look at the SlimTune project on Google Code to have a live sample, with sources, that you can check how to adapt and improve to work in a Asp.NET scenario.

Regards Massimo

Answer 2

With the profiling API of .Net you have to stop the server and it takes a lot CPU (but it gives you full control over all called methods).

I think the most “light way” solution is to doing this with MDbg, I put together a very small but useful little app called StackDump that does the following: 1) The debugger stops the application and generates a list of all CLR stacks running for the process. 2) The application is started again. This operation is a quick operation and can (maybe) be executed on a running production server with unchanged production code.

It just 80 lines of .Net code to manage this. I have published the source code on Codeplex.