xperf callstack tracing, specific to dlls

Question

I'm running into some problems getting full call stacks with my xperf traces. This is on a Win7 64bit pc. At a higher level, I have an exe that loads several dlls, which also do the bulk of the work. All pdb files are in a single directory, and I get valid symbols and call stacks in executable code, so I'm pretty sure my symbol path is setup correctly.

As soon as my exe calls into one of the loaded dlls, I lose the call stack information and the stack column displays plugin_name.dll!? which, when expanded, calls into itself. The weight tab shows a high percentage (50-90%) and a high count, but I'm unable to get any actual function calls for the stack. For example:

Process, Stack, Module, Function, Weight, % Weight, Count, TimeStamp
,    |     kernel32.dll!BaseThreadInitThunk, , , 59067.075556, 73.29, 59075, 
,    |     |- plugin_name.dll!?, , , 45036.186642, 55.88, 45042, 
,    |     |     plugin_name.dll!?, , , 45036.186642, 55.88, 45042, 

Based on these,

http://blogs.msdn.com/b/pigscanfly/archive/2009/08/06/stack-walking-in-xperf.aspx http://msdn.microsoft.com/en-us/library/windows/desktop/ff191014(v=vs.85).aspx
http://msdn.microsoft.com/en-us/library/ff191014.aspx

this seems to indicate that xperf

• "...doesn’t know what executable image was at the address"
• "...Maximum stack depth is exceeded".

The possible causes for this that I've found are either the FPO (frame pointer optimizations) is turned on (I've explicitly disabled with /Oy- even though they shouldn't be on), or the max stack depth is exceeded, which I'm not sure how to determine if this is what I'm hitting or not.

Without being able to see where time is being spent in the dlls, I'm not able to get very much useful information, so I'd like to figure out what is going on.

Does anyone have any recommendations for things to read or try? Are there any limitations with stackwalk that I'm missing? Is there any special setup for symbols, specific to call stacks in dlls?

Answer 1

create a new envvar DBGHELP, set it to 1. Now create a the envvar DBGHELP_LOG and set it to a log file like D:\DbgHelpLog.txt. After symbol loading is finished, close xperfview/wpa and look into the log why symbol loading failed.

Diagnosing DBGHELP symbol loading

http://blogs.msdn.com/b/matt_pietrek/archive/2005/04/12/407721.aspx

Answer 2

Try the following, from here: Disable Paging Executive

In order for tracing to work on 64-bit Windows you need to set the DisablePagingExecutive registry key. This tells the operating system not to page kernel mode drivers and system code to disk, which is a prerequisite for getting 64-bit call stacks using xperf, because 64-bit stack walking depends on metadata in the executable images, and in some situations the xperf stack walk code is not allowed to touch paged out pages. Running the following command from an elevated command prompt will set this registry key for you.

REG ADD "HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management" -v DisablePagingExecutive -d 0x1 -t REG_DWORD -f

After setting this registry key you will need to reboot your system before you can record call stacks. Having this flag set means that the Windows kernel locks more pages into RAM, so this will probably consume about 10 MB of additional physical memory.

Answer 3

Mostly symbol loading issue is occurred due to mismatched debug engine and symbol server libraries.

Do install latest sdk containing windows performance kit and running wpa/xperf from same directory should resolve the symbol loading issue. i 'm assuming symbol server path is correct. Use procexp.exe to see dbghelp.dll and symsrv.dll version and path to see xperf or any tool from directory is pointing out to latest available or shipped version of above mentioned dlls.

Thanks, SushantB