Windbg crash dump analysis

Question

I'm having a hard time getting any meaningful information from a crash dump I created with ProcDump, but I'm pretty sure it's relevant to a seemingly random crash I've been having.

I have a VB6 application running on Windows 7 64-bit. Every once in a while, it crashes, leaving an entry in the error log that faults ntdll.dll but gives no more information than that. So, I have been running the process with SysInternals' ProcDump running to automatically create crash dumps for me.

I've been unable to re-create the crash in-house, so I was pretty sure that if I had a dump at all, it would tell me what the issue was. However, after running for most of a day, I see that ProcDump wrote several dumps already, although the program is still running fine. It does seem to point to issues with ntdll.dll, but I have no idea where to begin applying a fix for this.

Running !analyze -v on one of the dumps gives me the following:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


FAULTING_IP: 
+0
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

FAULTING_THREAD:  000007c8

PROCESS_NAME:  application.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

APP:  application.exe

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT_AFTER_CALL

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT_AFTER_CALL

LAST_CONTROL_TRANSFER:  from 7754431f to 7752014d

STACK_TEXT:  
0382fdf4 7754431f 00000005 035e62c8 00000001 ntdll!ZwWaitForMultipleObjects+0x15
0382ff88 74cd339a 00000000 0382ffd4 77539ed2 ntdll!TppWaiterpThread+0x33d
0382ff94 77539ed2 035e6298 74e2a30c 00000000 kernel32!BaseThreadInitThunk+0xe
0382ffd4 77539ea5 775441f3 035e6298 00000000 ntdll!__RtlUserThreadStart+0x70
0382ffec 00000000 775441f3 035e6298 00000000 ntdll!_RtlUserThreadStart+0x1b


STACK_COMMAND:  ~0s; .ecxr ; kb

FOLLOWUP_IP: 
ntdll!ZwWaitForMultipleObjects+15
7752014d 83c404          add     esp,4

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ntdll!ZwWaitForMultipleObjects+15

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ce7ba58

FAILURE_BUCKET_ID:  STATUS_BREAKPOINT_AFTER_CALL_80000003_ntdll.dll!ZwWaitForMultipleObjects

BUCKET_ID:  APPLICATION_FAULT_STATUS_BREAKPOINT_AFTER_CALL_ntdll!ZwWaitForMultipleObjects+15

WATSON_STAGEONE_URL:  http://watson.microsoft.com/StageOne/BlackJack_exe/1_5_0_0/50227d4e/unknown/0_0_0_0/bbbbbbb4/80000003/00000000.htm?Retriage=1

Followup: MachineOwner

Can anyone point me in the right direction, in terms of making sense of this entry, and what I can do about it?

Answer 1

Just to make sure, I've performed some tests on my side, attaching to healthy processes and making dumps of just started process. In all cases output of !analyze -v was pretty similar to yours, except of the fact that my one is more verbose, I think it depends on debugger version.

For example, here is the output that I've got after attaching to just started Paint:

*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************

GetPageUrlData failed, server returned HTTP status 404
URL requested:     http://watson.microsoft.com/StageOne/mspaint_exe/6_1_7600_16385/4a5bca29/ntdll_dll/6_1_7601_17725/4ec4aa8e/80000003/00050530.htm?Retriage=1

FAULTING_IP: 
ntdll!DbgBreakPoint+0
00000000`76d90530 cc              int     3

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000076d90530 (ntdll!DbgBreakPoint)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 1
Parameter[0]: 0000000000000000

FAULTING_THREAD:  0000000000000cbc

DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT

PROCESS_NAME:  mspaint.exe

ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION}  Breakpoint  A breakpoint has been reached.

EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid

EXCEPTION_PARAMETER1:  0000000000000000

MOD_LIST: <ANALYSIS/>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT

BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT

LAST_CONTROL_TRANSFER:  from 0000000076e37ef8 to 0000000076d90530

STACK_TEXT:  


FOLLOWUP_IP: 
ntdll!DbgBreakPoint+0
00000000`76d90530 cc              int     3

SYMBOL_STACK_INDEX:  0

SYMBOL_NAME:  ntdll!DbgBreakPoint+0

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: ntdll

IMAGE_NAME:  ntdll.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4ec4aa8e

STACK_COMMAND:  ~8s ; kb

FAILURE_BUCKET_ID:  STATUS_BREAKPOINT_80000003_ntdll.dll!DbgBreakPoint

BUCKET_ID:  X64_APPLICATION_FAULT_STATUS_BREAKPOINT_ntdll!DbgBreakPoint+0

WATSON_STAGEONE_URL:      http://watson.microsoft.com/StageOne/mspaint_exe/6_1_7600_16385/4a5bca29/ntdll_dll/6_1_7601_17725/4ec4aa8e/80000003/00050530.htm?Retriage=1

Followup: MachineOwner
---------

I've also taken a look at the explanation of ProcDump flags here: http://technet.microsoft.com/en-us/sysinternals/dd996900.aspx. It looks like using command line like

C:\apps\procdump.exe -accepteula -e -h -n 10 -t -w application.exe

you make procdump stop on every sign of hanging or exception without setting specific parameters, like memory usage number or procent of CPU usage.

I would suggest to use DebugDiag, which provides nice UI, where you could configure rules describing when dump should be created. Here is an explanation from me, how to gather a dump when you have excessive memory usage problem, or high CPU usage:

http://kate-butenko.blogspot.com/2012/06/how-to-gather-dump-with-debugdiag.html

and here is another fine screenshot-based explanation, how to get a dump in DebugDiag for specific exception:

http://blogs.msdn.com/b/kaushal/archive/2012/05/09/using-debugdiag-to-capture-a-dump-on-first-chance-exception.aspx

From set of more lightweight tools you may also check the AdPlus tool (resides in C:\Program Files\Debugging Tools for Windows (x64) folder). I prefer DebugDiag, as it allows to catch specific types of exceptions.