This is my first time looking into WinDbg. We have one of our servers getting into a weird state regularly (once a month). The "weird state" - is that the application that processes the requests thinks there is not enough storage left, when there should be more than plenty. The issue seems to be related to w3p process. To get more details to help us analyse the potential root cause - we have collected a Dump file for the w3p process. But reading the dump file (using the latest version of WinDbg) - does not give me any hint at all. Can anyone please help me to understand this?
Here is the analysis of the dump file:
************* Preparing the environment for Debugger Extensions Gallery repositories **************
ExtensionRepository : Implicit
UseExperimentalFeatureForNugetShare : false
AllowNugetExeUpdate : false
AllowNugetMSCredentialProviderInstall : false
AllowParallelInitializationOfLocalRepositories : true
-- Configuring repositories
----> Repository : LocalInstalled, Enabled: true
----> Repository : UserExtensions, Enabled: true
>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.016 seconds
************* Waiting for Debugger Extensions Gallery to Initialize **************
>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.016 seconds
----> Repository : UserExtensions, Enabled: true, Packages count: 0
----> Repository : LocalInstalled, Enabled: true, Packages count: 36
Microsoft (R) Windows Debugger Version 10.0.25921.1001 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [C:\Users\user1\Downloads\w3wp\w3wp.DMP]
User Mini Dump File with Full Memory: Only application data is available
************* Path validation summary **************
Response Time (ms) Location
Deferred srv*https://msdl.microsoft.com/download/symbols
Symbol search path is: srv*https://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Version 14393 MP (8 procs) Free x64
Product: Server, suite: TerminalServer DataCenter SingleUserTS
Edition build lab: 10.0.14393.6343 (rs1_release.230913-1727)
Debug session time: Wed Dec 20 04:10:12.000 2023 (UTC - 8:00)
System Uptime: 22 days 5:04:36.302
Process Uptime: 0 days 6:05:14.000
................................................................
................................................................
................................................................
.......................................
Loading unloaded module list
..........................................
For analysis of this file, run !analyze -v
ntdll!NtWaitForSingleObject+0x14:
00007ffa`11625ea4 c3 ret
0:000> .sympath SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
KEY_VALUES_STRING: 1
Key : Analysis.CPU.mSec
Value: 3721436
Key : Analysis.Elapsed.mSec
Value: 3732286
Key : Analysis.IO.Other.Mb
Value: 0
Key : Analysis.IO.Read.Mb
Value: 1
Key : Analysis.IO.Write.Mb
Value: 2
Key : Analysis.Init.CPU.mSec
Value: 1061
Key : Analysis.Init.Elapsed.mSec
Value: 24185
Key : Analysis.Memory.CommitPeak.Mb
Value: 1576
Key : CLR.BuiltBy
Value: NET48REL1LAST_B
Key : CLR.Engine
Value: CLR
Key : CLR.Version
Value: 4.8.4645.0
Key : Failure.Bucket
Value: BREAKPOINT_80000003_w3wphost.dll!AppHostInitialize
Key : Failure.Hash
Value: {c8bd1ff0-1550-3c6c-7435-bfa31a670998}
Key : Timeline.OS.Boot.DeltaSec
Value: 1919076
Key : Timeline.Process.Start.DeltaSec
Value: 21914
Key : WER.OS.Branch
Value: rs1_release
Key : WER.OS.Version
Value: 10.0.14393.6343
Key : WER.Process.Version
Value: 10.0.14393.0
FILE_IN_CAB: w3wp.DMP
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
APPLICATION_VERIFIER_FLAGS: 0
EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 0000000000000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
FAULTING_THREAD: 0002f848
PROCESS_NAME: w3wp.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE_STR: 80000003
STACK_TEXT:
00000025`8cfdf5f8 00007ffa`0e366d1f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!NtWaitForSingleObject+0x14
00000025`8cfdf600 00007ff9`fec818ac : 00000204`1ffaa6a0 00000025`8cfdf740 00007ff6`00000000 00000000`000001f4 : KERNELBASE!WaitForSingleObjectEx+0x8f
00000025`8cfdf6a0 00007ff6`abb1156e : 00000000`fffffffe 00000204`1ffa6a40 00000204`1ffa6a40 00007ff9`fec80000 : w3wphost!AppHostInitialize+0x14c
00000025`8cfdf6e0 00007ff6`abb12dcd : 00000000`00000001 00000000`00000000 00000000`00000001 00000000`00000000 : w3wp!wmain+0x412
00000025`8cfdf880 00007ffa`0eec84d4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : w3wp!__wmainCRTStartup+0x14d
00000025`8cfdf8c0 00007ffa`115d1791 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14
00000025`8cfdf8f0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21
STACK_COMMAND: ~0s; .ecxr ; kb
SYMBOL_NAME: w3wphost!AppHostInitialize+14c
MODULE_NAME: w3wphost
IMAGE_NAME: w3wphost.dll
FAILURE_BUCKET_ID: BREAKPOINT_80000003_w3wphost.dll!AppHostInitialize
OS_VERSION: 10.0.14393.6343
BUILDLAB_STR: rs1_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 10.0.14393.2339
FAILURE_ID_HASH: {c8bd1ff0-1550-3c6c-7435-bfa31a670998}
Followup: MachineOwner
---------
UPDATED (based on some suggestions).
Here is the Hang analysis output, which seems to provide even less info:
0:000> .sympath SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
Expanded Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
************* Path validation summary **************
Response Time (ms) Location
Deferred SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
0:000> !analyze -hang
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
PROCESS_NAME: w3wp.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
SYMBOL_NAME: w3wphost!AppHostInitialize+14c
MODULE_NAME: w3wphost
IMAGE_NAME: w3wphost.dll
FAILURE_BUCKET_ID: APPLICATION_HANG_cfffffff_w3wphost.dll!AppHostInitialize
FAILURE_ID_HASH: {9f08a9ce-8bf0-138e-28fb-0c706c2e5fa0}
Followup: MachineOwner
---------
0:000> k
# Child-SP RetAddr Call Site
00 00000025`8cfdf5f8 00007ffa`0e366d1f ntdll!NtWaitForSingleObject+0x14
01 00000025`8cfdf600 00007ff9`fec818ac KERNELBASE!WaitForSingleObjectEx+0x8f
02 00000025`8cfdf6a0 00007ff6`abb1156e w3wphost!AppHostInitialize+0x14c
03 00000025`8cfdf6e0 00007ff6`abb12dcd w3wp!wmain+0x412
04 00000025`8cfdf880 00007ffa`0eec84d4 w3wp!__wmainCRTStartup+0x14d
05 00000025`8cfdf8c0 00007ffa`115d1791 kernel32!BaseThreadInitThunk+0x14
06 00000025`8cfdf8f0 00000000`00000000 ntdll!RtlUserThreadStart+0x21