I've been trying to understand why we're getting a OOM under Citrix - Windows 10, although it's not happening on "Normal" machines. Our application has a mix of native C++, C# and web browser plugins. The container is .net.
Details: Windows 10 Version 18363 MP (2 procs) Free x86 compatible Product: WinNt, suite: SingleUserTS 18362.1.wow64fre.19h1_release.190318-1202
2920000 730d0000 clr (pdb symbols)
Loaded symbol image file: clr.dll
Image path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Image name: clr.dll
Browse all global symbols functions data
Timestamp: Wed Aug 5 03:06:56 2020 (5F2A0630)
CheckSum: 007B81A2
ImageSize: 007B0000
File version: 4.8.4250.0
Product version: 4.0.30319.0
File flags: 8 (Mask 3F) Private
File OS: 4 Unknown Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
Information from resource tables:
CompanyName: Microsoft Corporation
ProductName: Microsoft® .NET Framework
InternalName: clr.dll
OriginalFilename: clr.dll
ProductVersion: 4.8.4250.0
FileVersion: 4.8.4250.0 built by: NET48REL1LAST_C
PrivateBuild: DDBLD342B
FileDescription: Microsoft .NET Runtime Common Language Runtime - WorkStation
LegalCopyright: © Microsoft Corporation. All rights reserved.
Comments: Flavor=Retail
The Exception on WinDbg
FAULTING_THREAD: ffffffff
STACK_TEXT:
011dd108 6e06611c PresentationCore_ni!System.Windows.Media.Imaging.BitmapSource.CreateCachedBitmap+0x47c
STACK_COMMAND: !D:\WindDbg\windbg\x86\sym\SOS_x86_x86_4.8.4250.00.dll\5F2A06307b0000\SOS_x86_x86_4.8.4250.00.dll.pe 0x26b56e80 ; ** Pseudo Context ** ManagedPseudo ** Value: 12630c10 ** ; kb
MODULE_NAME: Unknown_Module
IMAGE_NAME: Unknown_Image
FAILURE_BUCKET_ID: CLR_EXCEPTION_System.OutOfMemoryException_8007000e_nexus.framework.healthcare.exe!unknown_error_in_process
OS_VERSION: 10.0.18362.1
BUILDLAB_STR: 19h1_release
OSPLATFORM_TYPE: x86
OSNAME: Windows 10
FAILURE_ID_HASH: {196470ae-a638-36df-be48-e73b2df228f8}
*** Followup info cannot be found !!! Please contact...
The OOM comes rises on .Net and sometimes the application continues to work! Other times I get the exception already at ~700MB mem usage on taskmanager, which is really strange.
After getting the exception, I've managed to run PerfView and force a GC, getting it from > 750MB down to 350MB., on taskmanager. Somehow I've the feeling that the GC is not working as expected under Citrix.
PerfView shows actually a very small number of objects left behind, comparing with the beginning.
Also Vmap shows many ~75MB blocks on the heap - I guess this are from .Net, but the '?!?' bothers me. My guess is based on the fact that I've forced a GC on PerfView and these blocks were gone. Besides, the WPF blocks I saw in WPA are of the same wird size as on this ?!? block. It can't be a coincidence.
There is almost no fragmentation on the heap:
I have even tried to use wpr to find out what is causing the error, and I can see that WPF on .net is allocating ~75MB blocks, and this strange ?!? as well - I imagine this is jitted code. I have almost all symbols available, and still no luck - the ?1? stays there. Perhaps wpr was not saving .Net info - I will need to try with the profiles but i don't have access to the computer all the time.
This is the WPA views:
This is the what Windbg tells me about the memory:
As you can see, .net is using only around ~147MB of the part, the rest 200MB I'm not sure. VirtualAlloc? I thought they will go in the Heap part.
The reserved 200MB is also a big question, I will try to find out why it's there.
The dlls cannot be reduced - we need them all, specially if the user opens many plugins at once.
I wonder if this is a known Citrix problem. Is there anything else I can do to check why we get this OOM although we still have 175 MB free? I can't think why our application would need more than that at once. We are not showing big pictures nor videos.
I've checked GDI/Handles/User objects and all seems ok. They are not high.
One Idea I had, but did not try yet, is to change the GC settings to Workstation - Concurrent to see what happens. Not sure if this will help.
This is the stack:
0:000> k
# ChildEBP RetAddr
00 011dbd40 75def159 ntdll!NtWaitForSingleObject+0xc
01 011dbdb4 725e3370 KERNELBASE!WaitForSingleObjectEx+0x99
02 011dbde4 725e33cc clr!CLREventWaitHelper2+0x33
03 011dbe34 725e3319 clr!CLREventWaitHelper+0x2a
04 011dbe6c 725e333b clr!CLREventBase::WaitEx+0x14b
05 011dbe84 728d0125 clr!CLREventBase::Wait+0x1a
06 011dbef0 728d023a clr!Thread::WaitSuspendEventsHelper+0x8a
07 011dbf04 728ce36a clr!Thread::WaitSuspendEvents+0x14
08 011dbf1c 7267bcae clr!Thread::RareEnablePreemptiveGC+0x98
09 011dbf60 72b33ac5 clr!Thread::RareDisablePreemptiveGC+0x115
0a 011dbf84 72b2b333 clr!GCHolderEEInterface<0,1,1>::~GCHolderEEInterface<0,1,1>+0x61
0b 011dbfb8 72b2b4ea clr!Debugger::SendExceptionHelperAndBlock+0x133
0c 011dc030 72b2b86c clr!Debugger::SendExceptionEventsWorker+0x130
0d 011dc080 72b2b9d9 clr!Debugger::SendException+0x120
0e 011dc0a8 728841fa clr!Debugger::FirstChanceManagedException+0x19
0f 011dc0c4 726c15f4 clr!EEToDebuggerExceptionInterfaceWrapper::FirstChanceManagedException+0x31
10 011dc140 725e0754 clr!COMPlusThrowCallback+0x311
11 011dc40c 725e085a clr!Thread::StackWalkFramesEx+0x92
12 011dc740 726c1f01 clr!Thread::StackWalkFrames+0x9d
13 011dc8f0 726c238f clr!CPFH_RealFirstPassHandler+0x68e
14 011dc940 726c245a clr!CPFH_FirstPassHandler+0x119
15 011dc96c 72885fc7 clr!COMPlusFrameHandler+0x15d
16 011dc98c 770a8e72 clr!COMPlusFrameHandlerRevCom+0x17
17 011dc9b0 770a8e44 ntdll!ExecuteHandler2+0x26
18 011dca78 770942d6 ntdll!ExecuteHandler+0x24
19 011dca78 75e044c2 ntdll!KiUserExceptionDispatcher+0x26
1a 011dcf9c 726c28f0 KERNELBASE!RaiseException+0x62
1b 011dd038 726c3796 clr!RaiseTheExceptionInternalOnly+0x230
1c 011dd100 6e06611c clr!IL_Throw+0x146
WARNING: Stack unwind information not available. Following frames may be wrong.
1d 011dd1f0 6e04ff49 PresentationCore_ni+0x26611c
1e 011dd230 6e04feb2 PresentationCore_ni+0x24ff49
1f 011dd23c 6e04fe89 PresentationCore_ni+0x24feb2
20 011dd24c 6e04e3f0 PresentationCore_ni+0x24fe89
21 011dd390 6e04d910 PresentationCore_ni+0x24e3f0
22 011dd3cc 153582a3 PresentationCore_ni+0x24d910
23 011dd3dc 24be9b87 0x153582a3
24 011dd408 24be9a4d 0x24be9b87
25 011dd440 24be9986 0x24be9a4d
26 011dd458 24be96dd 0x24be9986
27 011dd470 1defd0c2 0x24be96dd
28 011dd4c4 1defcdb8 0x1defd0c2
29 011dd4d0 05de45c1 0x1defcdb8
2a 011dd51c 13ac31bb 0x5de45c1
2b 011dd52c 05de1fd8 0x13ac31bb
2c 011dd554 05de1e38 0x5de1fd8
2d 011dd56c 14a46136 0x5de1e38
2e 011dd588 14a46007 0x14a46136
2f 011dd5c0 725cf404 0x14a46007
30 011dd5e8 72694073 clr!COMToCLRDispatchHelper+0x6b
31 011dd644 0164e631 clr!COMToCLRWorker+0x4d5
32 011dd670 5b03e711 0x164e631
33 011dd6b0 5b03eeec NXIMFServer+0x7e711
34 011dd6f8 5afc480b NXIMFServer+0x7eeec
35 011dd79c 5b03aee0 NXIMFServer+0x480b
36 011dd82c 5b03ad7c NXIMFServer+0x7aee0
37 011dd8fc 5b03b147 NXIMFServer+0x7ad7c
38 011dd984 5b03cc47 NXIMFServer+0x7b147
39 011dd9bc 5b03afee NXIMFServer+0x7cc47
3a 011dda0c 760b829f NXIMFServer+0x7afee
3b 011dda34 7609a861 oleaut32!DispCallFunc+0x16f
3c 011ddcf8 76093e8d oleaut32!CTypeInfo2::Invoke+0x631
3d 011ddd24 5b013552 oleaut32!DispInvoke+0x2d
3e 011ddd4c 69f8e057 NXIMFServer+0x53552
3f 011dde10 69f8e45e mfc140!COleDispatchDriver::InvokeHelperV+0x227 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\oledisp2.cpp @ 399]
40 011dde30 25558ee1 mfc140!COleDispatchDriver::InvokeHelper+0x1e [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\oledisp2.cpp @ 557]
41 011dde5c 25558e01 MFListManAX!DllUnregisterServer+0x3951
42 011ddeb8 25558a3d MFListManAX!DllUnregisterServer+0x3871
43 011dded0 69fbe19b MFListManAX!DllUnregisterServer+0x34ad
44 011ddfa0 69fbd9cd mfc140!CWnd::OnWndMsg+0x79b [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 2698]
45 011ddfc4 69f2ade3 mfc140!CWnd::WindowProc+0x2d [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 2099]
46 011ddfec 69fbbcd7 mfc140!COleControl::WindowProc+0x133 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\ctlcore.cpp @ 1711]
47 011de064 69fbbf24 mfc140!AfxCallWndProc+0xb7 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 268]
48 011de084 2556bf30 mfc140!AfxWndProc+0x34 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 417]
49 011de0c0 753345cb MFListManAX!DllUnregisterServer+0x169a0
4a 011de0ec 753150dc user32!_InternalCallWinProc+0x2b
4b 011de1d0 7531462f user32!UserCallWinProcCheckWow+0x3ac
4c 011de208 679fe787 user32!CallWindowProcW+0x7f
4d 011de230 725cf36d ToolkitPro1631vc140!CXTPSkinManagerApiHook::OnHookCallWindowProcW+0x77
4e 011de270 079a7957 clr!PInvokeStackImbalanceHelper+0x22
4f 011de2b8 079a787a 0x79a7957
50 011de2fc 14741f41 0x79a787a
51 011de330 079a65a9 0x14741f41
52 011de338 079a6580 0x79a65a9
53 011de34c 079a7a56 0x79a6580
54 011de388 14508f10 0x79a7a56
55 011de3bc 753345cb 0x14508f10
56 011de3e8 753150dc user32!_InternalCallWinProc+0x2b
57 011de4cc 7531488d user32!UserCallWinProcCheckWow+0x3ac
58 011de530 75329f7d user32!SendMessageWorker+0x1fd
59 011de560 2555d0d9 user32!SendMessageA+0x11d
5a 011de58c 28f95acf MFListManAX!DllUnregisterServer+0x7b49
5b 011de788 12c7e319 ML3001_2G!CMFListManagerGridView::OnLButtonDblClkRowCol+0x89f
5c 011de7c8 12c683eb GX6082VC140R!CGXControl::LButtonDblClk+0xb9
5d 011de860 12cda891 GX6082VC140R!CGXGridCore::DoLButtonDblClk+0x6cb
5e 011de878 69fbdf55 GX6082VC140R!CGXGridView::OnLButtonDblClk+0x21
5f 011de94c 69fbd9cd mfc140!CWnd::OnWndMsg+0x555 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 2698]
60 011de970 12cdb074 mfc140!CWnd::WindowProc+0x2d [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 2099]
61 011de98c 69fbbcd7 GX6082VC140R!CGXGridView::WindowProc+0x44
62 011dea04 69fbbf24 mfc140!AfxCallWndProc+0xb7 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 268]
63 011dea24 2556bf30 mfc140!AfxWndProc+0x34 [d:\agent\_work\3\s\src\vctools\VC7Libs\Ship\ATLMFC\Src\MFC\wincore.cpp @ 417]
64 011dea60 753345cb MFListManAX!DllUnregisterServer+0x169a0
65 011dea8c 753150dc user32!_InternalCallWinProc+0x2b
66 011deb70 7531422e user32!UserCallWinProcCheckWow+0x3ac
67 011debe4 75314010 user32!DispatchMessageWorker+0x20e
68 011debf0 725cf36d user32!DispatchMessageW+0x10
69 011dec20 6eb374f1 clr!PInvokeStackImbalanceHelper+0x22
6a 011dec5c 6eb1b3d7 WindowsBase_ni+0xf74f1
6b 011deca4 6eb1b319 WindowsBase_ni+0xdb3d7
6c 011decb0 68782ecc WindowsBase_ni+0xdb319
6d 011decc0 68782a8a PresentationFramework_ni+0x2c2ecc
6e 011dece0 6878287e PresentationFramework_ni+0x2c2a8a
6f 011ded00 14e8e234 PresentationFramework_ni+0x2c287e
70 011ded28 016e0dff 0x14e8e234
71 011ded58 725cf036 0x16e0dff
72 011ded64 725d22da clr!CallDescrWorkerInternal+0x34
73 011dedb8 725d859b clr!CallDescrWorkerWithHandler+0x6b
74 011dee2c 7277b11b clr!MethodDescCallSite::CallTargetWorker+0x16a
75 011def50 7277b7fa clr!RunMain+0x1b3
76 011df1bc 7277b727 clr!Assembly::ExecuteMainMethod+0xf7
77 011df6a0 7277b8a8 clr!SystemDomain::ExecuteMainMethod+0x5ef
78 011df6f8 7277b9ce clr!ExecuteEXE+0x4c
79 011df738 72777305 clr!_CorExeMainInternal+0xdc
7a 011df774 7420fa84 clr!_CorExeMain+0x4d
7b 011df7ac 7477e80e mscoreei!_CorExeMain+0xd6
7c 011df7bc 74784338 mscoree!ShellShim__CorExeMain+0x9e
7d 011df7c4 76d16359 mscoree!_CorExeMain_Exported+0x8
7e 011df7d4 77087c24 kernel32!BaseThreadInitThunk+0x19
7f 011df830 77087bf4 ntdll!__RtlUserThreadStart+0x2f
80 011df840 00000000 ntdll!_RtlUserThreadStart+0x1b
And the Threads:
0:000> !threads
ThreadCount: 11
UnstartedThread: 0
BackgroundThread: 10
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
Lock
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
0 1 1fc8 0173da08 a6028 Preemptive 26B56FB0:00000000 01736698 1 STA System.OutOfMemoryException 26b56e80
3 2 2c84 01782be0 2b228 Preemptive 00000000:00000000 01736698 0 MTA (Finalizer)
6 3 279c 06031fc8 102a228 Preemptive 00000000:00000000 01736698 0 MTA (Threadpool Worker)
9 6 2404 0801d028 202b228 Preemptive 00000000:00000000 01736698 1 MTA
28 10 2298 21eddb10 a029228 Preemptive 00000000:00000000 01736698 0 MTA (Threadpool Completion Port)
33 8 29d0 0608bde0 1029228 Preemptive 00000000:00000000 01736698 0 MTA (Threadpool Worker)
34 7 2d08 2c5f1f10 1029228 Preemptive 26B51A60:00000000 01736698 0 MTA (Threadpool Worker)
35 4 1db0 21eb39c8 1029228 Preemptive 00000000:00000000 01736698 0 MTA (Threadpool Worker)
36 11 2550 2bc2d008 a1228 Preemptive 00000000:00000000 01736698 0 Ukn
38 5 2a0 0d7d2090 8029228 Preemptive 00000000:00000000 01736698 0 MTA (Threadpool Completion Port)
39 9 2010 2c502b50 8029228 Preemptive 00000000:00000000 01736698 0 MTA (Threadpool Completion Port)
******** Edit ********** So, i've searched for bitmaps:
Statistics:
MT Count TotalSize Class Name
6de2a780 1 12 System.Windows.Media.Imaging.BitmapCreateOptions
6de1144c 1 20 System.Windows.UncommonField`1[[System.Windows.Media.BitmapScalingMode, PresentationCore]]
6e796fdc 2 32 System.Windows.Media.Effects.BitmapEffectState
6de11368 2 40 System.Windows.UncommonField`1[[System.Windows.Media.Effects.BitmapEffectState, PresentationCore]]
6de20d8c 2 72 System.Windows.Media.Effects.DropShadowBitmapEffect
6de0e698 6 72 System.Windows.Media.BitmapScalingMode
6de2a748 7 84 System.Windows.Media.Imaging.BitmapCacheOption
6e7968f8 1 108 System.Windows.Media.Imaging.IconBitmapDecoder
6de202d8 15 300 System.Windows.Media.Imaging.BitmapPalette
6de2d9a4 19 304 System.Windows.Media.Imaging.BitmapSource+WeakBitmapSourceEventSink
6de217d4 4 432 System.Windows.Media.Imaging.JpegBitmapDecoder
6de20650 4 720 System.Windows.Media.Imaging.ColorConvertedBitmap
6de2d0e4 55 880 System.Collections.ObjectModel.ReadOnlyCollection`1[[System.Windows.Media.Imaging.BitmapFrame, PresentationCore]]
6de2d374 55 900 System.Windows.Media.Imaging.BitmapFrame[]
6de2d15c 55 1320 System.Collections.Generic.List`1[[System.Windows.Media.Imaging.BitmapFrame, PresentationCore]]
6de20434 9 1368 System.Windows.Media.Imaging.CachedBitmap
6de0a224 10 1400 System.Windows.Media.Imaging.UnmanagedBitmapWrapper
6de2cb5c 300 3600 System.Windows.Media.Imaging.BitmapInitialize
6de2200c 50 5400 System.Windows.Media.Imaging.PngBitmapDecoder
6de2d3e8 280 6720 System.Windows.Media.Imaging.BitmapSourceSafeMILHandle
6de2d22c 98 17640 System.Windows.Media.Imaging.BitmapFrameDecode
6de2a134 179 37232 System.Windows.Media.Imaging.BitmapImage
Total 1155 objects
The place where i get the OOM is not necesseraly the one responsible for the high memory usage, but anyway:
0:000> !pe
Exception object: 26b56e80
Exception type: System.OutOfMemoryException
Message: Nicht genügend Speicher verfügbar, um das Programm weiter auszuführen.
InnerException: <none>
StackTrace (generated):
SP IP Function
011DD108 6E06611C PresentationCore_ni!System.Windows.Media.Imaging.BitmapSource.CreateCachedBitmap(System.Windows.Media.Imaging.BitmapFrame, System.Windows.Media.Imaging.BitmapSourceSafeMILHandle, System.Windows.Media.Imaging.BitmapCreateOptions, System.Windows.Media.Imaging.BitmapCacheOption, System.Windows.Media.Imaging.BitmapPalette)+0x47c
StackTraceString: <none>
HResult: 8007000e
0:000> .exr -1
*** WARNING: Unable to verify checksum for PresentationCore.ni.dll
ExceptionAddress: 6e06611c (PresentationCore_ni+0x0026611c)
ExceptionCode: e0434f4d (CLR exception)
ExceptionFlags: 00000000
NumberParameters: 0
0:000> !clrstack
OS Thread Id: 0x1fc8 (0)
Child SP IP Call Site
011dd058 77091e4c [HelperMethodFrame: 011dd058]
011dd108 6e06611c System.Windows.Media.Imaging.BitmapSource.CreateCachedBitmap(System.Windows.Media.Imaging.BitmapFrame, System.Windows.Media.Imaging.BitmapSourceSafeMILHandle, System.Windows.Media.Imaging.BitmapCreateOptions, System.Windows.Media.Imaging.BitmapCacheOption, System.Windows.Media.Imaging.BitmapPalette)
011dd204 6e04ff49 System.Windows.Media.Imaging.CachedBitmap.FinalizeCreation()
011dd238 6e04feb2 System.Windows.Media.Imaging.CachedBitmap.EndInit()
011dd244 6e04fe89 System.Windows.Media.Imaging.CachedBitmap..ctor(System.Windows.Media.Imaging.BitmapSource, System.Windows.Media.Imaging.BitmapCreateOptions, System.Windows.Media.Imaging.BitmapCacheOption)
011dd25c 6e04e3f0 System.Windows.Media.Imaging.BitmapImage.FinalizeCreation()
011dd398 6e04d910 System.Windows.Media.Imaging.BitmapImage.EndInit()
Thanks, Marco
This will only be the case when
a) You can get an OOM exception at any memory usage. "Already" is not a scientific approach to this OOM. b) What is "mem usage"? Do you talk about virtual memory, .NET memory, native memory, working set size, peak working set size? Please familiarize yourself with the terms and use the correct one. Ideally, also name the tool you used for measuring it.
.NET may decide to do garbage collection whenever it wants. Sometimes, it prefers to run calculations over garbage collection. The fact that you could force a garbage collection and the memory decreased is an indicator to me that GC basically works.
One object can be enough for a memory leak.
What makes you guess that these are from .NET? .NET does not use the heap manager. It uses the Virtual Memory Manager. So these blocks are likely not from .NET.
Memory fragmentation can be an issue. But why care about fragmentation inside the native heaps?
Yes, that's likely.
Symbols aren't enough for debugging .NET and most applications can't deal with .NET. For .NET, the application needs to be able to handle the details on how .NET works. For WinDbg, that logic is encoded in the WinDbg SOS extension. And memory management details are available in MSCORDACWKS.DLL
As long as an application does not ask you for these, it'll likely not support .NET.
If your application wants to allocate 176 MB of memory at once, 175 MB will not be enough. Check how much memory your application needs at the time of the crash.
Why are you using a native stack command here but a managed thread command?
The problem
In the whole post I miss this distinction between managed and native stuff. You are mixing native commands like
kfor the stack with managed commands like!threads. You are analyzing a .NET exception with the native command!analyzeinstead of the managed command!pe. You are investigating a managed OOM exception but at the same time you have a look at the fragmentation of the native heaps.How to debug
Train yourself on how the memory works together. .NET memory will be part of the
<unknown>memory listed by!address -summary. However, some other libraries will also make use of the Virtual Memory Manager and thus be also listed as<unknown>. Bitmaps are a common example.To analyze the .NET memory, you need to use .NET specific commands that come from the SOS extension. Use
!dumpheap -statand!eeheap -gc. And do us a favor and don't post the output as images. Use.logopento get the text output.To better compare .NET memory, use a tool that is good at that. WinDbg isn't. Tools like JetBains dotMemory fit in here. Inside such a comparison, filter for objects that you know that need to be disposed.
What WinDbg lists as Heap, are native heaps. These typically come from a C++ library.
Just because you see a .NET OOM exception does not mean .NET is causing it. You could also have a look at other portions of the memory to avoid OOM: