Memory fragmentation?

Question

I'm not sure because I don't have much experience in analysing memory dumps, but I think we may have problems with memory fragmentation.

During load tests we see that memory usage is growing to the point where application restarts. It is ASP.NET MVC 4 app on 64 bit machine. I wasn't involved in writing it. I was just asked to try to analyze memory dumps.

So during last load test we created 3 memory dumps (below their sizes and total GC Heap size output from eeheap -gc):

1. 1.70GB, 292MB
2. 2.03GB, 337MB
3. 2.55GB, 347MB

So as you see managed heap isn't growing as much as dump files. When I do dumpheap -stat I see most space is used by Free objects (below for each dump file)

1. 147MB
2. 145MB
3. 213MB

Fragmented blocks larger than 0.5 MB:
            Addr     Size      Followed by
000000bcc668e0a8    0.7MB 000000bcc6738650 System.Object[]
000000bcc6949f88    4.4MB 000000bcc6dab820 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
000000bd4626c4b8    0.7MB 000000bd463165f8 System.Byte[]
000000bd463fcc48   51.5MB 000000bd4977baf0 System.Threading.ThreadStart
000000be463600c8    0.7MB 000000be464108f0 Free
000000bec67e50e0    1.1MB 000000bec690b020 System.Collections.Generic.List`1[[OurType, ANotherOurType]]
000000bec690b0b8    3.2MB 000000bec6c3b170 System.Byte[]
000000bfc6605e00    1.0MB 000000bfc6710190 Free
000000bfc6743c58   32.8MB 000000bfc8806fe8 System.Threading.ExecutionContext
000000c046200580    1.0MB 000000c0462ff2a0 SomeOurType
000000c0463a1270    3.6MB 000000c046732ac0 Microsoft.Win32.SafeHandles.SafeCapiKeyHandle

From what I understand it is not problem when Free objects are only small fraction of total amount of memory in all heap sizes. Here looks like it is a problem.

App is using two external libraries. One for creating PDF-s and other for creating Barcode files. Barcode library is throwing AccessViolationException (about 70 times for 2200 attempts). It throws with this stacktrace

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Drawing.SafeNativeMethods.Gdip.IntGdipDeleteGraphics(HandleRef graphics)
   at System.Drawing.Graphics.Dispose(Boolean disposing)
   at System.Drawing.Graphics.Dispose()
   at Lesnikowski.Barcode.BaseBarcode.Render()
   at Lesnikowski.Barcode.BaseBarcode.Save(Stream stream, ImageType imageType)

I've read that memory fragmentation is often caused by pinning memory, but this is !gchandles output

Handles:
    Strong Handles:       154
    Pinned Handles:       23
    Ref Count Handles:    2
    Weak Long Handles:    1794
    Weak Short Handles:   74
    SizedRef Handles:     17
    Dependent Handles:    1

I don't know what else I can check. Do we have memory fragmentation problem? Can you point me in some direction?

Edit: I attach performance counters collected during load test. Weird because it's showing a lot of pinned objects but !gchandles didn't show them.

red line   - user load
green line - bytes in all heaps
blue line  - pinned objects

Edit2: Added Fragmented blocks larger than 0.5 MB: output from !dumpheap -stat

Answer 1

You should use !address -summary to get an idea of the virtual memory usage in this process. Even though there seems to be some heap fragmentation, there is very likely some additional heavy memory consumer in your process. It could be the Win32 heap, it could be thread stacks, it could be assemblies you are loading dynamically, and so on.

For each type of leak you would then have to follow a slightly different approach. For Win32 heap inspection, you should use variations of the !heap command - !heap -stat, !heap -s -h 0. For assembly loading issues, you should look at the loader heap with !eeheap -loader and then inspect the various AppDomains you have with !dumpdomain to see which assemblies you are loading. These are just some examples - you will have to provide more details about your situation.