I have a question that keeps bothering me for the last week.
In Windows debugger there is the !heap -s command that outputs the virtual memory's heap status and calculates the external fragmentation using the formula :
External fragmentation = 1 - (larget free block / total free size)
Is there a similar method in linux, that outputs those statistics needed to calculate the effect?
Long story now:
I have a C application that keeps allocating and deallocating space of different sizes, using malloc and free, each allocation has different life span.
The platform I am using is Lubuntu, so ptmalloc2 algorithm is the default.
I am aware that those allocations are served in the virtual user space heap(except those that are larger than 128Kb, where the allocator uses mmap), and are mapped to physical pages when actually accessed .
The majority of the allocations is of size < 80 bytes, so they are served from FastBins.
Using Valgrind and Massif I can get the internal fragmentation, since it reports the extra bytes used for each allocation.
However, my main concern is how to figure out the external fragmentation.
I am aware of the /proc/[pid]/smaps heap size and the pmap-d[pid] anon statistics, but I find it difficult to interpret them in terms of external fragmentation.
I am also aware of LD_PRELOAD , and I can dynamically connect the /lib/i386-linux-gnu/libmemusage.so. This library outputs the heap total, peak and the distribution of the requested allocation sizes.
I know that __malloc__hook is deprecated now, and I do not really want to rely on implementation specific statistics like malloc_stats() and mallinfo() . However, If you have any suggestions using those two please let me know.
I can tell that the external fragmentation problem, is when a request can not be satisfied, because there is no contiguous space in the heap, but the requested total size is scattered all around that area.
I still have not figured out, how to get the statistics needed so I can calculate this effect. For example different formulas stating that I have to capture the live_memory or get the total_free_pages, or get the size of the largest_free_block.
How can I have a function to "traverse" through the heap and gather those statistics?
Thank you everyone in advance.
I believe that this will depend on the allocator you are using. That is, you would probably need a different strategy for whichever malloc (et al) and free implementation you are using. If the implementation doesn't offer the information you seek as an extension, you will probably have to read its source-code and type your own logic to examine the state of allocations.
I believe that the mapping of pages to swap-space and physical RAM is at a lower level and so won't particularly help you in your goal. The malloc (et al) and free implementation might or might not care about those lower-level details.
If you are certain that you are using ptmalloc2, are you able to find its source-code?