how to use Dtrace to check malloc on Solaris 10?

Question

I meet a troublesome bug about memory usage, so I want to use Dtrace to check malloc and free on Solaris 10.

I use the following command

dtrace -l | grep malloc  

The output is:

7000        fbt              unix                       prom_malloc entry
7001        fbt              unix                       prom_malloc return
7141        fbt           genunix                       cacl_malloc entry
7142        fbt           genunix                       cacl_malloc return
12319        fbt           genunix                   rmallocmap_wait entry
12320        fbt           genunix                   rmallocmap_wait return
13078        fbt           genunix                      rmalloc_wait entry
13079        fbt           genunix                      rmalloc_wait return
13526        fbt           genunix                        rmallocmap entry
13527        fbt           genunix                        rmallocmap return
16846        fbt           genunix                           rmalloc entry
16847        fbt           genunix                           rmalloc return
25931        fbt             tmpfs                      tmp_memalloc entry
25932        fbt             tmpfs                      tmp_memalloc return  

It seems there is no malloc.

I have checked Solaris Internal, and found the malloc calls sbrk. So I use the following command:

dtrace -l | grep sbrk

But there is nothing found.

So how can I use Dtrace to check malloc on Solaris 10?

Answer 1

There are various tools that already implement the logic required to identify memory leaks under Solaris,

• libumem & mdb (UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1 then mdb's ::findleaks)
• dbx (check -leaks)

Should you still want to go the dtrace way, you need to trace the process you suspect to leak memory using the pid provider. You searched the kernel probes with dtrace -l and found nothing but this is expected as the kernel does not implement malloc or brk. They are userland functions located in the C standard library.

This script will trace every malloc and free calls by a program:

dtrace -qn '
pid$target:libc:malloc:entry {
        self->size=arg0;
}
pid$target:libc:malloc:return /self->size/ {
        printf("malloc(%d)=%p\n",self->size,arg1);
        self->size=0;
}
pid$target:libc:free:entry {
        printf("free(%p)\n",arg0);
}
' -c program_to_trace

For more in-depth examples, have a look to http://ewaldertl.blogspot.fr/2010/09/debugging-memory-leaks-with-dtrace-and.html and http://www.joyent.com/blog/bruning-questions-debugging

Answer 2

For example, in order to trace all malloc calls with dtrace, by printing each allocation you would write (to a file named trace-malloc.d in this example) a script like below:

#!/usr/sbin/dtrace -s

    pid$1::malloc:entry
    {
        self->trace = 1;
        self->size = arg0;
    }

    pid$1::malloc:return
    /self->trace == 1/
    {
        /* log the memory allocation */
        printf("<__%i;%Y;%d;malloc;0x%x;%d;\n", i++, walltimestamp, tid, arg1, self->size);
        ustack(50);
        printf("__>\n\n");

        self->trace = 0;
        self->size = 0;
    }

and then call it by passing the process id of the process you want to trace, for example:

./trace-malloc.d 12345

In complex programs there are very frequent memory allocations and de-allocations so I've written a small program to help me identify memory leaks with dtrace. Each memory operation with malloc / calloc / realloc and free is traced and then an analysis program reads and processes all the traces and points out suspected memory leaks, also using various heuristics to point out strongly suspected memory leaks. If you are interested you can check it out here:

https://github.com/ppissias/DTLeakAnalyzer.