Unusable pstack output for core dump on Solaris

626 views Asked by At

Output of pstack applied to a coredump should contain a stack backtrace for every thread. But in some cases the output contains truncated backtraces for each thread, only one entry per thread. Here is an example, showing an excerpt of the pstack output:

core 'core' of 8714:    ./rds_rdprod.solsparc64
-----------------  lwp# 1  --------------------------------
 ffffffff7c2d7bd0 ???????? (ffffffff7ffff5b0, ffffffff7ffff570, 0, 4, 16cf50, ffffffff7c10d240)
-----------------  lwp# 2  --------------------------------
 ffffffff7c2d7bd0 ???????? (100bf12b0, 100bf12d8, 0, 4, 16cf50, ffffffff7c10e9c0)
-----------------  lwp# 3  --------------------------------
 ffffffff7c2d7bd0 ???????? (100bf12b0, 100bf12d8, 0, 4, 16cf50, ffffffff7c10e9c0)
-----------------  lwp# 4  --------------------------------
 ffffffff7c2d7bd0 ???????? (100bf12b0, 100bf12d8, 0, 4, 16cf50, ffffffff7c10e9c0)
-----------------  lwp# 5  --------------------------------
 ffffffff7c2d7bd0 ???????? (100bf12b0, 100bf12d8, 0, 4, 16cf50, ffffffff7c10e9c0)
... [omitting a number of threads]
-----------------  lwp# 52  --------------------------------
 ffffffff7c2d7bd0 ???????? (105286af0, 105286b18, 0, 4, 16cf50, ffffffff7c10f600)
-----------------  lwp# 53  --------------------------------
 0000000100196e80 _ZN9CAtomMap43AddEP6CAtom4S1_ (5ad245980, 19f6751f8, 19f6751f8, 19f6751f8, 25bf97af0, 28) + 60
-----------------  lwp# 54  --------------------------------
 00000001001a1de8 _ZN12CBasicSearch6ExistsER18ICacheItemIdentity (19b43ed78, ffffffff6faff3b8, 6, 0, 0, 2aa932388) + 28
-----------------  lwp# 55  --------------------------------
 ffffffff7d300838 ???????? (0, 287812ec8, 5382b3098, 45dde3558, 188ce9053, 15faa9d38)
... [omitting a number of threads]

As you can see, there is only one line per LWP, where we normally get a full backtrace. And you may also notice that a few threads have a valid mangled C++ symbol, and this is probably because the processor was executing appication code at the time of the core dump. But why do all backtraces get truncated?

The last time this happened the faulting thread threw a SIGABRT, according to pflags applied to the same core dump.

There is enough disk space for the core dump, and no issue with permissions to write the core dump. Machine is a Sparc with 32 cores, SunOS 5.10 Generic_144488-17. Application is built with gcc 4.7.2.

0

There are 0 answers