No symbol traces from stripped binaries in google breakpad

Question

From documentation, google breakpad is:

a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed

For the proof of the above quote we will try it with this minimal c++17 sample:

#include <thread>
#include <filesystem>

#include <client/linux/handler/exception_handler.h>

namespace breakpad = google_breakpad;

static bool DumpCallback(const breakpad::MinidumpDescriptor& md,
                         void* context,
                         bool success) {
    (void)md;
    (void)context;
    return success;
}

static void fault(unsigned after) {
    std::this_thread::sleep_for(std::chrono::seconds{after});
    delete reinterpret_cast<std::string*>(0xFEE1DEAD);
}

int32_t main(int argc, char** argv) {
    (void)argc;
    (void)argv;

    auto pwd = std::filesystem::current_path();
    const auto dumpDir = pwd.string() + "/dumps";
    std::filesystem::create_directory(dumpDir);
    breakpad::MinidumpDescriptor md(dumpDir);
    new google_breakpad::ExceptionHandler(
        md,
        /* FilterCallback */ nullptr,
        DumpCallback,
        /* callback_context */ nullptr,
        true,
        -1
    );

    fault(1U);

    return EXIT_SUCCESS;
}

In a normal Debug build, it's what is expected to be, so if we try to run it, and process the generated minidump file (with help of main utilities such as dump_syms and minidump_stackwalk) the result is a nice symbol trace:

Operating system: Linux
                  0.0.0 Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
CPU: amd64
     family 6 model 58 stepping 9
     1 CPU

GPU: UNKNOWN

Crash reason:  SIGSEGV /SEGV_MAPERR
Crash address: 0xfee1dead
Process uptime: not available

Thread 0 (crashed)
 0  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_data() const [basic_string.h : 176 + 0x4]
    rax = 0x00000000fee1dead   rdx = 0x00007ffc12803cc0
    rcx = 0x00007fc328087bc1   rbx = 0x000055cdac272940
    rsi = 0x00007ffc12803cc0   rdi = 0x00000000fee1dead
    rbp = 0x00007ffc12803c80   rsp = 0x00007ffc12803c80
     r8 = 0x0000000000000000    r9 = 0x000055cdac276bd8
    r10 = 0x0000000000000000   r11 = 0x0000000000000246
    r12 = 0x000055cdabc1af40   r13 = 0x00007ffc12803f80
    r14 = 0x0000000000000000   r15 = 0x0000000000000000
    rip = 0x000055cdabc1bc20
    Found by: given as instruction pointer in context
 1  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_is_local() const [basic_string.h : 211 + 0xc]
    rbx = 0x000055cdac272940   rbp = 0x00007ffc12803cb0
    rsp = 0x00007ffc12803c90   r12 = 0x000055cdabc1af40
    r13 = 0x00007ffc12803f80   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x000055cdabc1bf0b
    Found by: call frame info
 2  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() [basic_string.h : 220 + 0xc]
    rbx = 0x000055cdac272940   rbp = 0x00007ffc12803cd0
    rsp = 0x00007ffc12803cc0   r12 = 0x000055cdabc1af40
    r13 = 0x00007ffc12803f80   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x000055cdabc1bc3e
    Found by: call frame info
 3  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() [basic_string.h : 657 + 0xc]
    rbx = 0x000055cdac272940   rbp = 0x00007ffc12803cf0
    rsp = 0x00007ffc12803ce0   r12 = 0x000055cdabc1af40
    r13 = 0x00007ffc12803f80   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x000055cdabc1b5b6
    Found by: call frame info
 4  core!fault [main.cpp : 18 + 0xa]
    rbx = 0x000055cdac272940   rbp = 0x00007ffc12803d20
    rsp = 0x00007ffc12803d00   r12 = 0x000055cdabc1af40
    r13 = 0x00007ffc12803f80   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x000055cdabc1b070
    Found by: call frame info
 5  core!main [main.cpp : 38 + 0xa]
    rbx = 0x000055cdac272940   rbp = 0x00007ffc12803ea0
    rsp = 0x00007ffc12803d30   r12 = 0x000055cdabc1af40
    r13 = 0x00007ffc12803f80   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x000055cdabc1b182
    Found by: call frame info
 6  libc.so.6 + 0x2409b
    rbx = 0x0000000000000000   rbp = 0x000055cdabc43ec0
    rsp = 0x00007ffc12803eb0   r12 = 0x000055cdabc1af40
    r13 = 0x00007ffc12803f80   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007fc327ed909b
    Found by: call frame info
 7  core!fault [main.cpp : 19 + 0x3]
    rsp = 0x00007ffc12803ed0   rip = 0x000055cdabc1b082
    Found by: stack scanning
 8  core!google_breakpad::FileID::ElfFileIdentifier(google_breakpad::wasteful_vector<unsigned char>&) [file_id.cc : 158 + 0x10]
    rsp = 0x00007ffc12803ee8   rip = 0x000055cdabc1af40
    Found by: stack scanning
 9  ld-linux-x86-64.so.2 + 0xf476
    rsp = 0x00007ffc12803f40   rip = 0x00007fc3283ef476
    Found by: stack scanning
10  core!google_breakpad::FileID::ElfFileIdentifier(google_breakpad::wasteful_vector<unsigned char>&) [file_id.cc : 158 + 0x10]
    rsp = 0x00007ffc12803f58   rip = 0x000055cdabc1af40
    Found by: stack scanning

Loaded modules:
0x55cdabc08000 - 0x55cdabc43fff  core  ???  (main)
0x7fc327eb5000 - 0x7fc32801efff  libc.so.6  ???  (WARNING: No symbols, libc.so.6, A8A9B91823C5CFE5E5B5D946D605D0920)
0x7fc328076000 - 0x7fc32808afff  libpthread.so.0  ???
0x7fc328097000 - 0x7fc3280aafff  libgcc_s.so.1  ???
0x7fc3280b1000 - 0x7fc32815cfff  libm.so.6  ???
0x7fc328234000 - 0x7fc328368fff  libstdc++.so.6  ???
0x7fc3283e0000 - 0x7fc3283fefff  ld-linux-x86-64.so.2  ???  (WARNING: No symbols, ld-linux-x86-64.so.2, 7BFD5DF2BE95A34B86FD71080ACCAE8C0)
0x7ffc12932000 - 0x7ffc12933fff  linux-gate.so  ???

But in the deployment, the typical case is to have two release versions, one as normal Release, and the other as RelWithDebInfo(same release but with debugging symbols). so if you try the exact same routine as above but instead with the dump file generated from the normal release binary(deployed to the client) and symbols from RelWithDebInfo binary you get the following trace warn you about symbols for the main binary:

Operating system: Linux
                  0.0.0 Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
CPU: amd64
     family 6 model 58 stepping 9
     1 CPU

GPU: UNKNOWN

Crash reason:  SIGSEGV /SEGV_MAPERR
Crash address: 0xfee1dead
Process uptime: not available

Thread 0 (crashed)
 0  core + 0xf26a
    rax = 0x0000000000000000   rdx = 0x00005604fdc66209
    rcx = 0x00007f77d9bcfbc1   rbx = 0x00007fffbc001120
    rsi = 0x00007fffbc0010b0   rdi = 0x00007fffbc0010b0
    rbp = 0x00007fffbc0011e0   rsp = 0x00007fffbc0010a0
     r8 = 0x0000000000000000    r9 = 0x00005604ff997be8
    r10 = 0x0000000000000000   r11 = 0x0000000000000246
    r12 = 0x00007fffbc0010e0   r13 = 0x00007fffbc0010c0
    r14 = 0x00007fffbc0010b0   r15 = 0x00005604ff993940
    rip = 0x00005604fdc6626a
    Found by: given as instruction pointer in context
 1  core + 0x36995
    rbp = 0x00007fffbc0011e0   rsp = 0x00007fffbc0011b0
    rip = 0x00005604fdc8d995
    Found by: stack scanning
 2  ld-linux-x86-64.so.2 + 0xf530
    rbp = 0x00007fffbc0011e0   rsp = 0x00007fffbc0011b8
    rip = 0x00007f77d9f37530
    Found by: stack scanning
 3  core + 0xf520
    rbp = 0x00007fffbc0011e0   rsp = 0x00007fffbc0011c8
    rip = 0x00005604fdc66520
    Found by: stack scanning
 4  core + 0x36950
    rsp = 0x00007fffbc0011e8   rip = 0x00005604fdc8d950
    Found by: stack scanning
 5  libc.so.6 + 0x2409b
    rsp = 0x00007fffbc0011f0   rip = 0x00007f77d9a2109b
    Found by: stack scanning
 6  core + 0xef90
    rsp = 0x00007fffbc001210   rip = 0x00005604fdc65f90
    Found by: stack scanning
 7  core + 0xf520
    rsp = 0x00007fffbc001228   rip = 0x00005604fdc66520
    Found by: stack scanning
 8  ld-linux-x86-64.so.2 + 0xf476
    rsp = 0x00007fffbc001280   rip = 0x00007f77d9f37476
    Found by: stack scanning
 9  core + 0xf520
    rsp = 0x00007fffbc001298   rip = 0x00005604fdc66520
    Found by: stack scanning
10  core + 0xf54a
    rsp = 0x00007fffbc0012b0   rip = 0x00005604fdc6654a
    Found by: stack scanning

Loaded modules:
0x5604fdc57000 - 0x5604fdc8dfff  core  ???  (main)  (WARNING: No symbols, core, C33015040F685CBAD56AEFBFD7109D4C0)
0x7f77d99fd000 - 0x7f77d9b66fff  libc.so.6  ???  (WARNING: No symbols, libc.so.6, A8A9B91823C5CFE5E5B5D946D605D0920)
0x7f77d9bbe000 - 0x7f77d9bd2fff  libpthread.so.0  ???
0x7f77d9bdf000 - 0x7f77d9bf2fff  libgcc_s.so.1  ???
0x7f77d9bf9000 - 0x7f77d9ca4fff  libm.so.6  ???
0x7f77d9d7c000 - 0x7f77d9eb0fff  libstdc++.so.6  ???
0x7f77d9f28000 - 0x7f77d9f46fff  ld-linux-x86-64.so.2  ???  (WARNING: No symbols, ld-linux-x86-64.so.2, 7BFD5DF2BE95A34B86FD71080ACCAE8C0)
0x7fffbc1ad000 - 0x7fffbc1aefff  linux-gate.so  ???

Is there anything else we need to consider?

Update May 21'01

The actual script we use for symbols generation:

#!/bin/bash

#
# e.g ./dump.sh ./exec $PWD/dumps
#

set -e
set -u

DBG_INFO=$(realpath ${1})
DUMPS_DIR=$(realpath ${2:-$PWD/dumps})
DUMP_SYMS=${3:-~/WorkSpace/libraries/breakpad/src/tools/linux/dump_syms/dump_syms}
STAK_WALK=${4:-~/WorkSpace/libraries/breakpad/src/processor/minidump_stackwalk}

#
# Generate debug symbols
#
base=$(basename $DBG_INFO)
$DUMP_SYMS $DBG_INFO > $DUMPS_DIR/$base.sym

#
# Create dump dir structure
#
list=($(head -n1 $DUMPS_DIR/$base.sym))
hash=${list[3]}
mkdir -p $DUMPS_DIR/symbols/$base/$hash
mv $DUMPS_DIR/$base.sym $DUMPS_DIR/symbols/$base/$hash

#
# Produce stack trace
#
RED='\033[0;36m'
NC='\033[0m' # No Color
tree $DUMPS_DIR
for dmp in $DUMPS_DIR/*.dmp ; do
    filename=$(basename -- "${dmp}")
    filename="${filename%.*}"
    echo -e "generating stack trace for -> ${RED}${dmp}${NC}"
    $STAK_WALK ${dmp} $DUMPS_DIR/symbols > $DUMPS_DIR/${filename}.txt 2>/dev/null
done

Answer 1

minidump has no symbol.

breakpad docs told the developer will create symbol files for Breakpad’s use using the included dump_syms or symupload tools, or another suitable tool, and place the symbol files where the processor’s symbol supplier will be able to locate them.

So use dump_syms to generate the symbol file,for example, './dump_sys ./my-binary > my.sym'.

Then before using minidump_stackwalk, put the symbol file in the directory specified on the first line of the file. See symbolstore.py in the Mozilla repository. https://github.com/MozillaReality/symbolgenerator

Last,Generate a trace stack,use minidump_stackwalk,this command generates the trace stack and prints it.

Answer 2

You're shipping a binary that doesn't match your debugging symbols.

I think the symbol generation itself is fine. However, you're shipping a Release build while your symbols were generated for your RelWithDebinfo build.

If you take the binary from your RelWithDebInfo build and strip that (i.e. use strip on it), you should end up with a stripped binary that you can ship and that matches your extracted debugging symbols.

Answer 3

I faced the same problem when trying to use minidump_stackwalk for the release version of application. I'm using qmake and solved the problem using qmake's CONFIG flags force_debug_info and separate_debug_info. After build there would appear appName.debug executable in the build folder). To dump symbols just copy appName.debug file somewhere(e.g. create subfolder debug in build folder) and rename it to appName. Finally, call dump_syms build_folder/debug/appName > appName.sym and you'll be able to minidump_stackwalk over release version of application.

PS: Think there is a similar solution for CMake based and other projects.