How can I break on UBSan reports in gdb and continue?

Question

Recent versions of GCC and Clang feature Undefined Behavior Sanitizer (UBSan) which is a compile flag (-fsanitize=undefined) that adds runtime instrumentation code. On errors, a warning such as this one is shown:

packet-ber.c:1917:23: runtime error: left shift of 54645397829836991 by 8 places cannot be represented in type 'long int'

Now I would like to debug this and get a debug break on said line. For Address Sanitizer (ASAN) there is ASAN_OPTIONS=abort_on_error=1 which results in a fatal error that is catchable. The only UBSan option that seems usable is UBSAN_OPTIONS=print_stacktrace=1 which results in a call trace dump for reports. This however does not allow me to inspect the local variables and then continue the program. Use of -fsanitize-undefined-trap-on-error therefore not possible.

How should I break in gdb on UBSan reports? While break __sanitizer::SharedPrintfCode seems to work, the name looks quite internal.

Answer 1

While breaking on the detection functions (as described by @Mark Plotnick and @Iwillnotexist Idonotexist) is one option, a better approach is breaking on the functions that report these issues after detection. This approach is also used for ASAN where one would break on __asan_report_error.

Summary: You can stop on an ubsan report via a breakpoint on __ubsan::ScopedReport::~ScopedReport or __ubsan::Diag::~Diag. These are private implementation details which might change in the future though. Tested with GCC 4.9, 5.1.0, 5.2.0 and Clang 3.3, 3.4, 3.6.2.

For GCC 4.9.2 from ppa:ubuntu-toolchain-r/test, you need libubsan0-dbg to make the above breakpoints available. Ubuntu 14.04 with Clang 3.3 and 3.4 do not support the __ubsan::ScopedReport::~ScopedReport breakpoints, so you can only break before printing the message using __ubsan::Diag::~Diag.

Example buggy source code and a gdb session:

$ cat undef.c
int main(void) { return 1 << 1000; }
$ clang --version
clang version 3.6.2 (tags/RELEASE_362/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
$ clang -w -fsanitize=undefined undef.c -g
$ gdb -q -ex break\ __ubsan::ScopedReport::~ScopedReport -ex r ./a.out 
Reading symbols from ./a.out...done.
Breakpoint 1 at 0x428fb0
Starting program: ./a.out 
undef.c:1:27: runtime error: shift exponent 1000 is too large for 32-bit type 'int'

Breakpoint 1, 0x0000000000428fb0 in __ubsan::ScopedReport::~ScopedReport() ()
(gdb) bt
#0  0x0000000000428fb0 in __ubsan::ScopedReport::~ScopedReport() ()
#1  0x000000000042affb in handleShiftOutOfBoundsImpl(__ubsan::ShiftOutOfBoundsData*, unsigned long, unsigned long, __ubsan::ReportOptions) ()
#2  0x000000000042a952 in __ubsan_handle_shift_out_of_bounds ()
#3  0x000000000042d057 in main () at undef.c:1

---

Detailled analysis follows. Note that both ASAN and ubsan both originate from a LLVM project, compiler-rt. This is used by Clang and ends up in GCC as well. Links in the following sections point to the compiler-rt project code, release 3.6.

ASAN has made its internal __asan_report_error part of the documented public interface. This function gets called whenever a violation is detected, its flow continues in lib/asan/asan_report.c:938:

void __asan_report_error(uptr pc, uptr bp, uptr sp, uptr addr, int is_write,
                         uptr access_size) {
  // Determine the error type.
  const char *bug_descr = "unknown-crash";
  ...

  ReportData report = { pc, sp, bp, addr, (bool)is_write, access_size,
                        bug_descr };
  ScopedInErrorReport in_report(&report);

  Decorator d;
  Printf("%s", d.Warning());
  Report("ERROR: AddressSanitizer: %s on address "
             "%p at pc %p bp %p sp %p\n",
             bug_descr, (void*)addr, pc, bp, sp);
  Printf("%s", d.EndWarning());

  u32 curr_tid = GetCurrentTidOrInvalid();
  char tname[128];
  Printf("%s%s of size %zu at %p thread T%d%s%s\n",
         d.Access(),
         access_size ? (is_write ? "WRITE" : "READ") : "ACCESS",
         access_size, (void*)addr, curr_tid,
         ThreadNameWithParenthesis(curr_tid, tname, sizeof(tname)),
         d.EndAccess());

  GET_STACK_TRACE_FATAL(pc, bp);
  stack.Print();

  DescribeAddress(addr, access_size);
  ReportErrorSummary(bug_descr, &stack);
  PrintShadowMemoryForAddress(addr);
}

ubsan on the other hand has no public interface, but its current implementation is also much simpler and limited (less options). On errors, a stacktrace can be printed when the UBSAN_OPTIONS=print_stacktrace=1 environment variable is set. Thus, by searching the source code for print_stacktrace, one finds function MaybePrintStackTrace which is called though the ScopedReport destructor:

ScopedReport::~ScopedReport() {
  MaybePrintStackTrace(Opts.pc, Opts.bp);
  MaybeReportErrorSummary(SummaryLoc);
  CommonSanitizerReportMutex.Unlock();
  if (Opts.DieAfterReport || flags()->halt_on_error)
    Die();
}

As you can see, there is a method to kill the program on errors, but unfortunately there is no builtin mechanism to trigger a debugger trap. Let's find a suitable breakpoint then.

The GDB command info functions <function name> made it possible to identify MaybePrintStackTrace as function on which a breakpoint can be set. Execution of info functions ScopedReport::~ScopedReport gave another function: __ubsan::ScopedReport::~ScopedReport. If none of these functions seem available (even with debugging symbols installed), you can try info functions ubsan or info functions sanitizer to get all (UndefinedBehavior)Sanitizer-related functions.

Answer 2

A breakpoint set at __asan_report_error is not hit for me and the program simply exists after printing the diagnostics without the debugger triggering. __asan::ReportGenericError before printing the diagnostics and __sanitizer::Die after printing the diagnostics do get hit as described in the asan wiki.

Answer 3

As @Mark Plotnick points out, the way to do so is to breakpoint at UBSan's handlers.

UBSan has a number of handlers, or magic function entry points, that are called for undefined behaviour. The compiler instruments code by injecting checks as appropriate; If the check code detects UB, it calls these handlers. They all start with __ubsan_handle_ and are defined in libsanitizer/ubsan/ubsan_handlers.h. Here's a link to GCC's copy of ubsan_handlers.h.

Here's the relevant bits of the UBSan header (breakpoint on any of these):

#define UNRECOVERABLE(checkname, ...) \
  extern "C" SANITIZER_INTERFACE_ATTRIBUTE NORETURN \
    void __ubsan_handle_ ## checkname( __VA_ARGS__ );

#define RECOVERABLE(checkname, ...) \
  extern "C" SANITIZER_INTERFACE_ATTRIBUTE \
    void __ubsan_handle_ ## checkname( __VA_ARGS__ ); \
  extern "C" SANITIZER_INTERFACE_ATTRIBUTE NORETURN \
    void __ubsan_handle_ ## checkname ## _abort( __VA_ARGS__ );

/// \brief Handle a runtime type check failure, caused by either a misaligned
/// pointer, a null pointer, or a pointer to insufficient storage for the
/// type.
RECOVERABLE(type_mismatch, TypeMismatchData *Data, ValueHandle Pointer)

/// \brief Handle an integer addition overflow.
RECOVERABLE(add_overflow, OverflowData *Data, ValueHandle LHS, ValueHandle RHS)

/// \brief Handle an integer subtraction overflow.
RECOVERABLE(sub_overflow, OverflowData *Data, ValueHandle LHS, ValueHandle RHS)

/// \brief Handle an integer multiplication overflow.
RECOVERABLE(mul_overflow, OverflowData *Data, ValueHandle LHS, ValueHandle RHS)

/// \brief Handle a signed integer overflow for a unary negate operator.
RECOVERABLE(negate_overflow, OverflowData *Data, ValueHandle OldVal)

/// \brief Handle an INT_MIN/-1 overflow or division by zero.
RECOVERABLE(divrem_overflow, OverflowData *Data,
            ValueHandle LHS, ValueHandle RHS)

/// \brief Handle a shift where the RHS is out of bounds or a left shift where
/// the LHS is negative or overflows.
RECOVERABLE(shift_out_of_bounds, ShiftOutOfBoundsData *Data,
            ValueHandle LHS, ValueHandle RHS)

/// \brief Handle an array index out of bounds error.
RECOVERABLE(out_of_bounds, OutOfBoundsData *Data, ValueHandle Index)

/// \brief Handle a __builtin_unreachable which is reached.
UNRECOVERABLE(builtin_unreachable, UnreachableData *Data)
/// \brief Handle reaching the end of a value-returning function.
UNRECOVERABLE(missing_return, UnreachableData *Data)

/// \brief Handle a VLA with a non-positive bound.
RECOVERABLE(vla_bound_not_positive, VLABoundData *Data, ValueHandle Bound)

/// \brief Handle overflow in a conversion to or from a floating-point type.
RECOVERABLE(float_cast_overflow, FloatCastOverflowData *Data, ValueHandle From)

/// \brief Handle a load of an invalid value for the type.
RECOVERABLE(load_invalid_value, InvalidValueData *Data, ValueHandle Val)

RECOVERABLE(function_type_mismatch,
            FunctionTypeMismatchData *Data,
            ValueHandle Val)

/// \brief Handle returning null from function with returns_nonnull attribute.
RECOVERABLE(nonnull_return, NonNullReturnData *Data)

/// \brief Handle passing null pointer to function with nonnull attribute.
RECOVERABLE(nonnull_arg, NonNullArgData *Data)

---

ASan is even easier. If you look in libsanitizer/include/sanitizer/asan_interface.h, which you should browse here, you can read a dead giveaway of a comment:

  // This is an internal function that is called to report an error.
  // However it is still a part of the interface because users may want to
  // set a breakpoint on this function in a debugger.
  void __asan_report_error(void *pc, void *bp, void *sp,
                           void *addr, int is_write, size_t access_size);

Numerous other functions in this header are explicitly commented as having been made public so as to be callable from a debugger.

I definitely advise you to explore other headers of libsanitizer/include/sanitizer here. There are numerous goodies to be had there.

---

Breakpoints for UBSan and ASan can be added as follows:

(gdb) rbreak ^__ubsan_handle_ __asan_report_error
(gdb) commands
(gdb) finish
(gdb) end

This will breakpoint on the handlers, and finish immediately afterwards. This allows the report to be printed, but the debugger gets control right after it gets printed.