I am working on a program using GNU Scientific Library. It gives an "illegal instruction (core dump)" after solving a nonlinear equation in half. See below. What is the usual cause of such "illegal instruction" error, and how could I debug in such situations?
...
iter 0: A = 1.0000, lambda = 1.0000, b = 0.0000, cond(J) = 6.0000, |f(x)| = 101.0200
iter 1: A = 3.5110, lambda = -12.8820, b = 1.2364, cond(J) = 92.8216, |f(x)| = -nan
iter 2: A = 3.5110, lambda = -12.8820, b = 1.2364, cond(J) = nan, |f(x)| = -nan
iter 3: A = 3.5110, lambda = -12.8820, b = 1.2364, cond(J) = nan, |f(x)| = -nan
iter 4: A = 3.5110, lambda = -12.8820, b = 1.2364, cond(J) = nan, |f(x)| = -nan
Illegal instruction (core dumped)
With gdb, I got a bit of additional info.
Program received signal SIGILL, Illegal instruction.
0x00000000004d1030 in nielsen_reject (nu=<optimized out>, mu=<optimized out>) at nielsen.c:98
98 *nu <<= 1;
(gdb) p nu
$1 = <optimized out>
(gdb) x/i $pc
=> 0x4d1030 <trust_iterate+8912>: ud2
Above, nielsen.c98 looks like this
...
static int
nielsen_reject(double * mu, long * nu)
{
*mu *= (double) *nu;
/* nu := 2*nu */
*nu <<= 1;
return GSL_SUCCESS;
}
CPU is x86_64 according to uname -m. OS is Ubuntu 16.04 VirtualMachine on a Mac(host). GCC version is 5.4.
gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
The code is compiled with gcc, with Address Sanitizer and Undefined Sanitizer switched on (-fsanitize=address,undefined).
It seems that the compiler produces a "ud2" instruction when sanitizers are switched on. I would like to know whether this bug is a compiler bug, sanitizer bug, or code bug?