Determine CPU trap that caused SIGSEGV under GDB?

Question

I've got a program that's SIGSEGV'ing in library code. Nothing is jumping out at me when looking at the statement that's causing the SIGSEGV (see below). But the code uses Intel's AES-NI, and I'm not that familiar with it.

I issued handle all in hopes of catching the trap that's causing the SIGSEGV, but the program still just crashes rather than telling me the trap.

How can I get GDB to display the CPU trap that's causing the SIGSEGV?

---

Program received signal SIGSEGV, Segmentation fault.
0x00000000004ddf0b in CryptoPP::AESNI_Dec_Block(long long __vector&, long long __vector const*, unsigned int) (block=..., subkeys=0x7fffffffdc60, rounds=0x0)
    at rijndael.cpp:1040
1040            block = _mm_aesdec_si128(block, subkeys[i+1]);
(gdb) p block
$1 = (__m128i &) @0x7fffffffcec0: {0x2e37c840668d6030, 0x431362358943e432}
(gdb) x/16b 0x7fffffffcec0
0x7fffffffcec0: 0x30    0x60    0x8d    0x66    0x40    0xc8    0x37    0x2e
0x7fffffffcec8: 0x32    0xe4    0x43    0x89    0x35    0x62    0x13    0x43

Answer 1

How can I get GDB to display the CPU trap that's causing the SIGSEGV

You can't: GDB doesn't get to see the trap, only the OS does.

What you can see is the instruction that caused the trap:

(gdb) x/i $pc

It's likely that the problem is alignment. I don't know what long long __vector is, but if it's not a 16-byte entity, then subkeys[i+1] is not going to be 16-byte aligned, which would be a problem for _mm_aesdec_si128, since it requires 16-byte alignment for both arguments.

Answer 2

These instructions are quite new (AVX). It could also possibly be that the CPU doesn't support the instruction, or that the OS isn't configured to allow them. I know one would normally expect SIGILL in such a case, but x86 can be surprising in the exceptions it generates, particularly if the OS has disabled use of an instruction that the CPU supports, SIGSEGV is quite common. (In case it's not clear from my tone, I'm just guessing here, just saying that it is an avenue of investigation that you might want to look into.)