C illegal instruction

Question

Below is a print out of a c program I wrote, a demonstration of me running it, and finally some information on my compiler.

➜  illegalInstructionDebug cat illegal.c
#include <stdio.h>
#include <stdlib.h>
#include <math.h>

void func(int* Z){
    Z[-11] = acos(2);
}

int main(){
    fflush(stdout);
    printf("");
    fflush(stdout);
    int X[3];
    int Z[3];
    for (int n=0;0!=0;);
    func(Z);
}
➜  illegalInstructionDebug gcc illegal.c; ./a.out
[1]    28836 illegal hardware instruction  ./a.out
➜  illegalInstructionDebug clang --version
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
➜  illegalInstructionDebug

I was writing a program an encountered an illegal instruction error, which is something I had never seen before, so I decided to attempt to find a minimal working example so I could figure out what distinguished it from a segfault or some other kind of error. Oddly, it seemed minor changes in the program would return it to giving a segfault instead of an illegal instruction error. Nonetheless, I have managed to reduce the program down substantially to a smaller working example. That being said, the program is still rather large for a minimal working example.

My questions are first why am I getting an illegal instruction error and second what is an illegal instruction error. Also if this error is specific to my machine that would interest me also. There are a lot of strange attributes to this program. For example, it seems the number -11 is required to cause the error.

Answer 1

This

void func(int *Z){
    Z[-11] = acos(2);
}

most likely happens to overwrite some code address in the stack. Most likely a return address. Since the stack grows down on x86-64, and you're writing to stuff to lower address, it would mean a return address that was placed on stack after space for Z was reserved, so I suppose this would hit the return address from func(). And more importantly, you would be overwriting half of the return address.

acos(2) is a domain error, and returns NaN, which converted to int on my GCC results in INT_MIN being written there... You should actually use the debugger and see the context where the crash happens, I would guess it is at an address that contains lots of fs in its hex representation.

And invalid instruction is raised when the bytes starting from RIP do not decode to a valid x86-64 instruction, or for other reasons.