Why does GCC subtract the wrong value to the stack pointer when allocating a big array with no subsequent function calls?

Question

Really bizarre gcc quirk. Check this out:

main() { int a[100]; a[0]=1; }

produces this assembly:

   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   48 81 ec 18 01 00 00    sub    $0x118,%rsp
   b:   c7 85 70 fe ff ff 01    movl   $0x1,-0x190(%rbp)
  12:   00 00 00 
  15:   c9                      leaveq 
  16:   c3                      retq

The top of the stack is clearly 400, since its a 100 * 4 array. So when it writes to the first entry, it does rbp - 400 (line 'b'). Good. But why does it subtract 280 from the stack (line '4') pointer? Doesn't that point to the middle of the array?

If we add a function call afterward, gcc does the right thing:

b() {}
main() { int a[100]; a[0]=1; b(); }

produces this assembly:

0000000000000000 <b>:
   0:   55                      push   %rbp
   1:   48 89 e5                mov    %rsp,%rbp
   4:   c9                      leaveq 
   5:   c3                      retq   

0000000000000006 <main>:
   6:   55                      push   %rbp
   7:   48 89 e5                mov    %rsp,%rbp
   a:   48 81 ec 90 01 00 00    sub    $0x190,%rsp
  11:   c7 85 70 fe ff ff 01    movl   $0x1,-0x190(%rbp)
  18:   00 00 00 
  1b:   b8 00 00 00 00          mov    $0x0,%eax
  20:   e8 00 00 00 00          callq  25 <main+0x1f>
  25:   c9                      leaveq 
  26:   c3                      retq 

Here, it properly subtracts 400 (line 'a').

Why the change when you add a function call? Is gcc just lazy, and doesn't do it right because it doesn't matter? What's happening? Evidently this only happens when compiling for x86_64, but not for plain x86. Does this have something odd to do with x86_64's "redzone"? What's happening precisely?

Answer 1

Your guess is correct. It is a "red zone". The red zone is the space from rsp-128 to rsp, which may be used by a function for local variables and for temporary storage. This space is untouched by interrupt and exception handlers. Obviously, the red zone is destroyed by function calls, so if any function is called, no local variable can be in the red zone.

The red zone can only be used in 64 bit Linux, BSD and Mac. It is not available in kernel code.

It may be used to optimize for space, since with the red zone you can reference up to 512 bytes of local variables with short instructions, based on only rsp and ebp. Without the red zone only 384 bytes are available. All local variables outside of this limit are accessed with longer code or with additional registers.

For your example, using the red zone is not necessary, but gcc prefers to use it for all "leaf" functions. It is just easier to implement compiler this way.

Answer 2

The x86-64 ABI mandates a 'red zone' of 128 bytes beyond the stack pointer that can be used without modifying %rsp. In the first example, main() is a leaf function, so the compiler is optimizing the use of stack space - i.e., there are no function calls, so this region will not be overwritten.