GCC access memory above stack top

Question

I have C function that does some SSE calculations. When I compile it with GCC I get next code

/* Start of function */
mov    (%rdi),%rax
movslq %ecx,%rcx
...
mov    0x8(%rdi),%rax
pxor   %xmm12,%xmm3
movaps %xmm0,-0x28(%rsp)
movaps %xmm6,%xmm1
...
movaps 0x50(%rax,%rcx,1),%xmm2
movaps 0x60(%rax,%rcx,1),%xmm15
pxor   %xmm2,%xmm0
pxor   %xmm2,%xmm6
movaps -0x28(%rsp),%xmm2
pxor   %xmm15,%xmm5
pxor   %xmm15,%xmm2
movaps 0x70(%rax,%rcx,1),%xmm15
movaps (%rax,%rcx,1),%xmm11
mov    0x10(%rdi),%rax
movaps %xmm15,-0x18(%rsp)
pxor   %xmm11,%xmm4
pxor   %xmm12,%xmm11
pxor   %xmm15,%xmm12

Look at movaps instructions - it's access memory over stack top:

movaps %xmm15,-0x18(%rsp)

Isn't it an access to undefined memory? And why GCC generated such incorrect code?

Answer 1

There's no such thing as "undefined memory" at the assembly level. gcc is free to emit code that accesses the stack in whatever way it sees fit, so long as the behaviour is as expected.

My guess as to why this is happening is that this is a leaf function for which adjusting the stack pointer is fruitless. You could try to verify that by inspecting the assembly for any call instructions. (You could also inspect the C source, but inlining may make that a bit less reliable.)

This kind of trickery is explicitly allowed by the ABI of certain platforms, including x86-64. From the AMD64 ABI documentation:

The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers. Therefore, functions may use this area for temporary data that is not needed across function calls. In particular, leaf functions may use this area for their entire stack frame, rather than adjusting the stack pointer in the prologue and epilogue. This area is known as the red zone.

This blog post might make for interesting reading on the subject.