subq $40 %rsp crash with AS but GCC not

Question

I meet a strange phenomenon, I record the code in following. My test bed is x86_64 and gcc is 5.3.0 When I reserve some space in the stack for local value, sometimes it would crash.

                   | AS and LD | gcc    |
--------------------------------------------
 40 bytes in stack |  crash    | ok     |
--------------------------------------------
 32 bytes in stack |   ok      | crash  |
--------------------------------------------


.section .data

fmt:
    .ascii "0x%lx\n\0"

.section .text
.global _start

_start:
    subq $40, %rsp   # subq $32, %rsp is OK
                     # I want to reserve some place for local value.

    movq $8, %rsi
    movq $fmt, %rdi
    call printf      #print something

    addq $40, %rsp
    movq $1, %rax
    int $0x80

    as tsp.s -o tsp.o 
    ld -lc -I /lib64/ld-linux-x86-64.so.2 tsp.o -o tsp
    ./tsp
    Segmentation fault (core dumped)

This time I use gcc to compile and link. It is ok, when I reserve 40 bytes in the stack. It crash, when I reserve 32 bytes in the stack.

.section .data

fmt:
    .ascii "0x%lx\n\0"

.section .text
.global main 

main:
    subq $40, %rsp   # if subq $32, %rsp, it would crash.

    movq $8, %rsi
    movq $fmt, %rdi
    call printf

    addq $40, %rsp   
    movq $1, %rax
    int $0x80

    gcc tsp.s -o tsp
    ./tsp
    0x8

Answer 1

When I tested your code printf crashed when accessing xmm registers. There are two reasons for it. When you let gcc do the compilation and linking it will actually have additional code before main. That code will correctly align the stack and then call main.

Since main was called like a normal function the stack will be aligned at 8 mod 16 because of the call instruction, but when calling a function the stack has to be correctly aligned (0 mod 16). The reason for the alignment requirement is because of xmm registers (among others).

Now, why did printf touch xmm registers in the first place? Because you called printf incorrectly. The ABI for amd64 says:

When a function taking variable-arguments is called, %rax must be set to the total number of floating point parameters passed to the function in SSE registers.

Your rax probably has some non-zero value in it.

So, two things to fix your problems. xorl %eax, %eax to zero %rax before the call to printf. And be aware of how you have been called and how to align the stack. If you've been called as a normal function, you need to subtract 8+n*16 (n can be 0) from your stack pointer before doing a call. If you've been called as an entry point to be safe you need to properly align your stack pointer because I'm not sure if the kernel always guarantees that your stack pointer will be aligned.