Trying to perform a factorial in assembly but causing stack overflow

Question

Ok, so I am trying to test out a factorial program from my college book, and when I type it correctly as shown, it gives me a stack overflow error.

It happens at the line push ebp. Can anyone tell me what is wrong?

.code
main PROC
    mov ebp,0
    push 3
    call Factorial
    call WriteDec
    call Crlf
    exit
main ENDP

Factorial PROC

    push ebp
    mov ebp,esp
    cmp eax,0
    ja L1
    mov eax,1
    jmp L2

    L1:
        dec eax
        push eax
        call Factorial

    ReturnFact:
        mov ebx,[ebp+8]
        mul ebx

    L2:
        pop ebp
        ret 4

Factorial ENDP

Answer 1

It is just me or anyone else think that you missed a

mov eax, [ebp+8]

at the start of the function (after the prologue)? You are not getting the argument from the stack before comparing it with 0.

Answer 2

Perhaps the book had a stack going up (though, this is unusual). At the moment, it subtracts from ebp every time something is pushed, because the stack actually grows down. (This seems a little counterintuitive, but that's how its done in most computers)

So, what's happening, is that you're pushing 3 onto the stack at address 0, subtracting 4 from ebp, and having MAX UNSIGNED INT - 3 now in ebp. Which is out of the stack, so "stack overflow" or, accessing something not in your stack.

Answer 3

You probably meant to pop the 3 into eax, otherwise, eax is uninitialized.