Using x86's ENTER instruction with a non-zero nesting level?

Question

Consider the x86 instruction ENTER. From Intel's instruction set reference.

Creates a stack frame for a procedure. The first operand (size operand) specifies the size of the stack frame (that is, the number of bytes of dynamic storage allocated on the stack for the procedure). The second operand (nesting level operand) gives the lexical nesting level (0 to 31) of the procedure. The nesting level determines the number of stack frame pointers that are copied into the “display area” of the new stack frame from the preceding frame. Both of these operands are immediate values.

I was wondering how the ENTER instruction works when a nonzero nesting level is passed as a second operand. In this case, according to Intel's manual, the processor should push additional frame pointers on the stack. Sounds simple, but I can't figure why it doesn't work as expected on sample programs.

I compiled the following example with FASM and debugged with OllyDbg.

format PE

section '.text' code readable executable
entry start
start:
    enter 16, 8
    push 0
    call ExitProcess
...

The stack frame emitted by the ENTER instruction is given below.

000CFF58   00000000 ; new esp
000CFF5C   00000000
000CFF60   00000000
000CFF64   00000000
000CFF68   000CFF88 ; value of new ebp
000CFF6C   7EFDE000 ; ?
000CFF70   000CFF94 ; value of old ebp
000CFF74   76AD338A ; ?
000CFF78   7EFDE000 ; ?
000CFF7C   000CFF94 ; value of old ebp
000CFF80   76AD338A ; ?
000CFF84   7EFDE000 ; ?
000CFF88   000CFF94 ; value of old ebp
000CFF8C   76AD338A ; old esp

The results are weird. Let's do the same thing with gcc.

> cat enter.s
.globl  _start

.text
_start:
    enter $16, $8
    movl $1, %eax
    movl $0, %ebx
    int $0x80

> gcc -m32 -g -c enter.s && ld -melf_i386 enter.o
> gdb a.out
...
(gdb) r
...  
Program received signal SIGSEGV, Segmentation fault.
_start () at enter.s:5
5           enter $16, $8

Errr, ok...

I'm probably misunderstanding how it supposed to work. My only guess here is that the ENTER instruction is handled by OS in some way, but this is almost certainly wrong.

Answer 1

The nested form of the enter instruction assumes you already have a valid frame pointer in ebp, which is not the case on linux at the start of the process. Presumably that is why you get a fault.

Try this instead:

_start:
    enter $0, $0        # set up initial stack frame
    push $1
    push $2             # simulate two previous frame pointers
    enter $16, $3       # nested proc level 3, esp should be 32 less
    movl (%ebp), %eax   # should be previous ebp
    movl -4(%ebp), %eax # should be 1
    movl -8(%ebp), %eax # should be 2
    movl $1, %eax
    movl $0, %ebx
    int $0x80

See also section 6.5.1 ENTER Instruction in the Intel® 64 and IA-32 Architectures Software Developer's Manual Volume 1: Basic Architecture