Program has returned control to the operating system in Assembly 8086

Question

When I run my program the assembler says:

PROGRAM HAS RETURNED CONTROL TO THE OPERATING SYSTEM

when it reaches the RET instruction. The code looks like:

twoMash PROC
    push bp
    mov bp, sp
    sub sp, 2

    NOT ax
    ADD ax,1

    mov sp,bp
    ret
twoMash ENDP

main is defined this way:

main:
    mov ax,100b
    push ax
    call twoMash

Answer 1

The first instruction is push bp, so value on top of stack is now value from bp.

Then you do some more things, including manipulation of sp (pointer to top of stack), but just before ret it points back to the old bp value.

ret will pop value from stack, and set ip (instruction pointer) to that value. Under normal circumstances it is expected to have on top of stack the address of next instruction to execute (put there usually by call instructions, which does the counter-action of ret, push address of next instruction after call on stack, and then sets ip to the argument value from call instruction).

But in your code the ip is set to old bp value, which very likely points somewhere into stack memory (or "worse"), so the CPU will next try to execute data bytes as code, and the behaviour is unexpected (return to OS would be actually quite nice end result, such mistakes usually end in crash of application, or even loss of data).

To fix, add pop bp ahead of ret (after restoring sp value by mov sp,bp).

Whenever you manipulate stack, either explicitly by push/pop or add/sub sp, or implicitly by call/ret, make sure you end with correct sp value in every code path before using stack further. Ie. usually every push needs it's pairing pop and every call should return by ret, unless you are experienced enough to break such rules and adjust stack to correct state by different means.

---

BTW, it is questionable, whether there actually is return address at the stack at your entry point (it's not clear from your question, if you have some other code call-ing this, or it is entry point of your program).

If this is like DOS executable, and it was entry point, then you should end your program with int 21h, 4Ch OS service call.

Answer 2

This is not an error, the program has finished successfully, but if this message bothers you, you can use the HLT at the end of the code. Then the program will stop successfully and show the following text:
the emulator is halted