SP (Stack Pointer) Anti-debug Trick - x86

Question

Listing 7.1 The Decryptor of the Cascade Virus

lea si, Start ; position to decrypt (dynamically set)

mov     sp, 0682    ; length of encrypted body (1666 bytes)

Decrypt:
xor     [si],si ; decryption key/counter 1
xor     [si],sp ; decryption key/counter 2
inc     si  ; increment one counter
dec     sp  ; decrement the other
jnz     Decrypt ; loop until all bytes are decrypted

Start:  ; Encrypted/Decrypted Virus Body 

Note that this decryptor has antidebug features because the SP (stack pointer) register is used as one of the decryption keys.

Can somebody explain why using the SP register is acting like an anti-debug feature? Correct me if I'm wrong but I don't think having a debugger running changes the stack layout...

Thanks in advance

Answer 1

My x86-fu is rusty, but I seem to recall most breakpoint debugging tools work by triggering a fault in the CPU and asserting themselves as a supervisor process - which would give you a new stack, and a correspondingly-altered stack pointer. Thus, stepping through that code would give you values of sp which are different to those the process would normally see had it not been trapped by a debugger.

Answer 2

Most debuggers expect [e]sp to be valid and pointing to a stack area. I guess it's possible that some debuggers crash if sp does not point to valid memory, but I don't know of any.

Answer 3

If the stack segment is equal to the data segment (is it .com or .exe virus? seems .com, because the DS is already equal to CS) then any use of stack (debugger or even interrupt) will modify the memory where ss:[sp] is pointing, and it will be pointing somewhere in the virus body (because it's used as counter).

Answer 4

Taking a breakpoint or an interrupt will "push data onto the stack", which will damage the data bytes in the area that the stack pointer references. Thus, if you put a breakpoint (INT n) in the code using a debugger, your very act of debugging (encountering the breakpoint) will destroy the data that this code is trying to decrypt.

This code might work under DOS if no interrupts happen; maybe they disable interrupts first. You can't realistically use this under Windows or Linux (its 16 bit code anyway).