Allocating memory on the caller's stack

Question

I am writing in C++ using MSVC++ 2012 and my code is targeted for an x86 platform. I have a situation where it would be advantageous to write a function that (amongst other things) can allocate some memory on the stack of the CALLING function. Its not my intent in this post to debate the wisdom of doing this, but only to consider the technical feasibility.

My implementation plan is to write my function as a naked function with custom prolog code in inline assembly. In the prolog I would first determine how much memory I need and then move the return address, parameters, and this pointer down the stack by that amount. Finally I would then adjust the stack pointer by that amount. If I am not mistaken this would create an area on the stack of the calling function.

Does anyone see any holes in that plan?

Answer 1

One obvious disadvantage is that inline assembly is not supported for x64, so you would be limiting your portability in the future.

Another obvious disadvantage is that the calling function will be expecting the stack pointer in a certain place relative to their locals, which it no longer is. I believe that the compiler's generated code would not cope with such a situation. A simple example of this dissonance would be that the generated code would not know how much stack space to pop off.

The only way I have observed to do what you want to do is to use a default parameter. You can do e.g.

int f(void* p = alloca(55)) {
}

Technically since the default parameter is evaluated in the calling function, this would allocate memory from the calling function's stack. However as you may observe the core issue here is calculating how much space you need.

Answer 2

There were a lot of good points made by several posters. I will summerize here.

The technique I described WILL work but ONLY if the calling function was defined with CDECL calling convention AND the linker switch disabling stack frame pointers is not enabled. The calling function must be CDECL, not STDCALL or THISCALL because STDCALL and THISCALL would not correctly restore the stack pointer in the calling function's epilog since those conventions subtract from the current ESP rather than restoring the original value. They would subtract the wrong amount, not accounting for the added space, and the stack would thus be corrupted. The linker switch disabling stack frame pointer must not be active because if it is the calling function will reference parameters as offsets from ESP rather than EBP and thus the offsets will not be correct.

In addition to the above limitations the concsequences of using this technique would be loss of x64 portability and unpredictable impact to performance optimizations.

In view of the limitations I have elected not to continue with this technique.