How to get pointer to outcoming buffer using asm?

Question

I have to write function in asm, and i have prototype in C

void fdct(float *in, float *out, unsigned int n);

Where:

• in: pointer to incoming data buffer
• out: pointer to outcoming data buffer
• n: amount of data matrices.

Function doesn't return anything, but works with array of outcoming data and must change it according to result.

As I understand, in cdecl stack will look like:

esp: ret
esp+4: *in - address
esp+8: *out - address 
esp+12: value of n

I got, how to work in asm with incoming buffer, but I don't understand how to return new address of outcoming buffer - just putting new address into esp+8 isn't resultive, it doesn't change value of *out. How can I manage this problem?

Answer 1

That is not how this should work.

Your assembler function does not allocate a float (or an array of float) and return it in out.

Instead, the caller should allocate such an array (e.g. using malloc(), calloc() -- or whatever functions his or her language provides -- or simply as local variable on the stack) and pass the address of the first element in out. Your function just fills the array with suitable float values, probably up to n elements.

Example:

float a[16] = ... ;  /* fill it up with suitable values */
float b[16] = { 0 }; /* receives result values */

fdct(a, b, 16);