MASM x86 fastcall function declaration... how?

Question

I'm trying to make a VisualStudio 2010 C program call a fastcall convention assembly routine.

This is the declaration in the C code:

extern boolean _fastcall InNonSuspendableCriticalRegion(DWORD);

This is the declaration in the assembly code:

      public @InNonSuspendableCriticalRegion@4

  @InNonSuspendableCriticalRegion@4 proc near ; fastcall
         <code>
  @InNonSuspendableCriticalRegion@4 endp

I get the following linker error:

   Assembling: C:\DMS\Domains\PARLANSE\Tools\RunTimeSystem\Source\PARLANSE0.asm
   1>RuntimeSupport.obj : error LNK2001: unresolved external symbol @InNonSuspendableCriticalRegion@4

I'm sure I'm doing something silly wrong, but I can't see it.

The MS documentation is pretty hard to figure out since it is so vague. I recall in the mists of the past that the assembler does some name mangling, too, so I'm not sure how the names I'm providing are getting mangled, if they are.

This is the most explicit reference on how do it, and I think I'm following it exactly; it says,

13. FASTCALL Caller and Callee Summary

The following sample illustrates the code generated in the calling function and in the called function to support __fastcall, the fastcall calling convention:

   int __fastcall FastFunc( int a, int b );

      calling function    called function
      -------------------------------------------
      mov edx, b          @FastFunc@8 PROC NEAR
      mov ecx, a                       .
      call @FastFunc@8                 .
       .                               .
       .                              RET 8
       .                  @FastFunc@8 ENDP

Any clues?

Thanks...

Answer 1

Try running dumpbin on both object files to dump the symbol table. That should show both the emitted and the referencing function name. That's usually helpful in diagnosing these kinds of problems.

Answer 2

This is the declaration in the assembly code:

public @InNonSuspendableCriticalRegion@4

@InNonSuspendableCriticalRegion@4 proc near ; fastcall
     ;; code
@InNonSuspendableCriticalRegion@4 endp

I think you are close. The only real insight in this answer is the assembler does not mangle by default, so if you call your procedure InNonSuspendableCriticalRegion, then that's how it is exported. To get a mangled name for a higher level language, you can usually do what Alexey suggest. However there is no FASTCALL, so just use an ALIAS.

Do something like this instead:

title Really cool assembly routines
public InNonSuspendableCriticalRegion

.486
.MODEL FLAT

.CODE
ALIGN   8

ALIAS <@InNonSuspendableCriticalRegion@4> = <InNonSuspendableCriticalRegion>

OPTION PROLOGUE:NONE
OPTION EPILOGUE:NONE

InNonSuspendableCriticalRegion proc
    ;; code
InNonSuspendableCriticalRegion endp

OPTION PROLOGUE:PrologueDef
OPTION EPILOGUE:EpilogueDef

;; ...

end

I think public @InNonSuspendableCriticalRegion@4 is incorrect, too (but its not the cause of the name decoration problems). You can omit it, or you can use the undecorated InNonSuspendableCriticalRegion name.

You can find the doc for ALIAS here. It is not very impressive.

You can see what symbols are exported with dumpbin. There is an example below from the Crypto++ project.

_IF_ you are getting an unwanted leading underscore in the fastcall name, like _@InNonSuspendableCriticalRegion@8, then ensure you don't have a OPTION LANGUAGE:C somewhere.

Finally, I thought SYSCALL was mostly equivalent to FASTCALL, but I could not get the assembler to accept it. Maybe it used to work in the old days, but I can't get MASM to accept it with a modern Visual Studio.

---

The Crypto++ project uses C++ code to interface with MASM code using fastcall (the code calls RDRAND or RDSEED, and it needs to run fast).

C++ code

The following declaration is used for both X86 and X64.

#if MASM_RDRAND_ASM_AVAILABLE
extern "C" void CRYPTOPP_FASTCALL MASM_RDRAND_GenerateBlock(byte*, size_t);
#endif

#if MASM_RDSEED_ASM_AVAILABLE
extern "C" void CRYPTOPP_FASTCALL MASM_RDSEED_GenerateBlock(byte*, size_t);
#endif

Dumpbin

Notice the symbol MASM_RDRAND_GenerateBlock is exported with External, and it is exported again with the alias as @MASM_RDRAND_GenerateBlock@8 and shows up with WeakExternal.

If you don't have the line symbols (labels), then be sure to assemble with /Zi. Its your debug information with MASM files.

C:\>dumpbin /SYMBOLS rdrand-x86.obj

Dump of file rdrand-x86.obj

File Type: COFF OBJECT

COFF SYMBOL TABLE
000 00DF520D ABS    notype       Static       | @comp.id
001 00000011 ABS    notype       Static       | @feat.00
002 00000000 SECT1  notype       Static       | .text$mn
    Section length   6F, #relocs    0, #linenums    0, checksum        0
004 00000000 SECT2  notype       Static       | .data
    Section length    0, #relocs    0, #linenums    0, checksum        0
006 00000000 SECT3  notype       Static       | .debug$S
    Section length  534, #relocs   26, #linenums    0, checksum        0
008 00000000 SECT4  notype       Static       | .debug$T
    Section length   3C, #relocs    0, #linenums    0, checksum        0
00A 00000000 SECT1  notype ()    External     | MASM_RDRAND_GenerateBlock
00B 00000000 UNDEF  notype       WeakExternal | @MASM_RDRAND_GenerateBlock@8
    Default index        A Alias record
00D 00000038 SECT1  notype ()    External     | MASM_RDSEED_GenerateBlock
00E 00000038 UNDEF  notype       WeakExternal | @MASM_RDSEED_GenerateBlock@8
    Default index        D Alias record
010 00000000 SECT1  notype       Static       | $$000000
011 00000000 SECT1  notype       Label        | GenerateBlock_Top
012 00000005 SECT1  notype       Label        | Call_RDRAND_EAX
013 0000000A SECT1  notype       Label        | RDRAND_succeeded
014 0000000F SECT1  notype       Label        | Full_Machine_Word
015 00000019 SECT1  notype       Label        | Partial_Machine_Word
016 0000002A SECT1  notype       Label        | Bit_1_Not_Set
017 00000034 SECT1  notype       Label        | Bit_0_Not_Set
018 00000034 SECT1  notype       Label        | GenerateBlock_Return
019 00000038 SECT1  notype       Label        | GenerateBlock_Top
01A 0000003D SECT1  notype       Label        | Call_RDSEED_EAX
01B 00000042 SECT1  notype       Label        | RDSEED_succeeded
01C 00000047 SECT1  notype       Label        | Full_Machine_Word
01D 00000051 SECT1  notype       Label        | Partial_Machine_Word
01E 00000062 SECT1  notype       Label        | Bit_1_Not_Set
01F 0000006C SECT1  notype       Label        | Bit_0_Not_Set
020 0000006C SECT1  notype       Label        | GenerateBlock_Return