Calling API from a x86 DLL loaded into a x64 application

Question

This is an experiment to exploit the availability of a 32-bit "compatibility" mode selector in GDT in Windows x64 which has the value of 0x23.

I 've successfully loaded a x86 DLL into the address space of a x64 application. This is done by first taking all imports of the x86 dll and saving it into a XML, then using a modified MemoryModule to load the DLL from memory and patch the Import Address Table with those pointers.

Then, in the x64 host I jump to a compatibility code segment:

push 0x23    
xor rcx,rcx    
mov ecx,Start32  
push rcx
retf 

Start32 is a 32-bit entry point that would call a configured function:

USE32
push eax
push dword [WhatToPass]
mov eax, [WhatToCall]
call eax
pop eax

Then go back to 64-bit

USE64
db 0eah
ret_64:
dd 0
dw 0x33
nop

This works, as long as the configured exp1 function does practically nothing. The concept is fully discussed in my CodeProject article and the source is on GitHub.

It doesn't work when run in Visual Studio, but it does work when run in WinDbg; WinDbg shows a 'x86' running mode when switching.

Now the ambitious project is to call the API from the x86 module. This fails even with a simple MessageBeep(0);. I don't of course expect it at this point to go fully to the function but it doesn't even enter; I get an invalid execution instruction on the call. The address is valid.

What could be wrong? Yes this is too much perhaps but I'm still curious.

Patching the IAT works correctly. Sample code:

void PatchIAT(HINSTANCE h,std::vector<FANDP>* CustomLoading = 0)
{
    PCHAR codeBase = (PCHAR)h;
    XML3::XML x(xResult);
    PIMAGE_NT_HEADERS32           ntheaders = (PIMAGE_NT_HEADERS32)(PCHAR(h) + PIMAGE_DOS_HEADER(h)->e_lfanew);
    PIMAGE_SECTION_HEADER       pSech = IMAGE_FIRST_SECTION(ntheaders);//Pointer to first section header
    DWORD ulsize = 0;
    PIMAGE_IMPORT_DESCRIPTOR importDesc = (PIMAGE_IMPORT_DESCRIPTOR)ImageDirectoryEntryToData(h, TRUE, IMAGE_DIRECTORY_ENTRY_IMPORT, &ulsize);
    if (!importDesc)
        return;

    for (; importDesc && importDesc->Name; importDesc++) {

        PSTR pszModName = (PSTR)((PBYTE)h + importDesc->Name);
        if (!pszModName)
            break;

        XML3::XMLElement* module = ... ; // find the XML entry
        if (!module)
            continue;

        DWORD* thunkRef;
        DWORD* funcRef = 0;

        if (importDesc->OriginalFirstThunk) {
            thunkRef = (DWORD*)(codeBase + importDesc->OriginalFirstThunk);
            funcRef = (DWORD*)(codeBase + importDesc->FirstThunk);
        }
        else {
            // no hint table
            thunkRef = (DWORD*)(codeBase + importDesc->FirstThunk);
            funcRef = (DWORD *)(codeBase + importDesc->FirstThunk);
        }

        DWORD V = 0;
        for (; *thunkRef; thunkRef++, funcRef++) {

            DWORD* wr = (DWORD*)funcRef;

            if (IMAGE_SNAP_BY_ORDINAL(*thunkRef)) {
//                *funcRef = module->getProcAddress(handle, (LPCSTR)IMAGE_ORDINAL(*thunkRef), module->userdata);

                const char* fe = (LPCSTR)IMAGE_ORDINAL(*thunkRef);
                 // find V from xml 
                ...

            }
            else {
                PIMAGE_IMPORT_BY_NAME thunkData = (PIMAGE_IMPORT_BY_NAME)(codeBase + (*thunkRef));
//                *wr = module->getProcAddress(handle, (LPCSTR)&thunkData->Name, module->userdata);
             ... // Find V from XML
            }


            // Patch it now...
            DWORD dwOldProtect = 0;
            if (VirtualProtect((LPVOID)wr, 4, PAGE_READWRITE, &dwOldProtect))
            {
                memcpy((void*)wr, &V, 4);
                VirtualProtect((LPVOID)wr, 4, dwOldProtect, &dwOldProtect);
            }
            VirtualProtect((LPVOID)V, 4, PAGE_EXECUTE_READWRITE, &dwOldProtect);
        }

    }




}