Problem at detouring ShellExecuteEx, Any Idea?

Question

I have a process X that I inject my DLL into to detour some functions, and make some memory patches. I need to detour ShellExecuteEx(), because this process runs other processes, and then I need to inject my DLL into the child processes, too.

My detoured function seems to be called fine, and when I call the original function, it returns TRUE. But then the process where my DLL is injected closes a few seconds later when this is called (no injection to child process yet, since I haven't coded it). Any idea why?

static BOOL(WINAPI *t_ShellExecuteExW)(SHELLEXECUTEINFOW *pExecInfo) = ShellExecuteExW;

BOOL d_ShellExecuteExW(SHELLEXECUTEINFOW *pExecInfo)
{
    BOOL result;

    printf("ShellExecuteEx %ls \n", pExecInfo->lpFile);

    try
    {
        result = t_ShellExecuteExW(pExecInfo);
    }
    catch (const std::exception& e)
    {
        printf("Exception %s", e.what());
    }

    if (result)
        printf("Result True");
    else
        printf("Result False");

    return result;
}

void makeHooks()
{
    HMODULE module = LIBpatching_loadLibrary("shell32.dll", 10000);
    FARPROC address;

    if ((address = GetProcAddress(module, "ShellExecuteExW")) != nullptr)
    {
        printf("[shell32] [ShellExecuteExW] Address found\n");

        LIBpatching_hookFunction((PBYTE)address, (PBYTE)d_ShellExecuteExW);
    }
}

Answer 1

If you want to hook child processes, you should detour CreateProcess() instead of ShellExecuteEx(), which will just call CreateProcess() internally when it needs to create a new process.

In any case, the signatue of your d_ShellExecuteExW() hook is missing the required __stdcall calling convention, which is wrapped by the WINAPI macro that is present in your t_ShellExecuteExW type.

Change this:

BOOL d_ShellExecuteExW(SHELLEXECUTEINFOW *pExecInfo)`

To this:

BOOL WINAPI d_ShellExecuteExW(SHELLEXECUTEINFOW *pExecInfo)