Questions about SetWindowsHookEx() and hooking

Question

Here is a little background information. I'm working on replacing a dll that has been used in a dll injection technique via the AppInit_DLLs registry entry. Its purpose was to be present in every process and set hooks into the GDI32.dll to gather information about printing. This is kind of a funky way to get what we want. The .dll itself is over 10 years old (written in Visual Studio 97) and we'd like to replace it with something a little less invasive than an injected dll.

It appears SetWindowsHookEx() maybe what we are looking for. I've been having some trouble with it, but I've also had some discussions with co-workers about whether or not this tree is worth barking up. Here are some questions that we could not determine:

1. When we hook a routine out of a dll, for example StartDoc() from GDI32.dll, do we really get a notification every time any other process uses that rotuine out of that dll? This is kind of the functionality we were getting with our injected .dll and we need the same functionality going forward.

2. When the hook is triggered, does the hook handling procedure run in the process space of the process that initiated the actual call, or in the process space of the process that set up the hook? My opinion is that it has to run in the process space of the process that called the routine. For example, if a program calls StartDoc() from GDI32.dll, it will have the hook handling procedure code "injected" into its space and executed. Otherwise, there would have to be some inter-process communication that automatically gets set up between the calling process and the process that set up the hook, and I just don't see that as being the case. Also, its kind of necessary that this hook handling routine run in the process space of the calling process since one of the things it needs to know is the name of that calling process, and I'm not sure on how to get that information if it wasn't actually running in that process.

3. If the hook handling routine is written using the .NET managed environment, will it break when getting hooked into a process not utilizing the .NET managed environment? We'd really like to move away from C++ here and use C#, but what would happen if we our hook gets called from a process that isn't managed? As stated before, I think that our hook handling procedure will run in the process that originally called the routine that was hooked. But if this is true, than I would think that we'd run into trouble if this process was not using the .NET run time environment, but the incoming hooked handling code is.

Answer 1

1. Yes.

2. Generally, it's the former: it executes in the context of the process whose event it is hooking.

   After a successful call to SetWindowsHookEx, the operating system automatically injects the hook DLL (the one that contains the callback function) into the address space of all target processes that meet the requirements for the specified hook type. (Of course, the hooking code is not necessarily injected immediately.)

   The exception to this general rule are the low-level keyboard and mouse hooks (WH_LL_KEYBOARD and WH_LL_MOUSE). Since those hook types are not injected into the client processes, the callback is called in the same thread that originally called SetWindowsHookEx.

3. That last point is important to keep in mind to answer your third question. Because the low-level keyboard and mouse hooks are the only two global hooks that do not require DLL injection, they are also the only two types of hooks that can be written in managed .NET code.

   For the other hook types, your concerns expressed in the question are precisely correct. You would need to write these hook DLLs in C or C++. Of course, the rest of your application's pieces could still be written in a managed language. The only thing that matters is the hook DLL.

You might consider looking into either Microsoft Detours or EasyHook.