Why do Native Messaging Host processes spawned by Firefox have an odd DLL search order?

257 views Asked by At

I'm trying to create a browser extension using native messaging, where the messaging host is basically a simple launcher for another (3rd party) native executable.

The 3rd party application relies on various DLLs, some of which need to be loaded from the same directory the executable is in. This works perfectly on Chrome (and chromium based Edge as well), however I ran into issues when using Firefox where the wrong DLL would be loaded.

After monitoring, it appears that instead of looking in the executable directory first, messaging host processes (including all their child processes) started by Firefox have the following DLL search order:

  1. System directory
  2. 16-bit System directory
  3. Windows directory
  4. Directory of the executable
  5. Working Directory
  6. Directories in path

I was able to reproduce this exact behavior easily in my own launcher application as well by using LoadLibrary without an explicit path, so it is not just the 3rd party app.

Looking through MS documents related to DLL search order, I could not find anything that could cause this, so I am quite lost as to what the issue is, and whether I could go around it somehow.

Already tried the following with no success:

  • Using cmd.exe as an intermediary to start my process. Tried this both when Firefox starts the native host and when my launcher starts the 3rd party app, neither seem to make a difference.
  • Calling SetDllDirectory in the launcher. This just replaces Working Directory, but stays after the System folders in the order. Calling it with null does not change anything.
  • Redirection by adding a .local file/directory. This does force the app to load the DLLs from its own directory, but breaks the application for other reasons I have no control over. (I believe there are full-path dll loads in it which also get overridden this way)

These would only be workarounds either way for what seems to be caused by Firefox, as like I said, when launching the app manually, or using Chrome/Edge everything works as expected.

Any clue why this is happening, or a way I could force the application to use the default DLL search order when it's launched by a Firefox messaging host?

0

There are 0 answers