I developed a "keyboard layout DLL" for Windows but I cannot have it loaded and got no error message. I wonder how to troubleshoot this.
There is some sample code for keyboard layouts in https://github.com/microsoft/Windows-driver-samples/tree/main/input/layout
The idea is simple: Create a DLL which exports a function named KbdLayerDescriptor. This function only returns the address of a structure which describes the keyboard. This data structure contains pointers to other structures, etc.
Simple in principle. However, the keyboard layout is never loaded, without error message. There must be something missing but I do not know what.
Here are my investigations.
Step 1: I started from the sample kbdfr in the above repo and made a few modifications to make sure I can characterize the new keyboard. I copied the generated DLL in C:\Windows\System32. When trying to select the keyboard in the System Settings, the new keyboard is not listed. There was no further explanation in the sample repo.
Step 2: Searching the registry, I found that all keyboards layouts are referenced in [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layouts]. So, I added an entry for my new keyboard. This time, I can see it in the system settings. However, when I select it, the keyboard is not active. The previous layout (the French one) remains active. Even when I remove the French layout and only keep my new layout in the settings, the keyboard is still the French one.
Step 3: I browsed all processes and their mapped DLL's, searching which one had the keyboard layouts. I found two instances of ctfmon.exe which had KBDFR.DLL mapped (this is the standard French keyboard). If I switch to a US layout in the System Settings, I can see that ctfmon.exe immediately unmaps KBDFR.DLL and maps KBDUS.DLL. So, inspecting ctfmon.exe is a valid clue about which keyboard is active. If I select my new keyboard layout, ctfmon.exe keeps KBDFR.DLL and ignores my DLL. This explains why the new layout is not active.
Step 4: I copied a valid working keyboard DLL from a Windows virtual machine running on a Mac. Because of the differences in keyboards, the "Parallels Desktop" hypervisor installs Mac keyboard layouts in the virtual machine. These layouts are not present in Windows by default and they work correctly (I use them in the VM on Mac). They were installed in the VM through the installer of the "Parallels Tools". I do not know exactly what this installer does, in addition to copying the DLL's in System32. I copied this DLL and the associated registry entry from the VM on Mac to the target Windows 11 PC (not on a Mac). In the System Settings, I can see the Parallels keyboard for Mac. However, when I select it, the layout does not work and ctfmon.exe keeps KBDFR.DLL. Same symptom as my layout DLL.
So, the problem does not seem to be in my keyboard DLL but in the way it is installed on the system, because the Parallels layout which works on a VM where it was properly installed no longer works on a system where it was just copied (with its registry entries).
The classical regsvr32 is useless here since these KBD DLL's are not servers. Running regsvr32 on any of them raises an error about absent server entry point (seems normal).
Step 5: I tried to overwrite an existing DLL (KBDGR.DLL, the German layout) with my DLL to see if it would work when selecting the German layout. However, I get a "Permission Denied" error when trying to overwrite KBDGR.DLL, even using an Administrator account. There is probably something with the "trusted installer" here.
EDIT(1): Step 6: I applied the same ACL on my DLL as other keyboards without better result (Get-All kbdus.dll | Set-All kbdmine.dll in Powershell). My DLL is now owned by TrustedInstaller. But it is still not mapped.
EDIT(2): Step 7: In the KbdLayerDescriptor function of my DLL, before returning the data structure address, as a test, it creates a temporary text file and writes the path of the exe of the process which invokes it. The file is created when I select the language in the system settings and the process is C:\Windows\ImmersiveControlPanel\SystemSettings.exe. So, the System Settings app loads the DLL, calls its entry point and unloads it. However, the keyboard does not appear in the language bar and is never loaded by ctfmon.exe.
Additional notes:
- Of course, I searched online but only found zillions of obvious and useless hints on Windows keyboards...
- My code and a few tools are available here: https://github.com/lelegard/winkbdlayouts . Given the way a keyboard layout DLL works, it is easy to load one and inspect all the returned structures. So, I developed a tool to reverse-engineer an existing keyboard layout DLL and generate the corresponding C source file for an exact copy. Thus, it is easy to modify any available DLL. But, this does not solve the installation issue.
So, my question is: what are the necessary steps to make sure a personally-developed keyboard layout DLL is properly used?
I finally managed to find the solution by myself. I documented the explanations here: https://github.com/lelegard/winkbdlayouts#declaring-a-keyboard-layout-on-windows