Gracefully close application before installing through Wix managed custom action

Question

In my WiX installer I want to gracefully close the application that is about to be updated if it is still running. I do not want to prompt the user to close and I do not want to kill the process. I need to have a chance to perform some clean-up etc. before closing the application.

The app is a WinForms app that runs in the system tray. The main form has a title, let's say "mainwindow" for example, but is hidden and has ShowInTaskbar = false.

By playing around with some different tester apps trying Process.Kill() Process.CloseMainWindow() FindWindow, SendMessage, PostMessage etc. I have found that the best way for me to do this is to use PostMessage

var hWnd = FindWindow(null, "mainwindowtitle");
PostMessage(hWnd, WM_CLOSE, IntPtr.Zero, IntPtr.Zero);

That way I can override OnFormClosing and perform any cleanup I need. This works fine from a tester app I threw together. The problem is that it does not work when run in the WiX Installer. I have a c# Custom Action CA.dll and the installer definitely calls the custom action - I can see that from the msiexec logs and if I change the custom action code to just Process.Kill() it does stop the app correctly. However when it runs with the PostMessage code the application does not close and OnFormClosing never gets called.

Here is my CustomAction code

        private const int WM_CLOSE = 0x0010;

    [DllImport("user32.dll", SetLastError = true)]
    static extern IntPtr FindWindow(string lpClassName, string lpWindowName);

    [DllImport("user32.dll", SetLastError = true)]
    static extern bool PostMessage(IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);

    [CustomAction]
    public static ActionResult CloseApplicationGracefully(Session session)
    {
        session.Log("Starting the CloseApplicationGracefully Custom Action - attempting to stop DUC.");

        var hWnd = FindWindow(null, "mainwindowtitle");

        session.Log("Window handle found: " + hWnd);

        bool result = PostMessage(hWnd, WM_CLOSE, IntPtr.Zero, IntPtr.Zero);

        session.Log("Result of calling app to close: " + result);

        if (result)
        {
            return ActionResult.Success;
        }
        return ActionResult.Failure;
    }

here is the wx setup code

<Binary Id="WixCustomAction.dll"
        SourceFile="$(var.WixCustomAction.TargetDir)$(var.WixCustomAction.TargetName).CA.dll" />
<CustomAction Id="WixCustomAction"
              BinaryKey="WixCustomAction.dll"
              DllEntry="CloseDeploymentUpdater" />
<InstallExecuteSequence>
  <Custom Action="WixCustomAction" After="FindRelatedProducts"></Custom>
</InstallExecuteSequence>

I have tried calling this custom action in different sequences but no luck... The code works from a tester app and the custom action works when I use Process.Kill but the code does not work when put in the custom action - must be the sequence of events?

EDIT

Using the WixCloseApplications CA as suggested below in a answer results in the following log entries

WixCloseApplications:  App: DUC.exe found running, 1 processes, attempting to send close message.
WixCloseApplications:  Sending close message to process id 0x1978
WixCloseApplications:  Result 0x12
WixCloseApplications:  Sending close message to process id 0x1978
WixCloseApplications:  Result 0x0
WixCloseApplications:  Sending close message to process id 0x1978
WixCloseApplications:  Result 0x578
WixCloseApplications:  Sending close message to process id 0x1978
WixCloseApplications:  Result 0x0
.
.
.
MSI (s) (C8!D4) [15:00:47:985]: PROPERTY CHANGE: Adding WixCloseApplicationsDeferred property. Its value is 'DUC.exe5'.
MSI (s) (C8!D4) [15:00:48:000]: Doing action: WixCloseApplicationsDeferred
.
.
Action 15:00:48: WixCloseApplicationsDeferred. 
Action start 15:00:48: WixCloseApplicationsDeferred.
.
.
Action ended 15:00:48: WixCloseApplicationsDeferred. Return value 1.
Action ended 15:00:48: WixCloseApplications. Return value 1.

Answer 1

If understand the CloseApp CustomAction from Wix correctly you should enumerate over all windows in a process.

http://wix.codeplex.com/SourceControl/changeset/view/175af30efe78#src%2fca%2fwixca%2fdll%2fCloseApps.cpp

So you will need to implement an EnumWindows(EnumCallBack, HANDLE processToClose) And in the EnumCallBack you actually PostMessage WM_CLOSE for each window.