I have several services developed with topshelf that are installed using an MSI created in InstallShield (using basic MSI). We install them using a custom action passing the install parameter. When it comes to uninstall, we use a custom action passing the uninstall parameter (we also have custom actions that start and stop as appropriate). All of this is working, but the user is presented with a dialog asking them to shut down those services otherwise a reboot may be required otherwise. If, however, the user elects to continue anyway, our custom action runs, stopping and uninstalling the service.
I have been unable to move the stop action high enough in the execute sequence to avoid the dialog without causing a not in transaction error.
Is there some way I can get rid of that dialog?
To stop the service with a custom action you'd need to have it before InstallValidate, which means it has to be immediate, so it gets tricky if the install is canceled or fails because then you leave the system with a stopped service.
The real solution to this is that you're supposed to install services with the ServiceInstall element, and stop/start/delete them with the ServiceControl element. At uninstall, if Windows Installer sees that you're going to stop the service (with ServiceControl) then it doesn't do a FilesInUse dialog.
So the short answer is that you may not need a custom action at all. The ServiceControl element can be used to stop and start all services (not just ones being installed with ServiceInstall) so if you do a ServiceControl element to stop your service at uninstall it might solve the problem. In the long term I'd get away from command line installs and use ServiceInstall.