this is problem. I developed a shell extension of Shell Context Menus. (use SharpShell, enter link description here)
I can register it by using regasm.exe with "/codebase" attribute. And without this attribute the shell extension doesn't work. But this attribute let me can't update the .dll file without reboot explorer. Even worse, can't delete .dll file after that unregistered and reboot PC (in windows 8).
So, did that have any way to solve this problem? By the way, if anyone can tell me what is "/codebase" actually doing? (I saw MSDN already, but totally can't understand.)
The only solution is to enumerate the handles created by explorer.exe and filter them by the target file, which is your own locked dll file. Then you can remove the handle.
It would require P/invoke if you will use C#, or handle.exe by Sysinternals if you will use any scripting language during installation. However, removing handles in the background of the application that has the handle is an unsupported way and may result with undefined consequences.