Opening a C# File with ProcessStartInfo opens wrong Visual studio Instance

Question

I have a C# Code Analysis Application that opens C# using ProcessStartInfo And Process Start. It almost works.

I want to use two instance of VS to perform the analysis: one to Debug the Analysis App, and another to have the File that is specified in its appropriate Solution.

Currently the C# File always opens in the Analysis App Debug Session, not the Analyzed App Solution. How can I specify which Instance to use upon Process.Start(ProcessStartInfo)?

private void OpenFileExecute(string file)
{
    ProcessStartInfo psi = new ProcessStartInfo(file);   
    psi.UseShellExecute = true;
    Process.Start(psi);
}

Answer 1

With the standard open method you'll not be able to control which running instance will be used. Under the hood it uses DDE and it is undetermined which instance will handle DDE requests.

The same goes for when you use Activator.GetObject.

Luckily, in ole32.dll we can call GetRunningObjectTable and from there enumerate over all instances to find all registered OLE Servers, for each instance and that includes all Visual Studio processes.

Once we found one you can obtain an instance to its OLE Automation interface, EnvDTE and use that to inspect in-depth if this is the right instance to interact with, and if yes, execute any command we're interested in, for example loading a file.

private void OpenFileExecute(string file)
{
    IRunningObjectTable ir;
    // get all OLE/COM Automation servers
    GetRunningObjectTable(0, out ir);
    if (ir != null)
    {
        IEnumMoniker moniker;
        // Get an enumerator to iterate over them
        ir.EnumRunning(out moniker);
        if (moniker != null)
        {
            moniker.Reset();
            IMoniker[] results = new IMoniker[1];
            // iterate
            while (moniker.Next(1, results, IntPtr.Zero) == 0)
            {
                // we need a Bind Context
                IBindCtx bindCtx;
                CreateBindCtx(0, out bindCtx);
                // what is the name of the OLE/COM Server
                string objName;
                results[0].GetDisplayName(bindCtx, null, out objName);
                // what have we got ...
                Trace.WriteLine(objName);
                // I test with VS2010 instances, 
                // but feel free to get rid of the .10
                if (objName.StartsWith("!VisualStudio.DTE.10"))
                {
                    object dteObj; //
                    ir.GetObject(results[0], out dteObj);
                    // now we have an OLE automation interface 
                    // to the correct instance
                    DTE dteTry = (DTE)dteObj;
                    // determine if this is indeed
                    // the one you need
                    // my naive approach is to check if
                    // there is a Document loaded
                    if (dteTry.Documents.Count == 1)
                    {
                        dteTry.ExecuteCommand(
                            "File.OpenFile",
                            String.Format("\"{0}\"", file));
                    }
                }
                bindCtx.ReleaseBoundObjects();
            }
        }
    }
}

[DllImport("ole32.dll")]
public static extern int GetRunningObjectTable(int reserved,
                            out IRunningObjectTable prot);

[DllImport("ole32.dll")]
public static extern int CreateBindCtx(int reserved,
                                out IBindCtx ppbc);

Background reading: How to use Marshal.getActiveObject() to get 2 instance of of a running process that has two processes open