Instantiating VisualStudio.VCProjectEngine.10.0 and later fails mysteriously, 8.0 doesn't

Question

Note: the question was originally about Python and win32com, but it turns out that the issue exists within the COM server, not the client-side. Therefore the information in the question and the answer are applicable to all languages that are usable as COM clients.

Maybe someone can help out. What I am facing is the following situation. I can instantiate VisualStudio.VCProjectEngine.8.0 just fine with my code. The relevant code:

import win32com
vcver = {
         'VisualStudio.VCProjectEngine.8.0' : ('{FBBF3C60-2428-11D7-8BF6-00B0D03DAA06}', 0, 8, 0),
         'VisualStudio.VCProjectEngine.10.0' : ('{0CD36BB6-D828-4DB9-91BF-AD493EE76B79}', 0, 10, 0),
         }
for ProgID, vc in vcver.iteritems():
    # Error also happens without the following line
    win32com.client.gencache.EnsureModule(vc[0], vc[1], vc[2], vc[3])
    o = win32com.client.gencache.GetClassForProgID(ProgID)()
    if o:
        try:
            # The following line causes the exception ...
            o.CreateProject('test')
            print "Success for %s" % ProgID
        except Exception as e:
            print str(e)
            raise

... gives this error:

Success for VisualStudio.VCProjectEngine.8.0
(-2147352567, 'Exception occurred.', (0, u'mscorlib', u"Could not find a part of the path 'C:\\Python27\\ProjectComponents'.", None, 0, -2147024893), None)
Traceback (most recent call last):
  File "D:\scriptname.py", line 123, in <module>
    o.CreateProject('test')
  File "C:\Users\Username\AppData\Local\Temp\gen_py\2.7\0CD36BB6-D828-4DB9-91BF-AD493EE76B79x0x10x0.py", line 2756, in CreateProject
    ret = self._oleobj_.InvokeTypes(213, LCID, 1, (9, 0), ((8, 1),),projectName
pywintypes.com_error: (-2147352567, 'Exception occurred.', (0, u'mscorlib', u"Could not find a part of the path 'C:\\Python27\\ProjectComponents'.", None, 0, -2147024893), None)

So as you can see, one works fine and the other fails mysteriously. What gives?

---

Here's what makepy.py -i gives:

>makepy.py -i VisualStudio.VCProjectEngine.8.0
VCProjectEngineLibrary
 {FBBF3C60-2428-11D7-8BF6-00B0D03DAA06}, lcid=0, major=8, minor=0
 >>> # Use these commands in Python code to auto generate .py support
 >>> from win32com.client import gencache
 >>> gencache.EnsureModule('{FBBF3C60-2428-11D7-8BF6-00B0D03DAA06}', 0, 8, 0)

>makepy.py -i VisualStudio.VCProjectEngine.10.0
VCProjectEngineLibrary
 {0CD36BB6-D828-4DB9-91BF-AD493EE76B79}, lcid=0, major=10, minor=0
 >>> # Use these commands in Python code to auto generate .py support
 >>> from win32com.client import gencache
 >>> gencache.EnsureModule('{0CD36BB6-D828-4DB9-91BF-AD493EE76B79}', 0, 10, 0)

---

Note: I just noticed that the CLSID given by makepy.py -i matches for VisualStudio.VCProjectEngine.8.0, but not for VisualStudio.VCProjectEngine.10.0 (where the registry shows {4547a58d-fc1c-4502-84fa-0163ee766635}):

However, even adjusting the tuple for VisualStudio.VCProjectEngine.10.0 to be ('{4547a58d-fc1c-4502-84fa-0163ee766635}', 0, 10, 0) doesn't help with the exception.

---

The error appears to be with Visual Studio 2010 and later. I also tried VisualStudio.VCProjectEngine.11.0and VisualStudio.VCProjectEngine.12.0 as ProgID with the values given by makepy.py -i, but the errors were similar:

Error for VisualStudio.VCProjectEngine.11.0: (-2147352567, 'Exception occurred.', (0, u'mscorlib', u"Could not find a part of the path 'C:\\Python27\\ProjectComponents'.", None, 0, -2147024893), None)
Error for VisualStudio.VCProjectEngine.10.0: (-2147352567, 'Exception occurred.', (0, u'mscorlib', u"Could not find a part of the path 'C:\\Python27\\ProjectComponents'.", None, 0, -2147024893), None)
Error for VisualStudio.VCProjectEngine.12.0: (-2147352567, 'Exception occurred.', (0, u'mscorlib', u'One or more errors occurred.', None, 0, -2146233088), None)

Answer 1

TL;WR (To long won't read)

The gist is that Microsoft has quietly changed the semantics of an automation interface, VisualStudio.VCProjectEngine, starting with Visual Studio 2010 and makes almost no mention of that fact in the official documentation. Instead developers seem to be expected to collect tidbits MSFT employees leave all over the MS Connect and forum pages and connect them in a meaningful way.

---

Wow, so Microsoft decided to change the semantics of how VCProjectEngine. Compare:

• VS2005 documentation (and here)
• VS2008 documentation (and here)
• VS2010 documentation (both respective links combined) ... has a mysterious remark added

The remark reads:

You cannot instantiate a stand-alone VCProjectEngine object because additional binaries must be deployed in the folder that contains the application. Instead, obtain a VCProjectEngine object from a project that has been opened in Visual Studio.

Unfortunately Microsoft seems to have decided that it's not legit to create a new project using an old interface, i.e. change the semantics of the interface.

This post is an attempt to collect and connect the information available - and that's available not in the official documentation.

And to add insult to injury, the documentation of this ends up in a meager remark at the bottom of the documentation for said interface.

If you want a better explanation ... or rather tidbits ... do a web search. Perhaps you can come up with anything better than I found. What I came up with was this statement here (emphasis mine):

As of Visual Studio 2010, this API is no longer supported. Please see this link for more information on how to programmatically access a project: http://blogs.msdn.com/b/visualstudio/archive/2010/01/15/how-to-read-write-the-new-visual-c-project-properties.aspx

It's "funny" how this vital information has not made it into the official documentation. So following the above link, all we get is an explanation of basics that existed before VS 2010, with the implication that the API is not supported only mentioned in the quoted MS Connect case.

This MS Connect case mentions setting an environment variable (VisualStudioVersion=11.0), but unsurprisingly this doesn't help either. Also using the Developer Command Prompt for VS201x for the respective Visual Studio version yields no results either. Another dead end.

Another MS forum thread seems relevant as well: Opening .vcxproj files in the VS2010 along with this one: VS 2010 C++ VCProjectEngine failure?.

Quoting the relevant part of the latter:

Cause

The error about not finding the path has nothing to do with the path to the .vcxproj file, actually. This error code bubbled up from an internal attempt to find the ProjectComponents folder under the InstallPath (the path to the running executable, actually).

Resolution

I just copied the Microsoft Visual Studio 10.0\Common7\Tools\ProjectComponents folder under the Debug folder where my test app was built, and it no longer fails. To be specific, the folder it is looking for is from a string composed (in C#) like this:

Path.Combine(System.AppDomain.CurrentDomain.BaseDirectory, "ProjectComponents")

One of the developers suggested I might "...also need to add this directory to the assembly probing path in your app config file." But I was able to get it to run the simple test without doing that. It may be required for other project engine operations.

A long journey ends at last? Not so ... Trying to locate this mysterious folder on a Visual Studio installation yields ... well, nothing.

C:\>for %i in (10.0 11.0 12.0) do dir "%ProgramFiles(x86)%\Microsoft Visual Studio %i\Common7\IDE\ProjectComponents"

... gives (stripped redundant information):

C:\>dir "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ProjectComponents"
 Directory of C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE
File Not Found
C:\>dir "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ProjectComponents"
 Directory of C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE
File Not Found
C:\>dir "C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\ProjectComponents"
 Directory of C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE
File Not Found

So the folders that are required for the alleged workaround don't exist in (all?) Visual Studio installations.

---

Side-note: the mysterious CLSID {4547a58d-fc1c-4502-84fa-0163ee766635} I found resolves to the name Microsoft.VisualStudio.Project.VisualC.VCProjectEngine.VCProjectEngineShim under the key HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{4547a58d-fc1c-4502-84fa-0163ee766635}. So I guess the actual thing was replaced by a shim that doesn't do its job very well.