We have previously developed an RSA MSCAPI CSP for use with the classical Windows crypto API and this has worked fine for years. Unfortunately, newer versions of Outlook refuse to work with this CSP in the case of AES encryption. It still supports 3DES but not AES. This is quite bizarre because it is actually not the CSP which handles the symmetric decryption, but apparently Microsoft doesn't want to support the AES-case for MS-CAPI. For AES-support, the RSA key needs to be in a newer provider type namely a Key Storage Provider conforming to the CNG framework.
OK fine, but here's the rub: How can I ensure backwards-compatibility for clients that have software that relies on the MS-CAPI interface?
According to my understanding (which could be wrong), the certificate store is the same for MSCAPI and CNG. The difference comes in in terms of how the private key is referenced. The certificate has an attribute "CERT_KEY_PROV_INFO_PROP_ID" containing a number of fields including provider name, container name and provider type. If provider type is "0" (which was not a legal value in the old API) it indicates that the provider specified is actually one of the new CNG providers.
An old application will use the values from CERT_KEY_PROV_INFO_PROP_ID to acquire a crypto context using the legacy functions, i.e. CryptAcquireContext(). However, this function fails in the case of a CNG provider (i.e. provider type = 0) - here it seems the program will have to use the new CNG functions i.e. NCryptOpenStorageProvider, NCryptOpenKey etc. again passing in the values from CERT_KEY_PROV_INFO_PROP_ID. So if this understanding/testing is correct it means it is not possible to migrate to a CNG provider and still have the same certificates/keys working from the point-of-view of legacy application programs. I have not been able to find this stated explicitly in the documentation, but it seems each application program which will need to look at the CERT_KEY_PROV_INFO_PROP_ID content, and have a switch: If it is provider type = 0, this is a key in a CNG provider so the program will use the new CNG functions. On the other hand, if provider type is > 0 the program should use the legacy functions. But of course, a legacy program will not have this logic and will thus fail in case of the keys in CNG providers. Meaning it is not possible to satisfy the needs of both new and legacy programs because you have to either put in a reference to a legacy provider og a new provider in CERT_KEY_PROV_INFO_PROP_ID, but you can't have both. And Outlook only wants a reference to a new provider, and old programs can only work with old providers.
But can this really be so, or is there some thing I am missing or some error in my understanding? It seems there would have been multiple ways for Microsoft to help programs have some type of interoperability (for instance so old programs could use the new KSP's using the old API).
There's to my knowledge no solution that can solve the problem exactly as stated. BUT the new CNG API provides a function NCryptTranslateHandle() that a program can use to translate an old-style reference from CERT_KEY_PROV_INFO_PROP_ID to a handle to a corresponding KSP. When the KSP registers itself it can specify one or more "aliases" i.e. names of classic MS-CAPI providers it is aliases for. If a program then calls the NCryptTranslateHandle() handle on a handle from the classic MS-CAPI CSP, Windows will look to see if there's an installed KSP that has registered itself as an alias of that CSP. If so, the handle will get translated into a handle for the corresponding CSP. The program can then go on and use the new CNG functions on that handle, which will invoke the new provider.
So what to do: 1) Keep the CERT_KEY_PROV_INFO_PROP_ID reference in the certificate what it is now (i.e. pointing to old CSP), 2) Register a KSP provider that has the old CSP as alias (don't have the SDK in front of me now but it is one of the registration parameters you can provide. Let me know if you can't find it!).
What is a bit unsatisfactory about this is that it all rests on the calling program calling the NCryptTranslateHandle() handle. An old legacy program - written before the advent of CNG - obviously won't call this function, since the function didn't exist when it was developed. But Microsoft Outlook (at least newer functions) - which is a very common use case - does know to call this function. So if you offer a dual CSP+KSP solution as I described in the previous paragraphs, it will work out alright: Outlook will end up calling your solution via the KSP API, due to its use of the translate functions.
When implementing the solution you should write a program to test it before testing with Outlook. Let the program get a handle to the key the old-fashioned way, and then use the translation function and see you get a handle, then use the KSP functions on the resulting handle.