CryptoApi to CommonCrypto

Question

I have cryptographic code used in the Windows platform, which uses the Crypto API functions and need to convert this to using Common Crypto on OS X.

Essentially the original code is this, with error checking removed for brevity: -

CryptAcquireContext(&m_hProv, NULL, NULL, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT));
CryptCreateHash(m_hProv, CALG_MD5 ,0, 0, &hHash);
CryptHashData(hHash,(LPBYTE)pszInputData, lstrlen(pszInputData)*sizeof(TCHAR), 0);

CryptDeriveKey(m_hProv, CALG_RC4, hHash, CRYPT_EXPORTABLE | 0x00280000, &m_hKey);
CryptDecrypt(m_hKey, 0, bFinal, 0, pData, pdwDataSize);

As far as I understand, this is what's happening: -

CryptAcquireContext - Get an object to handle the cryptography

CryptCreateHash - Create an MD5 hashing object

CryptHashData - Hash the input data with MD5

CryptDeriveKey, CryptDecrypt - Decode pData with RC4, using the key m_hKey

The size of pszInputData is 12 bytes and the output array of the MD5 hashed object is the same on both platforms.

To decode with RC4, I'm doing the following with Common Crypto: -

CCCryptorRef cryptor = NULL;
CCCryptorCreate(kCCDecrypt, kCCAlgorithmRC4, 0,
                      (void*)m_hKey.data(), m_hKey.length(), NULL, &cryptor);

char outBuffer[12];
size_t outBytes;
CCCryptorUpdate(cryptor, (void*)pData, *pdwDataSize, outBuffer, 12, &outBytes);

Testing the output (outBuffer array) from Common Crypto with an online RC4 decoder matches, so this is decoding correctly.

However, the final output from the Windows code in pData doesn't match the RC4 decoded in Common Crypto.

Are there some steps I'm missing or not understanding with the Windows Crypto API calls here; why do the outputs differ?

(Please note, I'm not looking for comments on the security or flaws in using RC4)

Answer 1

Try using the API described in Open SSL (EVP_BytesToKey - password based encryption routine).

Answer 2

The problem turns out to be understanding how CryptDeriveKey uses the number of specified bits with the RC4 Decryption.

CryptDeriveKey(m_hProv, CALG_RC4, hHash, CRYPT_EXPORTABLE | 0x00280000, &m_hKey);

Here it's stating that we want a 40 bit key (0x00280000 = 40 << 16).

However, when calling CryptDecrypt, Windows actually uses 16 bytes for the key, taking the first 40 bits and setting the rest of the array to zero.

So, passing a 16 byte key, with just the first 40 bits set, to the CCCryptorCreate function generates the matching output to Windows.