RC2 decryption using CCCrypt

1.5k views Asked by At

I'm trying to decrypt RC2 data with only 64 effective bits.

Since I can only have 64-bits, it's my understanding that before calling CCCrypt I must use a method to reduce the key down to this many bits. Since I couldn't find any such method in Apple's CommonCrypto library, I am using this keyschedule method I found.

These are the method's parameters:

void rc2_keyschedule( unsigned short xkey[64], 
                     const unsigned char *key, 
                     unsigned len, 
                     unsigned bits ) 

For the actual decryption part, I'm trying to work off of an example that uses AES 256. This is what I have so far:

// setup the key to send to CCCrypt
unsigned char originalKey[16] = /* derived from some other method */;
unsigned short key[64];
unsigned effectiveBits = 64;
rc2_keyschedule(key, originalKey, 16, effectiveBits);
// key is now 128 bytes, and I manually checked it for accuracy

// setup the cipherText to send to CCCrypt
NSData *cipherText = /* derived from some other method */;
// cipherText was manually checked for accuracy

// setup the buffer to send to CCCrypt
size_t bufferSize = [cipherText length] + kCCBlockSizeRC2;
void *buffer = malloc(bufferSize);
size_t numBytesDecrypted = 0;

// call CCCrypt
CCCryptorStatus cryptStatus = CCCrypt(kCCDecrypt, 
                                      kCCAlgorithmRC2, 
                                      kCCOptionPKCS7Padding,
                                      key, 128,
                                      NULL /* initialization vector (optional) */,
                                      [cipherText bytes], [cipherText length],
                                      buffer, bufferSize, /* output */
                                      &numBytesDecrypted);


if (cryptStatus == kCCSuccess) {
  return [NSData dataWithBytesNoCopy:buffer length:numBytesDecrypted];
}

free(buffer); //free the buffer;
return nil;

When I run this, cryptStatus has the value kCCDecodeError which is documented as:

@constant kCCDecodeError Input data did not decode or decrypt properly.

The reason I am sending 128 as the keyLength to CCCrypt is because my key is 64 short ints long, and I believe that 1 short is equal to 2 bytes. Thus, 64 * 2=128.

I have no idea what I need to send for the option. I just used kCCOptionPKCS7Padding which was taken from the AES example. The other options available are kCCOptionECBMode and CBC. When I try the other two options, cryptStatus does become kCCSuccess, but the data is always null. I think that it is erroneously reporting success.

When I say that I "manually checked something for accuracy", I mean that I compared the key and cipher at those points to a JavaScript implementation which works successfully.

How can I use Apple's Libraries (i.e. CommonCrypt) to decrypt RC2 data?

0

There are 0 answers