How to decrypt using openssl EVP?

Question

I'm trying to decrypt a file using C (but I can change to C++), but I don't know how to use the EVP library correctly.

The console command that I want to replicate is:

openssl enc -rc2-ecb -d -in myfile.bin -iter 1 -md sha1 -pbkdf2 -pass pass:'Yumi'

My actual C code using EVP is:

unsigned char salt[8] = "Salted__";

ctx=EVP_CIPHER_CTX_new();
PKCS5_PBKDF2_HMAC_SHA1("Yumi", -1,
                    salt, 8, 1,
                    EVP_MAX_KEY_LENGTH, key);
EVP_DecryptInit(ctx, EVP_rc2_ecb(), key, iv);

pt = (unsigned char *)malloc(sz + EVP_CIPHER_CTX_block_size(ctx) + 1);
EVP_DecryptUpdate(ctx, pt, &ptlen, ciphertext, sz);

if (!EVP_DecryptFinal(ctx,&pt[ptlen],&tmplen)) {
        printf("Error decrypting on padding \n");
} else {
        printf("Succesful decryption\n");
        
}

I suppose that's not working because I have to declare something to use SHA1 instead of SHA256 (default on rc2-ecb), but I'm not seeing how to do it.

Any help will be appreciated

Answer 1

The following hex encoded ciphertext could be decrypted with the posted OpenSSL statement, if this data would be contained hex decoded in myfile.bin:

53616C7465645F5F2DC4C4867D3B9268C82E23A672D6698FB51D41EA8601367A9112623EC27CDEB18FD1444BDB8D8DE16F1A35706EC7FED266CB909D28BF6BEC

The decryption would result in:

The quick brown fox jumps over the lazy dog

---

For simplicity, the ciphertext is assigned directly and the loading of the ciphertext from a file is skipped:

unsigned char data[] = {
     0x53, 0x61, 0x6C, 0x74, 0x65, 0x64, 0x5F, 0x5F, // Salted__
     0x2D, 0xC4, 0xC4, 0x86, 0x7D, 0x3B, 0x92, 0x68, // Salt
     0xC8, 0x2E, 0x23, 0xA6, 0x72, 0xD6, 0x69, 0x8F, 0xB5, 0x1D, 0x41, 0xEA, 0x86, 0x01, 0x36, 0x7A, // Ciphertext...
     0x91, 0x12, 0x62, 0x3E, 0xC2, 0x7C, 0xDE, 0xB1, 0x8F, 0xD1, 0x44, 0x4B, 0xDB, 0x8D, 0x8D, 0xE1,
     0x6F, 0x1A, 0x35, 0x70, 0x6E, 0xC7, 0xFE, 0xD2, 0x66, 0xCB, 0x90, 0x9D, 0x28, 0xBF, 0x6B, 0xEC };
int dataLength = sizeof(data) / sizeof(unsigned char);

For decryption, salt and ciphertext must first be separated, e.g.:

int ciphertextLength = dataLength - 16;
unsigned char* salt = (unsigned char*)malloc(sizeof(unsigned char) * 8);
unsigned char* ciphertext = (unsigned char*)malloc(sizeof(unsigned char) * ciphertextLength);
memcpy(salt, data + 8, 8);                       // Get 8 bytes salt (starts at index 8) 
memcpy(ciphertext, data + 16, ciphertextLength); // Get ciphertext (starts at index 16) 

The next step is to derive the key, see PKCS5_PBKDF2_HMAC, e.g.:

#define KEYSIZE 16
unsigned char key[KEYSIZE];
const char* password = "'Yumi'"; // The quotation marks (') in the openssl-statement are part of the password.
int passwordLen = strlen(password);
PKCS5_PBKDF2_HMAC(password, passwordLen, salt, 8, 1, EVP_sha1(), KEYSIZE, key);

Finally the decryption can be performed, see here and here, e.g.:

EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();
EVP_DecryptInit_ex(ctx, EVP_rc2_ecb(), NULL, NULL, NULL);
EVP_CIPHER_CTX_set_key_length(ctx, KEYSIZE); // RC2 is an algorithm with variable key size. Therefore the key size must generally be set.
EVP_DecryptInit_ex(ctx, NULL, NULL, key, NULL);
unsigned char* plaintext = (unsigned char*)malloc(sizeof(unsigned char) * ciphertextLength);
int length;
EVP_DecryptUpdate(ctx, plaintext, &length, ciphertext, ciphertextLength);
int plaintextLength = length;
EVP_DecryptFinal_ex(ctx, plaintext + plaintextLength, &length);
plaintextLength += length;
printf("Plaintext: "); for (int i = 0; i < plaintextLength; i++) { printf("%c", plaintext[i]); } printf("\n");

For simplicity, the code doesn't include exception handling and memory release.

---

Note the following:

• Some of the parameters used in the posted OpenSSL statement are insecure, e.g. the ECB mode and an iteration count of 1. Instead, a mode with an IV should be used and an iteration count of 10,000 or larger if performance allows.
• OpenSSL generates a random 8 bytes salt during encryption, which is used to derive the key. The ciphertext consists of the ASCII encoding of Salted__, followed by the 8 bytes salt, followed by the actual ciphertext, here. Just this format is expected by the posted OpenSSL statement for decryption.
• RC2 defines a variable key size. The posted OpenSSL statement uses a size of 16 bytes.
• The -pass pass: option specifies the password without quotation marks, i.e. in the posted OpenSSL statement the quotation marks are part of the password.