Need help porting calls to EVP_PKEY_get1_EC_KEY/i2d_ECPrivateKey & EVP_PKEY_get1_RSA/i2d_RSAPrivateKey to OpenSSL 3.0.8

Question

I have C/C++ code using OpenSSL 1.1.1i and in the process of switching to OpenSSL 3.0.8. Most things are working nicely but I've run into an issue related obtaining DER-encoded bytes for a private key.

Below is a snippet of the code - it supports two key types: EC & RSA. The private key is loaded elsewhere and is valid. Then I determine what type the key is and, for EC, use EVP_PKEY_get1_EC_KEY and i2d_ECPrivateKey to get the DER-encoded bytes. For RSA, I use EVP_PKEY_get1_RSA and i2d_RSAPrivateKey.

These 4 functions are deprecated in 3.x.x and I can't for the life of me figure out how I can get those bytes into my tmp variable - which is then used in the ... do stuff with tmp portion of the code.

Does anyone have insight on how to accomplish this in OSSL3?

EVP_KEY *privateKey;
int keyId;
uint_8_t *tmp;
int len;

privateKey = ... // privateKey is loaded elsewhere and is valid

keyId = EVP_PKEY_id(privateKey);

if(keyId == EVP_PKEY_EC)
{
    tmp = nullptr;
    EC_KEY *ec = EVP_PKEY_get1_EC_KEY(privateKey);
    len = i2d_ECPrivateKey(ec, &tmp);
    
    if(len > 0)
    {
        // ... do stuff with tmp

        OPENSSL_free(tmp);
    }
}
else if(keyId == EVP_PKEY_RSA)
{
    tmp = nullptr;
    RSA *rk = EVP_PKEY_get1_RSA(privateKey);
    len = i2d_RSAPrivateKey(rk, &tmp);

    if(len > 0)
    {
        // ... do stuff with tmp

        OPENSSL_free(tmp);
    }
}

I've tried various methods including calls to EVP_PKEY_get_bn_param/BN_bn2bin, as well as pouring through examples using what looks like customized encoding using OSSL_ENCODER.... No luck. But, frankly, I'm not entirely sure what I'm doing with those functions anyway.

Thank you kindly!

Answer 1

I think I may have answered my own question... It involves OSSL_ENCODER_xxx calls, importantly to specify OSSL_KEYMGMT_SELECT_ALL when creating the encoder context as well as "DER" and "type-specific" in the parameters.

This is working for an EC private key right now. I will get around to testing RSA keys in due course but thought I'd post what I've found so far in case anyone finds it useful.

// Private key DER bytes
#if (OPENSSL_VERSION_MAJOR == 1)
    keyId = EVP_PKEY_id(privateKey);
    if(keyId == EVP_PKEY_EC)
    {
        EC_KEY *ec = EVP_PKEY_get1_EC_KEY(privateKey);
        tmp = nullptr;
        len = i2d_ECPrivateKey(ec, &tmp);
        if(len > 0)
        {
            // ... do stuff with tmp
            OPENSSL_free(tmp);
        }
    }
    else if(keyId == EVP_PKEY_RSA)
    {
        tmp = nullptr;
        RSA *rk = EVP_PKEY_get1_RSA(privateKey);
        len = i2d_RSAPrivateKey(rk, &tmp);
        if(len > 0)
        {
            // ... do stuff with tmp
            OPENSSL_free(tmp);
        }
    }
#elif (OPENSSL_VERSION_MAJOR == 3)
{  
    tmp = nullptr;
    size_t dataLen = 0;
    OSSL_ENCODER_CTX *ctx = OSSL_ENCODER_CTX_new_for_pkey(privateKey, OSSL_KEYMGMT_SELECT_ALL, "DER", "type-specific", NULL);

    if((ctx != nullptr) && (OSSL_ENCODER_to_data(ctx, &tmp, &dataLen) == 1) && (dataLen > 0))
    {
        // ... do stuff with tmp
    }

    if(tmp != nullptr)
    {
        OPENSSL_free(tmp);
    }

    OSSL_ENCODER_CTX_free(ctx);
}
#else
#error Unsupported OpenSSL version
#endif