Generate signature using private key with OpenSSL API

Question

I'm trying to implement the SSL equivalent of:

openssl dgst -sha1 -binary -out digest.txt what.txt

openssl rsautl -sign -inkey key.pem -in what.txt -out signature.txt

I have my 7 lines private key, 304 chars:

const char* k = "-----BEGIN... \n... \n... END RSA.."

The code I use is this one:

BIO* bio = BIO_new_mem_buf((void*)k, (int)strlen(k));

RSA* privateKey = PEM_read_bio_RSAPrivateKey(bio, NULL, 0, NULL);

std::string appName = "hello";

unsigned char SHA[20];

SHA1((const unsigned char*)appName.c_str(), appName.size(), SHA);

Then I sign using the following code, passing the SHA and it's size as buffer and bufferSize (which should always be 20 in this case and it's correct) while the signature size is 32:

unsigned int privateKeySize = RSA_size(privateKey);
*signature = new unsigned char[ privateKeySize ];

RSA_sign(NID_sha1, buffer, bufferSize, *signature, signatureSize, privateKey);

RSA_sign return 0, I can't find any error code etc. The SHA[] array content is the same of the file digest.txt, but the signature is totally wrong. I can't find any example of this.. Do you have any idea how this should work, or point me into the right direction?

Thank You.

Answer 1

First of all, the digest is never treated as hexadecimals in a signature, which is what you seem to assume by the extension of digest.txt. The output of the dgst command within OpenSSL is also binary (although the manual pages on my system seem to indicate otherwise - openssl's documentation isn't known to be very precise). This is only a superficial problem as the rest of your code does seem to treat the SHA-1 hash as binary.

What's probably hurting you is that, although openssl rsautl -sign seems to pad the data, it doesn't seem to surround it with the structure that indicates the hash algorithm used. You need to append this structure yourself. This is probably a legacy from OpenSSL's support for SSLv3 RSA authentication where this structure may be absent.

A quick and dirty fix that is actually present in the PKCS#1 standard is to prefix the hash before signing. In the case of SHA-1 you should prefix the hash with the binary value of the following hex representation:

30 21 30 09 06 05 2b 0e 03 02 1a 05 00 04 14

this should be easy to do in your shell language of choice. Even the windows command line is able to do this through the COPY command.