openssl RSA public key does not match key read from C code

Question

I've created a public key file using the following commands:

openssl genrsa -out mykey.pem 2046
openssl rsa -in mykey.pem -pubout > mykey.pub

I then read it from some C code:

public_key = PEM_read_RSA_PUBKEY(fp, NULL, NULL, NULL);

and then obtain the DER form of this public key:

int len = i2d_RSAPublicKey (public_key, NULL);
buf = (unsigned char *) malloc (len);
i2d_RSAPublicKey (public_key, &buf);

here len == 269

but the command line openssl tells me:

openssl rsa -outform der  -pubin -inform pem -in mykey.pub | wc
writing RSA key
       1      10     293

When I print out the DER forms from openssl and my C code they, obviously don't match and oddly, the DER for that my C code obtains has a lot of zeros.

What could I be doing wrong?

Answer 1

I should have obtained the DER form using i2d_RSA_PUBKEY(). Also, passing &buf to the method should have warned me that the pointer was being modified. I have no idea why it gets modified and what it points to after the call.

So now I do

der_form = throw_away = (unsigned char *) malloc (len);
i2d_RSAPublicKey (public_key, &throw_away);

and use the contents of der_form.