Sign on OS X, Verify on iOS and OSStatus -9809

Question

I am using Apple's Security Framework. I am able to sign and then successfully verify all on OS X, but when I try to use SecKeyRawVerify on iOS it fails with -9809 error.

I've played with various PKCS padding options and many other attributes but I'm just not able to get this to verify correctly.

Note that the code below probably has leaks all over the place, just trying to get this to function properly first.

OS X Signing code:

NSData* signData(NSData* plainData, SecKeyRef privateKey) {
    CFErrorRef error;

    /* Create the transform objects */
    SecTransformRef signer = SecSignTransformCreate(privateKey, &error);
    if (error) { CFShow(error); exit(-1); }                                    

    /* Explicitly set padding type (necessary?) */
    SecTransformSetAttribute(signer,
                             kSecPaddingKey,
                             kSecPaddingPKCS1Key,
                             &error);
    if (error) { CFShow(error); exit(-1); }

    /* Specify digest type */
    SecTransformSetAttribute(signer,
                             kSecDigestTypeAttribute,
                             kSecDigestSHA1,
                             &error);
    if (error) { CFShow(error); exit(-1); }

    /* OS X calculates SHA1 hash/signature for us */
    SecTransformSetAttribute(signer,
                             kSecTransformInputAttributeName,
                             plainData,
                             &error);
    if (error) { CFShow(error); exit(-1); }

    CFTypeRef signature = SecTransformExecute(signer, &error);
    if (error || !signature) { CFShow(error); exit(-1); }

    CFRelease(signer);

    return signature;
}

and iOS verification code:

+ (NSData *)verifyData:(NSData *)data
              usingKey:(SecKeyRef)publicKey {
    size_t signatureByteLength = SecKeyGetBlockSize(publicKey);

    if (signatureByteLength > data.length)
    {
        NSLog(@"Signature length is greater that data length.");
        return nil;
    }

    NSData *signature = [data subdataWithRange:NSMakeRange(0, signatureByteLength)];
    NSData *plainData = [data subdataWithRange:NSMakeRange(signatureByteLength, data.length - signatureByteLength)];

    NSLog(@"signatureLength='%lu', signatureBytes='%@', plainDataLength=%lu", (unsigned long)signature.length, signature, (unsigned long)plainData.length);

    size_t hashByteLength = CC_SHA1_DIGEST_LENGTH;
    uint8_t* hashBytes = (uint8_t *)malloc(hashByteLength);

    if (CC_SHA1([plainData bytes], (CC_LONG)[plainData length], hashBytes))
    {
       NSData *b = [NSData dataWithBytes:hashBytes length:hashByteLength];
       NSLog(@"hashBytesLength='%lu', hashBytes=%@", (unsigned long)b.length, b);

        OSStatus status = SecKeyRawVerify(publicKey,
                                  kSecPaddingPKCS1SHA1, // I have also tried kSecPaddingPKCS1, doesn't work
                                  hashBytes,
                                  hashByteLength,
                                  (uint8_t *)[signature bytes],
                                  signatureByteLength);

        switch (status)
        {
            case errSecSuccess:
                NSLog(@"SecKeyRawVerify success.");
                return plainData;

            case errSSLCrypto:
               NSLog(@"SecKeyRawVerify failed: underlying cryptographic error");
               break;

            case errSecParam:
               NSLog(@"SecKeyRawVerify failed: one or more parameters passed to a function where not valid.");
               break;

            default:
               NSLog(@"SecKeyRawVerify failed: error code '%d'", (int)status);
               break;
       }
    }
   return nil;
}

I created the private and public keys via command line using the following OpenSSL commands:

1.    // Generate private and public key pair
openssl genrsa -out rsaPrivate.pem 1024

1a. // Generate public key
openssl rsa -in rsaPrivate.pem -pubout -outform PEM -out rsaPublic.pem

2. //Create a certificate signing request with the private key
openssl req -new -key rsaPrivate.pem -out rsaCertReq.csr

3. //Create a self-signed certificate with the private key and signing request
openssl x509 -req -days 3650 -in rsaCertReq.csr -signkey rsaPrivate.pem -out rsaCert.crt

4. //Convert the certificate to DER format: the certificate contains the public key
openssl x509 -outform der -in rsaCert.crt -out rsaCert.der

Any help is greatly appreciated.

Answer 1

I figured out the issue. The code I posted is correct, but the padding needs to be set as kSecPaddingPKCS1SHA1 per the SecKey.h header file:

If you are verifying a proper PKCS1-style signature, with DER encoding
of the digest type - and the signedData is a SHA1 digest - use
kSecPaddingPKCS1SHA1.

Also, you might want to make sure your public key in .der format is the correct one :)

Answer 2

if (CC_SHA1([plainData bytes], (CC_LONG)[plainData length], hashBytes))
{
    NSData *b = [NSData dataWithBytes:hashBytes length:hashByteLength];
    NSLog(@"hashBytesLength='%lu', hashBytes=%@", (unsigned long)b.length, b);
    ...
}

It appears you are not encoding and applying the padding correctly. The encoding and padding gets applied before the hashing. See RFC 3447, Public-Key Cryptography Standards (PKCS) #1: RSA Cryptography Specifications Version 2.1 (or the PKCS #1 specification).