CryptVerifyDetachedMessageSignature() fails with SHA256withDSA and self-signed certificate

55 views Asked by At

I'm trying to verify a signed JAR file using the Windows crypto APIs. This code has actually worked for a few years now and it still works with SHA1withDSA (using the same self-signed certificate) and SHA256withRSA (using the official public certificate chain). But it fails with SHA256withDSA and the self-signed certificate.

Both jarsigner and openSSL are able to verify this JAR successfully:

jarsigner -verify -verbose -certs xyz.jar
openssl cms -in LOCALSIG.DSA -inform der -verify -content LOCALSIG.SF -CAfile localSigning.cer

Here's my slightly simplified code:

void verify_signature(std::string const & message, std::string const & signature) {
    CRYPT_VERIFY_MESSAGE_PARA verify_param {
        sizeof(CRYPT_VERIFY_MESSAGE_PARA),
        X509_ASN_ENCODING | PKCS_7_ASN_ENCODING,
        0,
        nullptr,
        nullptr
    };
    BYTE const * message_array[] { (BYTE const *)message.data() };
    DWORD message_size_array[] { (DWORD)message.size() };
    HCERTSTORE store = nullptr;
    HCRYPTMSG crypt_msg = nullptr;
    for (DWORD signer_index = 0; true; ++signer_index) {
        PCCERT_CONTEXT signer_cert = nullptr;
        BOOL res = ::CryptVerifyDetachedMessageSignature(&verify_param, signer_index,
                                                        (BYTE const *)signature.data(), (DWORD)signature.size(),
                                                         1, message_array, message_size_array,
                                                         &signer_cert);
        if (res && signer_cert) {
            // verify the certificates' chain of trust and the timestamp here...
        } else {
            DWORD const err = ::GetLastError();
            if (err == ERROR_SUCCESS || err == CRYPT_E_NO_SIGNER) {
                return;
            } else {
                throw last_error{ L"Signature verification failed", err };
            }
        }
    }
}

CryptVerifyDetachedMessageSignature() returns FALSE, signer_cert == nullptr.
::GetLastError() == 50 == ERROR_NOT_SUPPORTED == "The request is not supported."
According to the docs, this error code shouldn't even be returned or propagated by this function.

The used algorithms are pretty much standard and should be supported by WinCrypt. See here. The self-signed certificate itself uses SHA1withDSA. Perhaps WinCrypt doesn't like this combination of algorithms?

I tried passing a callback function in verify_param, but nothing could be learned by that. The function is called, but the passed certificate struct is mostly null in both the working and failing cases, except for the serial number, issuer and subject, all of which are correct.

I'm stuck. I don't know what else to try with my code. I tried various ways of working around the issue by varying parameters and trying differently signed JARs, but no luck. Any idea what else I could try or why CryptVerifyDetachedMessageSignature() behaves like that?

0

There are 0 answers