WinVerifyTrust error code handling

Question

I've been tasked with determining whether a particular DLL from a third party company has been tampered with, after installation on a user's system. I've never done anything related to digital signing before. I'm trying to set up a test on my own system using WinVerifyTrust.

{
    WINTRUST_FILE_INFO wtfi;
    wtfi.cbStruct = sizeof(WINTRUST_FILE_INFO);
    wtfi.pcwszFilePath = TEXT("*****.dll");
    //wtfi.hFile = DllHandle;
    wtfi.pgKnownSubject = NULL;

    GUID wtvPolicyGUID = DRIVER_ACTION_VERIFY;

    WINTRUST_DATA wtd;
    wtd.cbStruct = sizeof(WINTRUST_DATA);
    wtd.pPolicyCallbackData = NULL;
    wtd.pSIPClientData = NULL;
    wtd.dwUIChoice = WTD_UI_NONE;
    wtd.fdwRevocationChecks = WTD_REVOKE_NONE;
    wtd.dwUnionChoice = WTD_CHOICE_FILE;
    wtd.pFile = &wtfi;
    wtd.dwStateAction = WTD_STATEACTION_IGNORE;
    wtd.pwszURLReference = NULL;
    wtd.dwProvFlags = WTD_REVOCATION_CHECK_NONE;
    //wtd.pSignatureSettings = NULL;            // Win8 and Server2012 only?

    LONG result = WinVerifyTrust( NULL, &wtvPolicyGUID, &wtd);
    debugf(TEXT("Validation result: 0x%08x"), result);
}

This is returning 0x57. From what I've gathered from MSDN, errors come from a supplied trust provider. I don't really know what the trust provider is or what error messages it can return.

1. I've linked in WinTrust.dll and WinTrust.lib, so I presume that means I'm using Microsoft's "Software Publisher Trust Provider". Is this recommended, or are there better ones out there? Should you be using one specific to / provided by the software publisher whose product you're analyzing?
2. SoftPub.h contains the GUID input value, but does not seem to provide output error codes. Any help in tracking down their response code list is appreciated.

Thanks in advance.

EDIT: I have since figured out that this library uses error codes provided by winerror.h. 0x57 is "ERROR_INVALID_PARAMETER", so I'm reviewing what it could be complaining about. I also tried switching the Policy GUID to WINTRUST_ACTION_GENERIC_VERIFY_V2, which returned error TRUST_E_SUBJECT_FORM_UNKNOWN. Neither error code is particularly illuminating about what the ultimate issue is.

EDIT 2: I also ran Microsoft's SignTool.exe on the dll in question, and got the following output:

SignTool Error: A certificate chain processed, but terminated in a root
        certificate which is not trusted by the trust provider.

Number of errors: 1

So it seems like I need to change the trust provider I'm using. After discussing with the software manufacturer, the task is being dropped in favor of another approach.

Answer 1

According to MSDN it seems that you should set

dwStateAction = WTD_STATEACTION_VERIFY;

Also I would try setting

wtfi.hFile = NULL;

or when giving the file handle setting

wtfi.pcwszFilePath = NULL;

(It's not quite clear to me if you are providing hFile or not. And I would not set both hFile and pcwszFilePath to valid values.)

Another point to check: if you are compiling for Windows 8 or Windows Server 2012 you will have the struct member pSignatureSettings and will need to initialize it. Take care to set cbStruct so that pSignatureSettings is not included or to properly initialize pSignatureSettings.