I am using DSS library to verify digital signature. Recently, DSS introduced the ability to send just the hash of the document being verified and it's signature without sending the actual content. That may be a good thing for privacy, because you could potentially avoid sending the document to the server (on our system the verification component runs on the backend). In order to do that we need to strip the content away from the signature, calculate hash over it and send it to the DSS to verify.
This all sounds good, but when we actually try to strip the content and send the created "detached" signature for verification along with the hash we get an exception from DSS:
WARN [eu.europa.esig.dss.xades.validation.XAdESSignature] (default task-7) Determining signing certificate from certificate candidates list failed: [Certificate #1: Signature verification failed, Certificate #2: Signature length not correct: got 256 but was expecting 512
This doesn't make much sense as all of the certificate digests are 256 bit long, not 512.
Here's the actual stripped signature:
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#" Id="id-448b15cfb37f10f659949fe53afb3bcc"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/><ds:Reference Id="r-id-448b15cfb37f10f659949fe53afb3bcc-1"><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/TR/1999/REC-xpath-19991116"><ds:XPath>not(ancestor-or-self::ds:Signature)</ds:XPath></ds:Transform><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><ds:DigestValue>VkPkiQYDbE3NZ2fQv7pwDInIY0YjQAbVJvulFHITSoI=</ds:DigestValue></ds:Reference><ds:Reference Type="http://uri.etsi.org/01903#SignedProperties" URI="#xades-id-448b15cfb37f10f659949fe53afb3bcc"><ds:Transforms><ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></ds:Transforms><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><ds:DigestValue>04YiTp2wqxQpL0DlG0NvJcnnVwdacoykFMBbsfZhajU=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue Id="value-id-448b15cfb37f10f659949fe53afb3bcc">v4fKeQwUI0XYAsTJPti3GYDUCsdyCvUJV0RUXAd9vqyuij7pqaVNK/6/uSnGViokCB8w5w3/T1NPNJTXZ5ahY183Fo86j7MHf2BYjy0K+jSbflG0GGOnVPtpQ05qjVgfKRTAo/xjjKZWgBEAR2hQGSm4eF79I302i9SPDSqy6BuKMCa0d32lyzsmJRSN64ySCbAGx3qxtLjUskhQf73rZnYS8t5TLz5h6wA6hPMLTAIHp5J/LVcznuCSjcP14dll/ZqPvJI5pMp+J3dGU3XjYkylGAX8fx2gO52d1/IRJGubVPM2Sc60xV+iwk3ufS4PwOHZwu7svQwU8Ei8LZ+gCQ==</ds:SignatureValue><ds:KeyInfo><ds:X509Data><ds:X509Certificate>MIIG4jCCBMqgAwIBAgIJROVSTMAL7eJgMA0GCSqGSIb3DQEBCwUAMEoxCzAJBgNVBAYTAlJTMRAwDgYDVQQHDAdCZW9ncmFkMQ8wDQYDVQQKDAZOZXRTZVQxGDAWBgNVBAMMD0Nsb3VkIENBIE5ldFNlVDAeFw0xOTEyMTgxMzQyMTlaFw0yMjEyMTgxMzQyMTlaMIGdMQswCQYDVQQGEwJSUzEPMA0GA1UECAwGU2VyYmlhMREwDwYDVQQHDAhCZWxncmFkZTEOMAwGA1UEEQwFMTEwNzAxGjAYBgNVBAkMEVBhcmlza2Uga29tdW5lIDI0MRIwEAYDVQQEDAlSYWRvc2V2aWMxDzANBgNVBCoMBk5pa29sYTEZMBcGA1UEAwwQTmlrb2xhIFJhZG9zZXZpYzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM8uNqgZGbmTy1K4gjWiWLKwCKdwhipCwXJOllKCamOUvyQmykUJRrs3JjBhYkdOdQA9UyGCs/5JLgmVA6EeEEKWJGEZmKwv8JQvcp09J1Iovc7hcv+rQvsQUJqnlsrZX9eagvyO4p90xMoNdqpk9vFtW29cKB7qhw6nBUYzpbW26wa3q7tgdS9s9cF8OWfh1LGkAdVNdXnCVsBLn3wpOn2cjFZuStksMldN4dqa/N2yglBuLPd94xxUP1KR5DlIBwXSRhQtQedWbpLxz0EVwoboino6VkqTlc9kolcPxpHHQ5OQdpXXx2pjFxI6U0Kw6jkjm6DpOiyGLi6v6HowZK0CAwEAAaOCAnUwggJxMAkGA1UdEwQCMAAwDgYDVR0PAQH/BAQDAgbAMB8GA1UdIwQYMBaAFAPpPpldBu73UZtn3v+c6oGACeRWMB0GA1UdDgQWBBTpMNMINjCsPaBc769k7862JW3fVDBABgNVHR8EOTA3MDWgM6Axhi9odHRwOi8vc3RhcmZpc2h0ZXN0Lm5ldHNldGdsb2JhbC5ycy9jbG91ZGNhLmNybDCBnQYIKwYBBQUHAQMBAf8EgY0wgYowOgYIKwYBBQUHCwIwLgYHBACL7EkBATAjhiFodHRwOi8vY2EubmV0c2V0LnJzL2Rva3VtZW50YWNpamEwCAYGBACORgEBMAgGBgQAjkYBBDAWBgYEAI5GAQIwDBMDZXVyAgEAAgID6DALBgYEAI5GAQMCARQwEwYGBACORgEGMAkGBwQAjkYBBgEwgZgGCCsGAQUFBwEBBIGLMIGIMDsGCCsGAQUFBzAChi9odHRwOi8vc3RhcmZpc2h0ZXN0Lm5ldHNldGdsb2JhbC5ycy9jbG91ZGNhLmNydDBJBggrBgEFBQcwAYY9aHR0cDovL3Brc2NhLm5ldHNldGdsb2JhbC5yczo4MDgwL29jc3AvcmVzcG9uZGVyL3Brc2Nsb3Vkb2NzcDCBlgYDVR0gBIGOMIGLMCcGBgQAizABATAdMBsGCCsGAQUFBwIBFg9odHRwOi8vZXRzaS5jb20wJwYGBACLMAECMB0wGwYIKwYBBQUHAgEWD2h0dHA6Ly9ldHNpLmNvbTALBgcEAIvsQAECMAAwKgYJKoF6AWkKBAEDMB0wGwYIKwYBBQUHAgEWD2h0dHA6Ly9uZWtpLmNvbTANBgkqhkiG9w0BAQsFAAOCAgEAJp8rru1bx3LDYG2lfruitO9qcV2CwgGNWXPt10zAi/nyZjY/HvMonjQGC86bUkYyEMw21cjYfHdCs8BLUid5I5Hbj2htGYZvRXx+R6ci0nlhmzXXv1acws9Zn9E8gfk4zosI8tHrJhPnJZuPWcDjS1QDJG6U74kbQpN2JnfM2VFvXiyjb0t4J/ZxskY3hb+IBEVPL8Kn3oQRTPYC0kEECq2UPZkRmKOFimWs8bRoqbAG3IWf+qRlENFiU+cjhdV1NYAiiMnd64EU6+zbxOJn0X4YjJz01xgLeaS4uridA9alAR5gHxL8LBMvM+i4cgkqTf9hguC0VTaiPfsrfu7/s0VA4wbE0ebk1ZUIb3bs9eLhq3E6h6V4Wuqe6lGNORuJgok/K64DoU7RPL/yVJRxPBt55tvNxQBZ+d+StAAMWXxJmw5lTn3vs0jdzzXLP1HrtNk0sDldzXlSoqwcGrXcyCcbIfVCdsDaOpGlTc9o2bMMJ06WjEZ9Wz8YhTWYftiJhES8h3m0BYicVJYGgYY5huvM1CZq495M68C/FffZG/tBcI34epWm2kb7lRt7mq/+5Ts4519YL5uFHqYWhfHmABQmT8i/yzFpdlP6aMmXyssDP9TK/f/f2fttalICfMZoDBQY15At2xEkXMAG3yMyI0dA19RXn1ZzMwjZfZII4YA=</ds:X509Certificate><ds:X509Certificate>MIIGbTCCBFWgAwIBAgIJF2ZKhDYQKGe2MA0GCSqGSIb3DQEBCwUAMEkxCzAJBgNVBAYTAlJTMRAwDgYDVQQHDAdCZW9ncmFkMQ8wDQYDVQQKDAZOZXRTZVQxFzAVBgNVBAMMDlJvb3QgQ0EgTmV0U2VUMB4XDTE5MTIxNzA4NDI0NVoXDTQ0MTIxNjE3MzA0NVowSjELMAkGA1UEBhMCUlMxEDAOBgNVBAcMB0Jlb2dyYWQxDzANBgNVBAoMBk5ldFNlVDEYMBYGA1UEAwwPQ2xvdWQgQ0EgTmV0U2VUMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAtHsLnZdvGEV3Ga9qFHiA4AbsHNRyuswxpYMn6yqn4WLg4RMSiwoL7RlQqK4L/5YvEykovUNKfgyIfBfmEPJD69oEGfpLL7pL2mQ3ls99l6fo4LCCizg8hQ6V2HIQ3uNn9kUg/7wJNyBliomfn5iOt6CNi8oGFeG25PoFjQwfALeywJsFCZqhk67E6Ym65NWb/J0ETigZJ/u4jp0jz5Z2x1sniQzwTeB6OLDRKfod4f0QzDYHV3OK3JvZZ3v+qBKqhhFynPOhFM1iM+2cTHBaEOXG8D9xF7/rBJiMPZahk+GFBt3ESBnbfQg7wM2tgzW64KekSypWxAS7Q8h2c+mrnsMh80HLrRcXatd9tmOiC2dswzjGs25YqhPx5hrVlu0XbZ0rC902w8q6siQQYH1i+XyDSYd++r2vFtzknvSmfHeAsbswuSUL21+9rYTVbUO+NITKblvew2ZqIwq1/sRrBWvu3IsuQ7thBvaC1QJtM5PXP2UvKoPbFdOtSlvpt98NBNigvLurwveRxy7ObTkRk2qcPqmdS39vEB0Vq0UDL2k/vvGnjZC0jeI+87cBSMGDPlY7zJJZZ37RX4ad/xEUkr2YYiEIKr2sHQzh0sVoOiXk2yfISagY2P4cwBIHNFZTMFdJh4xtCdsOPwS9rkRARqgpTGQOr/81EUufcaHodHUCAwEAAaOCAVUwggFRMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMB8GA1UdIwQYMBaAFOJl5dmktrA3AM0gptZbcUDxvk5bMB0GA1UdDgQWBBQD6T6ZXQbu91GbZ97/nOqBgAnkVjA/BgNVHR8EODA2MDSgMqAwhi5odHRwOi8vc3RhcmZpc2h0ZXN0Lm5ldHNldGdsb2JhbC5ycy9yb290Y2EuY3JsMEoGCCsGAQUFBwEBBD4wPDA6BggrBgEFBQcwAoYuaHR0cDovL3N0YXJmaXNodGVzdC5uZXRzZXRnbG9iYWwucnMvcm9vdGNhLmNydDBeBgNVHSAEVzBVMFMGBFUdIAAwSzAuBggrBgEFBQcCARYiaHR0cDovL25ldHNldC5ycy9kb2t1bWVudGFjaWphLnBkZjAZBggrBgEFBQcCAjANDAtOZXRTZXQgZG9jczANBgkqhkiG9w0BAQsFAAOCAgEAGsToOTQswJ7CPbtvTjJbbi9JTVAv1r2nCV5GmfS0d0ZPsQiY7/egnUDFH8njrWfwjKl2KVXOibnWB2VRYY1mW2VTK0JsIQGrrQ9SihmqnBjZn3Hb2MkpnLtQthCGevJsOlnp7ZlWd6tJCnXYj9JFLaz8uIt1ILUA9Uq2brphLAgkSN1RICzOPqUISozPgEXepoBNap05lBTyyGoBKqm+tXc9iOZwVgI+IDoAbpMICAXuPXtXyFNyFxRb5Rw8llg+mBoPEKPzH/qRTqUxyqOWC68CM2qczycUOQVyF4med9fYa0a/AjVKu3uhhXhDSiCRi06x1LdTlHwHRXpoLWn4k4xcJdgLwM0k/CtBu+TsuMy+zatodVcGyfnfRi/jJm8cHGy8QFWH2yLiENhHwZIQv+HL2lr1mUpmUVJ6gZrie9Cd6DWcUKfIAVWXjhfDLmTm4ZHD7jn5LoJplFe2RKSzbia7/YA/w3sV1B87uFXD4ZnQGibW1rl+obn7xQlkJGRJxsWAKZiE50FlaskH3ETzSLFxtYH/0jv9liukLnyJh27jxSEqXdEEatWlVH/XYogbMEW4CbxJIxfXPp69l4gCirQooheNuXJGbBa1lQAmcPCJFUUW8MwX9jUldf6NMUsX/uGqYQxeKnDHyfj5iAsdyZpQ2bqfTbRp97wsYa+Qrk0=</ds:X509Certificate></ds:X509Data></ds:KeyInfo><ds:Object><xades:QualifyingProperties xmlns:xades="http://uri.etsi.org/01903/v1.3.2#" Target="#id-448b15cfb37f10f659949fe53afb3bcc"><xades:SignedProperties Id="xades-id-448b15cfb37f10f659949fe53afb3bcc"><xades:SignedSignatureProperties><xades:SigningTime>2020-07-19T12:46:29Z</xades:SigningTime><xades:SigningCertificateV2><xades:Cert><xades:CertDigest><ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha512"/><ds:DigestValue>3oKjV/svZGkzE8RfAtPDgk7+CFifLKrTwDKlAJKlUr1uKyC4HP6IkOdkOjWb8/QY8W8E1TPl0FKFadiMof0mpQ==</ds:DigestValue></xades:CertDigest><xades:IssuerSerialV2>MFswTqRMMEoxCzAJBgNVBAYTAlJTMRAwDgYDVQQHDAdCZW9ncmFkMQ8wDQYDVQQKDAZOZXRTZVQxGDAWBgNVBAMMD0Nsb3VkIENBIE5ldFNlVAIJROVSTMAL7eJg</xades:IssuerSerialV2></xades:Cert></xades:SigningCertificateV2></xades:SignedSignatureProperties><xades:SignedDataObjectProperties><xades:DataObjectFormat ObjectReference="#r-id-448b15cfb37f10f659949fe53afb3bcc-1"><xades:MimeType>application/octet-stream</xades:MimeType></xades:DataObjectFormat></xades:SignedDataObjectProperties></xades:SignedProperties></xades:QualifyingProperties></ds:Object></ds:Signature>
EDIT: Since the problem seems to be about the actual signature and the digest value used as the input, like dave_thompson_085 has pointed out, I tried to get the digest from the SignatureValue manually to try to figure out what's going on.
So, I took the certificate #1 value and run the following command to extract the public key:
openssl x509 -pubkey -noout -in cert.pem > pubkey.pem
Then I took the value from the SignatureValue tag and run the following command to decrypt the signature and get the digest:
openssl rsautl -inkey pubkey.pem -pubin -in sig.bin -out result.bin
However, when I base64 encoded the result using some online tool to compare it to the digest that I was passing and that was included in the actual signature, I got the following 'MDEwDQYJYIZIAWUDBAIBBQAEIDBaM9z9ZgShg0Ev6X38gyfjQXHlWsyRTbJ3RBUvMeNB'. This is weird because it's longer that the actual 32-byte digest that I expected and doesn't resemble anything I expected (when I check the binary version of the file I got it's actually 64-byte long). What am I doing wrong?