Problems with pkcs7 file validation

Question

I have a big and strange problem with PKCS7 securities files. I create a p7s file like what is suggested in http://www.thatsjava.com/java-tech/85019/ where it creatse the file and validates it using only sun libraries. It works fine.
The problem starts when I want to validate this file. It returns this exception:

java.security.SignatureException: Signature encoding error
        at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:185)
        at java.security.Signature$Delegate.engineVerify(Signature.java:1140)
        at java.security.Signature.verify(Signature.java:592)
        at sun.security.pkcs.SignerInfo.verify(SignerInfo.java:374)
        at sun.security.pkcs.PKCS7.verify(PKCS7.java:494)
        at sun.security.pkcs.PKCS7.verify(PKCS7.java:511)
        at sun.security.pkcs.PKCS7.verify(PKCS7.java:533)
        at firma.FirmaDigitalImpl.firmarCadenaSun(FirmaDigitalImpl.java:553)
        at firma.FirmaDigitalImpl.firmarCadena(FirmaDigitalImpl.java:249)
        at firma.FirmaDigitalImpl.firmarCadena(FirmaDigitalImpl.java:147)
        at firma.TestFirma.main(TestFirma.java:75)
Caused by: java.io.IOException: Sequence tag error
        at sun.security.util.DerInputStream.getSequence(DerInputStream.java:280)
        at sun.security.rsa.RSASignature.decodeSignature(RSASignature.java:209)
        at sun.security.rsa.RSASignature.engineVerify(RSASignature.java:174)
        ... 10 more

But the problems don't appear always, just only with a type of certificates used to sign. I will explain better. I have two certificates (stored into smartcard) and the first works fine; I create the p7s and next I validate correctly, but the second certificate allows me to create the p7s, file but when I validate it, it returns the SignatureException. I thought the p7s file was wrong, but I tested it with others' applications, and it looks correct. Besides, this file is sent throws webservice and it returns that it is ok!

And for more information, if the p7s file is created with different tools (.net and capicom) and the same certificate, then I can validate correctly.

I have looked for the solution, but nothing. I have found the same exception in similar situations, but either the solution suggested doesn't work for me, or it doesn't appear.

Any advice to find the solution will be very appreciated.

Answer 1

It might be encoding problem: text vs. binary. PKCS7 can be in either DER-encoded binary format or PEM-format (which is base64 encoded DER). Open the files in a text editor (e.g. notepad) and see whether the one which works is binary or text.

Answer 2

After more than two weeks, I have noticed what was the problem, although I didn't find the full solution.

First, I extracted the problem out of the PKCS7 file, and I reproduced the exception with the following source code:

import diz.firma.keyStore.SmartCard;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.Provider;
import java.security.PublicKey;
import java.security.SecureRandom;
import java.security.Security;
import java.security.Signature;

public class TestFirma3 {
public static void main(String args[]) throws Exception {
    //Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
    SmartCard sm = new SmartCard();
    KeyStore ks = sm.loadKeyStore("1234");
    //KeyPair keyPair = generateKeyPair(999);


    byte[] data = "original".getBytes("UTF-8");
    //byte[] data = { 65, 66, 67, 68, 69, 70, 71, 72, 73, 74 };
    //byte[] digitalSignature = signData(data, keyPair.getPrivate());
    byte[] digitalSignature = signData(data,
            (PrivateKey)ks.getKey(sm.getAlias(), null),ks.getProvider());

    boolean verified;

    //verified = verifySig(data, keyPair.getPublic(), digitalSignature);
    verified = verifySig(data, ks.getCertificate(sm.getAlias()).
            getPublicKey(),ks.getProvider(), digitalSignature);
    System.out.println("verified:" + verified) ;

    //keyPair = generateKeyPair(888);
    //verified = verifySig(data, keyPair.getPublic(), digitalSignature);
    //System.out.println(verified);

  }

  public static byte[] signData(byte[] data, PrivateKey key,Provider p) throws Exception {
    Signature signer = Signature.getInstance("SHA1withRSA",p);
    //Signature signer = Signature.getInstance("SHA1withRSA",Security.getProviders()[10]);

    signer.initSign(key);
    signer.update(data);
    return (signer.sign());
  }

  public static boolean verifySig(byte[] data, PublicKey key, Provider p, byte[] sig) throws Exception {
    Signature signer = Signature.getInstance("SHA1withRSA",p);
    //Signature signer = Signature.getInstance("SHA1withRSA");
    signer.initVerify(key);
    signer.update(data);

    boolean valido = false;
    try{
        valido = signer.verify(sig);
    }catch(Exception e){
        e.printStackTrace();
        valido = false;
    }
    return valido;

  }

  public static KeyPair generateKeyPair(long seed) throws Exception {
    Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());
    KeyPairGenerator keyGenerator = KeyPairGenerator.getInstance("DSA");
    SecureRandom rng = SecureRandom.getInstance("SHA1PRNG", "SUN");
    rng.setSeed(seed);
    keyGenerator.initialize(1024, rng);

    return (keyGenerator.generateKeyPair());
  }
}

In this code, (extracted from an example in the net), I changed the generators key and took it from my card. I got the exception again, without using PKCS7 files.

Looking at all providers (with Security.getProviders() you can get all of them; they are defined in java.security file, but it can be added to or taken from in runtime), and I used everyone to sign and validate, I found out that:

1. If no one provider is selected when Signature is instantiated, it uses SunRsaSign by default.
2. My Signature is validated with 2 providers:
   a) With SunMSCAPI provider, which uses Microsoft Crypt API.
   b) With the provider created by smart card reader, in my case Siemens.
3. Reading another smart card that does not belong to Siemens, it is validated with SunRsaSign and SunJSSE.

At this moment, I find out the problem and a workarournd using SunMSCAPI to validate, but I have to validate into an IBM machine. Siemens software provides an option to install into Unix, but I cannot get it to work. And using SunMSCAPI, I suppose it isn't available for IBM.

So, I have to find out a provider that validates correctly in IMB the Signature that SunRsaSign returns SignatureException.