mscapi java.security.ProviderException: java.security.KeyException:: The parameter is incorrect

Question

I get "The parameter is incorrect" exception when I try decrypt the encrypted file with MSCAPI. I use RSAEncryptUtil.java utility and JDK 1.8.0_201 for this homework project.

package hu.infokristaly.homework4rsaencdec;

import java.security.KeyStore;
import java.security.PrivateKey;
import java.security.PublicKey;
import java.security.cert.Certificate;

import sun.security.mscapi.SunMSCAPI;

public class Homework4RSAEncDec {

    private static final String alias = "papp zoltán";
    private static final String fileName = "c:\\temp\\test.txt";
    
    public static void main(String[] args) throws Exception {
        KeyStore keyStore = KeyStore.getInstance("Windows-MY", new SunMSCAPI());
        keyStore.load(null, null);
        
        Certificate cert = keyStore.getCertificate(alias);          
        PublicKey pubKey = cert.getPublicKey();

        RSAEncryptUtil.encryptFile(fileName, fileName + ".enc", pubKey);

        KeyStore.PasswordProtection  keyPassword = new KeyStore.PasswordProtection ("".toCharArray());
        KeyStore.PrivateKeyEntry selectedKey = (KeyStore.PrivateKeyEntry)keyStore.getEntry(alias, keyPassword);
        PrivateKey privKey = selectedKey.getPrivateKey();

        RSAEncryptUtil.decryptFile(fileName + ".enc", fileName + ".dec", privKey);
    }

}

The exception is occurred in cipher.doFinal at line 98 of RSAEncryptUtil.

public static byte[] decrypt(byte[] text, PrivateKey key) throws Exception
{
    byte[] dectyptedText = null;
    // decrypt the text using the private key
    Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding","SunMSCAPI");
    cipher.init(Cipher.DECRYPT_MODE, key);
    dectyptedText = cipher.doFinal(text);
    return dectyptedText;

}

When I generate keypair with RSAEncryptUtil and no MSCAPI, everything works fine.

KeyPair keyPair = RSAEncryptUtil.generateKey();
RSAEncryptUtil.encryptFile(fileName, fileName + ".enc", keyPair.getPublic());
RSAEncryptUtil.decryptFile(fileName + ".enc", fileName + ".dec", keyPair.getPrivate());

When I use 1024 key length RSA in Windows-My, it works fine.

Answer 1

I changed the buffer size for decrypton of RSAEncryptUtil:

byte[] buf = cipherMode == Cipher.ENCRYPT_MODE ? new byte[100] : new byte[KEYSIZE / 8];

I set KEYSIZE to 4096 and this helped for 4096 bit RSA key and this is solved the problem.