Encrypt data in flutter with a public key

Question

I need to create the following code in flutter, encrypt data with a public key using RSAES-OAEP/SHA-256/MGF1-SHA-1

textPayPass=document.getElementById('tpassword').value;
$('tpassword').val('');
s_cert = "-----BEGIN PUBLIC KEY-----MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA5xdSrXnpj8pUFBWvEWuEPmzL/iuCfoHwGsMou4PE5Y/wNmnM1E8niUww8yjLHgdOBOtPtLaBua/dkL+EKVXLylt0xpKtCC6KWBr07bvNK/CNXFUhd/9MMyV90UWJP5XHSzrxYz//ZJYn0XUicsKPDox7TJZFtMnw+euAwPDA7VhRupc4x8QHTkdTORUaeKSI3jLr6LdQC19BjeC5M4uZ45/U+6Hd9PPgKqp90dXrIyOcMXb65HrQoba6FsZnrN6qr7Z/4v4r6Xxdxo9Qwl6YBZJ3+1RlT/uSxjcxJTYgIropjI3O3r90+e7/DxE1ITHxxEjZhUkySTUBOkzeQdpRmHyL+ctSDvip7IO3Fd+axVvhSZq5w9x5DDWJcMhTiJpkjF3LHXaB1+T+oet36ClyMGmnnsnyOZ2s/kWiAaGFmqBebtk6iDkbq0y6hKtpNmPrx5hoUiQomo/cfoDpfnIUeJd4Kf+QRvTqGXP2yXbu8861LjWuaJ+kyTDUAoGV/Haw28P/Kwnr7oZTtP0mWU7bKqrPwoWAc9CHv2SKy4qjZCuS036VaAe2vd5gmGBni4hjA8YdWrLcPF5LwSDWoaXpPm9Ve8KB+5XpxLjyCjve/8y+9yGDtoYpTltOTsm9Au7BcZSFy5XyEAs6CQHaOZ6NGp8pIEhskUj7fv7/E6owkEMCAwEAAQ==-----END PUBLIC KEY-----";

var s_certForge = forge.pki.publicKeyFromPem(s_cert);
// encrypt data with a public key using RSAES-OAEP/SHA-256/MGF1-SHA-1
// compatible with Java's RSA/ECB/OAEPWithSHA-256AndMGF1Padding
var encData = s_certForge.encrypt(textPayPass, 'RSA-OAEP', {
md: forge.md.sha256.create(),
mgf1: {
    md: forge.md.sha1.create()
}
});
var s_encData = forge.util.encode64(encData);

my code in flutter

static String publicKeyTE ='''-----BEGIN PUBLIC KEY-----MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgsOgXZqI0joBljDjJ/9lOIHP1kGZJg8JsulkxK3CWOis/BO0N7Lqj70qGb8T3dgd/MOb/mmBkwxZ/U/PAIUS9YynPPPOd4ROXpEnkWYj5ntMKr7KWY/VKHSm+RzLWX8yi+8NmmK3NJosWzOXAwMIKY8ZSB+BBw//uI+aaRQlE4bAICufff9KcoJd8EEldQgLa74F+TgLg5VtV+2Ik9xgV2h2NaM6wvldp69OicjEgGPnX2uEwYJbo2HYiGjm5HnFtT70c5w8vO88O5/OvYafNfLmXZsszR/JYdPpOdKOf45NX2v50V43VnBgVlskki70kZAdi5roLjEpSC2vxjbh9I3dJrApp7UgnTE6+iaJg7onsXuXe1VPgp0evKM/BpVo2Jb95uD24oBop1OghQX6UyKYr7MAaPT3yT40vBOe5BxiAjbHDob/K8tzWt6fh1qKkLNry5jWOD/pQC7yrEBLIezB/Hzai+cIfBLqgWBEc1uA0D4/81SBu7MXTko1ne9HNF2zQwPtQOk0l/F8YOdCpyv4X0vxejc4pIwE+Iuk+KGqXvzRp2RnQZ6U08Bc4kkxXoEYSJ9ICyxSQCVxzJ9AMW8mlAOon71ZVr4ZWj9dicoQDUt+kXZV2E3itoy0guIabyk12ONvEekgzoRyRnrIv+mHWw6sV2Luo8bUh159sIsCAwEAAQ==-----END PUBLIC KEY-----''';

String encryptPassword(String password) {
RSAPublicKey publicKey=RsaKeyHelper().parsePublicKeyFromPem(publicKeyTE);
    final encrypter = Encrypter(
    RSA(
      publicKey: publicKey,
      encoding: RSAEncoding.OAEP,
      digest: RSADigest.SHA256,
    )
);
final encrypted = encrypter.encrypt(password);
return encrypted.base64;

}

I don't know what I might be missing but when I encrypt the password and send it to the server it appears incorrect. Could you help me see what I'm missing?

Answer 1

As already suspected in the comments, the digests used in the context of OAEP cause a compatibility problem, so that decryption fails.

For OAEP, various parameters must be specified, including the OAEP digest and the mask generation function (see RFC 8017). Although the latter is configurable in principle, in practice the MGF1 function is always used. MGF1 itself has a digest as parameter, so that in the context of OAEP there are actually two digests to be specified, the OAEP digest and the MGF1 digest.
Although in principle both digests can be specified differently, in practice the same digest is usually applied for both. This is not the case for your node-forge example, where SHA256 is used for the OAEP digest and SHA-1 for the MGF1 digest.

The encrypt package you are using (which is actually just a wrapper for some PointyCastle functionalities) does not allow the use of different digests.
The more powerful PointyCastle library is also designed to use identical digests by default: The MGF1 digest is stored in the property OAEPEncoding#mgf1Hash, but is implicitly initialized with the OAEP digest in OAEPEncoding#init().
However, if the mgf1Hash parameter is reinitialized immediately after the init() call, the reinitialized value is used for further processing. In this (slightly concealed) way, PointyCastle allows the specification of different digests for the OAEP and MGF1 digests.

Sample implementation (the sample ciphertext was generated with the node-forge code):

import 'dart:convert';
import 'dart:typed_data';
import 'package:basic_utils/basic_utils.dart';
import 'package:pointycastle/export.dart';
...
String privatePem = '''-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCynSxDzTtiAX4s
aLf70EIIFS2SoNnISjtQdzkLgu5btj5A49MvhzQ19JuIJkM2fSalCSyaccyuPqiY
X4UW23OVdI29cw5N95LY1gkXjthxpWxCof/v1xJtsY+6qOvR/c7OWcPJYM5zhZu1
sT8WJLc5kHbOxnyCdVua47348aZqiByU0ZRNCK3n8j8iQI56N3CbBz/M/JPKdqDd
SxOgZWIGxIMS62bUvDL0SZ0Y/EDuMWAs3JH0f8ToExit7Lx3l5WG+H9CslUHnw39
GTFTTimbyIawM8ffDQjiP2B0hJVORYcVa+ipy6VRXrsUjKUQUyy0ybtr9iEQR4eI
imUy5GVbAgMBAAECggEAGYw9rp0GZ2hLQdLgrhYFu947nIOBYxKrFUFYWXsq2Ndi
ZCmyJVQzdZw9OuYuKvcPtf9v789jNXEk8FaJVNGi4LfBJl30p9ZHnNhQiJwX/Xlp
2MYPM6ERs7r4Efzjbi/dispwJLfCD+gGc6Cco3Aw/Pza4YymdoqOxcLZg2gdI5OD
RwOZrmfbrroFrIgvB8AzOsc4EBVf/F0BA7JotWh9GyX2h8DiYtpd2NOCxjUar6GF
w+B76F9TYgHMexuluUe0Gi1J35yZqRR+ySSSuDCQNSOZFZ0UIP2KFhu+Vl2BbTJv
glbujfT/iLNBtDlxpznJJqbH908tfqmXVD2wLq5tUQKBgQDWDiXCf/6B4YXKz2zg
sCBPDkSLmJFv618SzjlZvfk+AYBgbCrk0hOuI7snb+XCuZVg9XnhIwyFM9OXwDET
as9nwxh3mVluSmpkNm3MgWYQvYbqtfXn/VTQk8HZZVtjx2c5UO1ENj7pd1u0OrQY
/jgqUL7yeTkWjUHprHAZv/LAtQKBgQDVnSZ5pTwlZnPDLbCSGWsbycIGiO4Bd/Gq
jQlptWIvnBhW7f3JZ6DoWlLTjvu0iA/OWT4aC7/w2i0VadMDvz0GqTzTlbnexf7B
+3L52tRCcQX1hRRDA3u6JV3kEpsweEnwWP39ZtXv924Ru/4S6DpZtwKvOFi9AvcI
lxdHc6MnzwKBgQCijDXCYH39VSvLWf6dFUJDplsJAr+WlM8qsa29Z1To8Czzi9B6
2MiXGY0aoo+AcntsGJ0ICRyN3lBU0V6zFw4PBokC8VGHPj1Sgj6Y6L6AAdx9SdZF
6AtLJJk1JBHlUFwjmz/B58uYcMoTr+xpekteXtjRuppOdNBTyV2LQEbwbQKBgDmt
1VXDaBoxL9Dj5WmNs34tXFanYpzC5l5G9uO0Nm7kly3h1UUs1iXnPbYiRZTZqGrv
bfVadtlyD/pYOMIqQTArQmFfbHDaxY9bdhBBJk9KdXF2HaJ6rk31CQUsgPr1gAGG
Bg8GVX4WMYJYYKJ6UkcnQ3JMpKlkw49uPLruXn/BAoGANCI+9Q26Gd395Wu0ZcjO
0yr4INfx8ilrFiyjf1KpLC2G6kmL78zQrQP1I344Yplu9eJllDtj4AmxPa+/bCQz
VN4DOnDY3599CLvG74f/oA0jB4jQB+vfMcTt5QNhvLQeWxhV5CRna36S//5Y9eab
8Xh1UaEAkxxVwUjUDCasMhs=
-----END PRIVATE KEY-----''';
const ciphertextB64 = "m1nvGHz3RohBU40YWbWv0hDUghpLSdQxXzoQ9N0Tz2yMXDSYWEZLZ2L62x7x5ghTobUx3iHi1Rnsybs+ybmV2kt35YUfWe5/D3TpsvREBe0Bz2JaZveuPyE28iOe5w4BwCxd8rvakbmlbxdIx0T6SSRIvR/nq1qBhQfeIlJxjEnySi37Rk8cF5ci6FBQniWrF+n7HR/nRmPJKuwZ+EAYPmSgGDYXvyU73rOF/gn/ADlB8Is6cKA0JCadn16hJNLLpR68wur0yhcrR8LPlz5t4x5cfsWkI8vLsfdmtmSfAXcQgGp+guwAKQh275Rju+SlSMwql7BYRNYeAcuYNMKNHQ==";
final Uint8List ciphertext =  base64Decode(ciphertextB64);
final RSAPrivateKey privateKey = CryptoUtils.rsaPrivateKeyFromPem(privatePem);
final OAEPEncoding oaepEncoding = OAEPEncoding.withSHA256(RSAEngine()); // specify SHA-256 as OAEP digest
final PrivateKeyParameter<RSAPrivateKey> privateKeyParams = PrivateKeyParameter(privateKey);
oaepEncoding.init(false, privateKeyParams); // initializes MGF1 digest with OAEP digest (implicitly)
oaepEncoding.mgf1Hash = SHA1Digest(); // overwrite initialization and specify SHA-1 as MGF1 digest
final Uint8List plaintext = oaepEncoding.process(ciphertext);
final String plaintextStr = utf8.decode(plaintext);
print(plaintextStr);

This successfully decrypts the ciphertext generated with the node-forge code.