Large file decryption error on nodejs openpgp

Question

I have a few pgp encrypted binary files in an s3 bucket. I'm trying to decrypt them using a nodejs lambda function. It's working fine for the files which are less than 2 GB in size, but the decryption is failing for the files which are greater than 2 GB in size.

My implementation is as follows:

const fileStream = fs.createReadStream(encryptedFile);
const message = await openpgp.readMessage({
  binaryMessage: fileStream // parse armored message
});
const { data: decrypted, signatures } = await openpgp.decrypt({
  message,
  format: 'binary',
  decryptionKeys: privateKey,
  config: { allowUnauthenticatedStream: true, preferredCompressionAlgorithm: openpgp.enums.compression.zlib }
});

The error message is as follows:

"[ERROR] Message: Error decrypting message: Cannot read the properties of undefined (reading 'length')"

I did some debugging and identified that it's able to parse the armored message and it's failing on the decryption step itself. Few other observations:

1. After reading the message successfully, it's taking almost 4 minutes to throw this error.
2. This error is happening for the files encrypted from a particular source only. I have tried to replicate the decryption process on another lambda with a 2.5 GB file from another source with the same decryption implementation and I succeeded in it.

My lambda is set to have maximum configs (10 GB ram and 10 GB storage)

What could be the possible reason for this? If there was an issue with the file itself, it should be failing while reading the message. Is there any other way to verify if there's something missing in the encrypted file? Should we consider any other parameter while decrypting large files?

Answer 1

Computers have lots of memory nowadays. However, that doesn't mean that it is advantageous to stream all the data into memory. Instead it is better to read data into a buffer, encrypt or decrypt it and then immediately stream it back into a file that will contain the cipherext. This can be seen in this code example within the documentation.

2 Gi is exactly the amount of elements that you can address with a signed 32 bit integer, and it is the maximum array size on many platforms. Currently the code is using up to 2 GiB of internal memory in the best scenario to decrypt the data. Streaming directly to file will usually take less than 1 MiB of volatile memory (i.e. RAM).