I'm trying to create an AWS Lambda in python that:
- downloads a compressed and encrypted file from an S3 bucket
- decrypts the file using
python-gnupg - stores the decrypted compressed contents in another S3 bucket
This is using python 3.8 and python-gnupg package in a Lambda layer.
I've verified the PGP key is correct, that it is being loaded into the keyring just fine, and that the encrypted file is being downloaded correctly.
However, when I attempt to run gnupg.decrypt_file I get output that looks like it's been successful, but
the decrypt status shows not ok and the decrypted file does not exist.
How can I get PGP decryption working in Lambda?
Here is the relevant code extracted from the lambda function:
import gnupg
from pathlib import Path
# ...
gpg = gnupg.GPG(gnupghome='/tmp')
# ...
encrypted_path = '/tmp/encrypted.zip'
decrypted_path = '/tmp/decrypted.zip'
# ...
# this works as expected
status = gpg.import_keys(MY_KEY_DATA)
# ...
print('Performing Decryption of', encrypted_path)
print(encrypted_path, "exists :", Path(encrypted_path).exists())
with open(encrypted_path, 'rb') as f:
status = gpg.decrypt_file(f, output=decrypted_path, always_trust = True)
print('decrypt ok =', status.ok)
print('decrypt status = ', status.status)
print('decrypt stderr = ', status.status)
print('decrypt stderr = ', status.stderr)
print(decrypted_path, "exists :", Path(decrypted_path).exists())
Expectation was to get output similar to the following in CloudWatch:
2022-11-08T10:24:43.939-05:00 Performing Decryption of /tmp/encrypted.zip
2022-11-08T10:24:44.018-05:00 /tmp/encrypted.txt exists : True
2022-11-08T10:24:44.018-05:00 decrypt ok = True
2022-11-08T10:24:44.018-05:00 decrypt status = [SOME OUTPUT FROM GPG BINARY]
2022-11-08T10:24:44.018-05:00 decrypt stderr = ""
2022-11-08T10:24:44.214-05:00 /tmp/decrypted.txt exists : True
Instead what I get is:
2022-11-08T10:24:43.939-05:00 Performing Decryption of /tmp/encrypted.zip
2022-11-08T10:24:44.018-05:00 /tmp/encrypted.txt exists : True
2022-11-08T10:24:44.018-05:00 decrypt ok = False
2022-11-08T10:24:44.018-05:00 decrypt status = good passphrase
2022-11-08T10:24:44.018-05:00 decrypt stderr = [GNUPG:] ENC_TO XXXXXX 1 0
2022-11-08T10:24:44.214-05:00 /tmp/decrypted.txt exists : False
It appears as though decryption process starts to work, but something kills it, or perhaps the gpg binary is expecting some TTY input and halts?
I've tried locally running gpg decryption using the cli and it works as expected, although I am using GnuPG version 2.3.1, not sure what version exists on Lambda.
After a lot of digging I managed to get this working. I'm not 100% sure if the cause is the older
GnuPGbinary installed on the Lambda image by default, but to be sure I decided to build a GnuPG 2.3.1 layer for lambda which I confirmed was working as expected in a Docker container.I used https://github.com/skeeto/lean-static-gpg/blob/master/build.sh as a foundation for compiling the binary in Docker, but updated it to include compression, which was required for this use case.
Here is the updated updated
build.shscript I used, optimized for building for Lambda:Below is the Dockerfile used to build the layer:
Once the code is compiled in the image I copied it to my local directory, zipped it, and published the layer:
docker cp MY_DOCKER_ID:/opt/bin ./gnupgcd ./gnupg && zip -r gnupg-layer.zip binTo publish the layer:
I decided to not use the
python-gnupgpackage to have more control over the exact GnuPG binary flags so I added my own binary wrapper function:And then added an import key and decode function:
And updated the relavent Lambda code with:
And now the Cloudwatch log output is as expected and I've confirmed the decoded file is correct!