Gnu Parallel - decrypt and send file content to a Python script

329 views Asked by At

I would like to decrypt a bunch of very large files and use the decrypted version of each file as input to a Python script which will process its content. So, if I have a file named

 file1.sc.xz.gpg

after running the GnuPG decryption tool the output should be stored in a file named

 file1.sc.xz 

inside the same directory and this file should be the input to the Python script which will process its contents. Ideally I would like to do this inside one single Bash command, but I couldn't find the right way to do it. What I tried is:

 find test/ -type f | parallel 'f="{}"; g="${f%.*}"; gpg "$f" > "$g" | python iterating-over-tokens.py "$g" '

but is not working. Any other suggestions? Many thanks in advance.

Later edit: if I could send the decrypted file (*.sc.xz) content directly to the Python script as an argument, that would be even better.

1

There are 1 answers

4
Ole Tange On BEST ANSWER

Directly piped to Python:

parallel gpg -o - {} '|' python -c "'import sys; print sys.stdin.read().upper()'" ::: *.gpg 

Create decrypted file first:

parallel gpg -o {.} {} ';' python -c "'import sys; print sys.argv'" {.} ::: *.gpg 

You need to be able to decrypt without entering a pass phrase. If gpg asks for a pass phrase run gpg-agent first.