python -c vs python -<< heredoc

Question

I am trying to run some piece of Python code in a Bash script, so i wanted to understand what is the difference between:

#!/bin/bash
#your bash code

python -c "
#your py code
"

vs

python - <<DOC
#your py code
DOC

I checked the web but couldn't compile the bits around the topic. Do you think one is better over the other? If you wanted to return a value from Python code block to your Bash script then is a heredoc the only way?

Answer 1

The main flaw of using a here document is that the script's standard input will be the here document. So if you have a script which wants to process its standard input, python -c is pretty much your only option.

On the other hand, using python -c '...' ties up the single-quote for the shell's needs, so you can only use double-quoted strings in your Python script; using double-quotes instead to protect the script from the shell introduces additional problems (strings in double-quotes undergo various substitutions, whereas single-quoted strings are literal in the shell).

As an aside, notice that you probably want to single-quote the here-doc delimiter, too, otherwise the Python script is subject to similar substitutions.

python - <<'____HERE'
print("""Look, we can have double quotes!""")
print('And single quotes! And `back ticks`!')
print("$(and what looks to the shell like command substitutions and $variables!)")
____HERE

As an alternative, escaping the delimiter works identically, if you prefer that (python - <<\____HERE)

Answer 2

How to use here-docs with input

tripleee's answer has all the details, but there's Unix tricks to work around this limitation:

So if you have a script which wants to process its standard input, python -c is pretty much your only option.

This trick applies to all programs that want to read from a redirected stdin (e.g., ./script.py < myinputs) and also take user input:

python - <<'____HERE'
import os

os.dup2(1, 0)
print(input("--> "))
____HERE

Running this works:

$ bash heredocpy.sh
--> Hello World!
Hello World!

If you want to get the original stdin, run os.dup(0) first. Here is a real-world example.

---

This works because as long as either stdout or stderr are a tty, one can read from them as well as write to them. (Otherwise, you could just open /dev/tty. This is what less does.)

In case you want to process inputs from a file instead, that's possible too -- you just have to use a new fd:

Example with a file

cat <<'____HERE' > file.txt
With software there are only two possibilites:
either the users control the programme
or the programme controls the users.
____HERE

python - <<'____HERE' 4< file.txt
import os

for line in os.fdopen(4):
  print(line.rstrip().upper())
____HERE

Example with a command

Unfortunately, pipelines don't work here -- but process substitution does:

python - <<'____HERE' 4< <(fortune)
import os

for line in os.fdopen(4):
  print(line.rstrip().upper())
____HERE

Answer 3

If you prefer to use python -c '...' without having to escape with the double-quotes you can first load the code in a bash variable using here-documents:

read -r -d '' CMD << '--END'
print ("'quoted'")
--END
python -c "$CMD"

The python code is loaded verbatim into the CMD variable and there's no need to escape double quotes.

Answer 4

If you are using bash, you can avoid heredoc problems if you apply a little bit more of boilerplate:

python <(cat <<EoF

name = input()
print(f'hello, {name}!')

EoF
)

This will let you run your embedded Python script without you giving up the standard input. The overhead is mostly the same of using cmda | cmdb. This technique is known as Process Substitution.

If want to be able to somehow validate the script, I suggest that you dump it to a temporary file:

#!/bin/bash

temp_file=$(mktemp my_generated_python_script.XXXXXX.py)

cat > $temp_file <<EoF
# embedded python script
EoF

python3 $temp_file && rm $temp_file

This will keep the script if it fails to run.