Python Popen waiting while it shouldn't (bg and output redirected)

Question

When I run directly in a terminal:

echo "useful"; sleep 10 &> /tmp/out.txt & echo "more";

I get both outputs while sleep goes on in the background. I was this same behavious with Popen (python 2.7):

p = Popen('echo "useful"; sleep 10 &> /tmp/out.txt & echo "more";', shell = True, stdout = PIPE, stderr = PIPE)
print p.communicate()

It was my understanding that a background process with redirected stdout and stderr would achieve this, but it does not; I have to wait for sleep. Can someone explain?

I need the other output so changing stdout/stderr arguments in Python is not a solution.

---

EDIT: I understand now that the behaviour I want (get the output but stop when no more output rather than when completed) is not possible from Python.

However, the behaviour appears more or less automatically when using ssh:

ssh 1.2.3.4 "echo \'useful\'; cd ~/dicp/python; nohup sleep 5 &> /tmp/out.txt & echo \'more\';"

(I can ssh to this address without password). So it's not entirely impossible by working around Python; now I need a way to do it without ssh...

Answer 1

As Martijn Pieters points out, this is not how Python behaves (or is meant to behave). However, because the desired behaviour appears when running the command through ssh with nohup, I found this similar trick:

p = Popen('bash -c "echo \'useful\'; cd ~/dicp/python; nohup sleep 5 &> /tmp/out.txt & echo \'more\';"', shell = True, stdout = PIPE, stderr = PIPE)
print p.communicate()

So if I understand correctly, this starts a new shell (bash -c), which then starts a process not attached to it (nohup). The shell terminates as soon as all other processes complete, but the nohup-process keeps running. Desires behaviour achieved!

Maybe not pretty and probably not efficient, but it works.

EDIT: assuming, of course, that you are using bash. A more general answer is welcome.

EDIT2: actually, if my explanation is correct, I am not sure why nohup does not detach the process even if not using bash -c... Seems like bash -c would be redundant, just detach it from the shell started by Popen, but that does not work.

Answer 2

That's because the shell process still has to wait for the background process to finish.

You don't normally realize this is happening because you normally are working in the shell where you backgrounded something. You put a process in the background so you can get control of the shell again and continue to work with it.

In other words, the background process is relative to the shell, not your Python process.