I have a folder on my local computer named move and inside move, there are three files names movefile1.txt, movefile2.txt and movefile3.txt. I also have the following bash script named minscp.sh on my computer:
#! /bin/bash
PORT="22"
SOURCE_PATH="/path/to/folder/move"
SAVE_PATH="/home/ubuntu/"
USER="ubuntu@"
IP="<aws public ip>"
scp -r -P $PORT $SOURCE_PATH $USER@$IP:$SAVE_PATH
If I cd into the folder that contains this scrip and I run ./minscp.sh then the following is returned:
movefile1.txt 100% 0 0.0KB/s 00:00
movefile2.txt 100% 0 0.0KB/s 00:00
movefile3.txt 100% 0 0.0KB/s 00:00
I want to call this script with subprocess.Popen() and I want to capture each line that is printed out. Usually the following woks for me, but in this case using scp, I am not getting the result that I expected to:
>>> with subprocess.Popen(
... [
... path_to_bash,
... ],
... stdout=subprocess.PIPE,
... universal_newlines=True
... ) as p:
... for i, line in enumerate(p.stdout):
... print(f"{i} : {line}")
...
>>>
This function runs and will successfully move the files, however nothing is printed to the console. I tried using stderr instead and I got the following:
>>> with subprocess.Popen(
... [
... path_to_bash,
... ],
... stderr=subprocess.PIPE,
... universal_newlines=True
... ) as p:
... for i, line in enumerate(p.stderr):
... print(f"{i} : {line}")
...
movefile1.txt 100% 0 0.0KB/s 00:00
movefile2.txt 100% 0 0.0KB/s 00:00
movefile3.txt 100% 0 0.0KB/s 00:00
>>>
Notice that nothing was actually printed in the for loop and that the response I know got but did not get above when trying to call the response from stdout happens sometime before the loop.
How can I capture the response line by line as the processes is running?
EDIT
I found out if I add the verbose tag to the script, then I almost get what I want. In other words, if I change the scp call in minscp.sh to the following:
scp -v -r -P $PORT $SOURCE_PATH $USER@$IP:$SAVE_PATH
then subprocesses.Popen() gives me the following
>>> with subprocess.Popen(
... [
... path_to_bash,
... ],
... stderr=subprocess.PIPE,
... universal_newlines=True
... ) as p:
... for i, line in enumerate(p.stderr):
... if line.startswith("Sending"):
... f =line.split(" ")[-1]
... print(f"{i} : {f}", end="")
...
90 : movefile1.txt
movefile1.txt 100% 0 0.0KB/s 00:00
95 : movefile2.txt
movefile2.txt 100% 0 0.0KB/s 00:00
100 : movefile3.txt
movefile3.txt 100% 0 0.0KB/s 00:00
>>>
and then if I add in stdout this will become
>>> with subprocess.Popen(
... [
... path_to_bash,
... ],
... stderr=subprocess.PIPE,
... stdout=subprocess.PIPE,
... universal_newlines=True
... ) as p:
... for i, line in enumerate(p.stderr):
... if line.startswith("Sending"):
... f =line.split(" ")[-1]
... print(f"{i} : {f}", end="")
...
90 : movefile1.txt
95 : movefile2.txt
100 : movefile3.txt
>>>
This technically accomplishes what I want but does not really answer my question as to how can I capture the following, which I am suppressing with stdout and replacing with the line from the scp call with the -v tag in place:
movefile1.txt 100% 0 0.0KB/s 00:00
movefile2.txt 100% 0 0.0KB/s 00:00
movefile3.txt 100% 0 0.0KB/s 00:00