docker '--mount' flag not identified while running from a child process

Question

I am trying to run a docker command as a child process from a Python script using subprocess module, but I get the below error.

b"unknown flag: --mount\r\nSee 'docker buildx build --help'.\r\n"

The script code I am trying to run is,

command = 'docker build -f /root/magneto/docker_env/Dockerfile.build -t magneto_build /root/magneto && docker run -ti --mount src=$(pwd),dst=/tmp/magneto/build,type=bind -e BUILD_NUMBER=${BUILD_NUMBER:-local} magneto_build'
command_args = shlex.split(command)
    
p = subprocess.Popen(command_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, error = p.communicate()

The output variable is blank and the error is received in error variable.

I also tried with the pexpect library, just to check if I am using the correct library and functions but that too threw the same error

Please suggest the reason for this and if possible a solution.

Answer 1

The problem is that you are running shlex.split on two commands, which obviously need the shell. Basically, you end up running docker build with arguments like && and docker and run and --mount, rather than running docker build and docker run as two separate commands.

For a more understandable example, compare:

>>> import subprocess, shlex
>>> subprocess.run(shlex.split('echo foo && echo bar >/dev/null'))
foo && echo bar >/dev/null
CompletedProcess(args=['echo', 'foo', '&&', 'echo', 'bar', '>/dev/null'], returncode=0)

You see? && and the other echo command and >/dev/null are all parsed into arguments for the first echo.

(Perhaps shlex.split should warn you if one of the resulting tokens is ; or || or && or > etc; but it is simply not designed to parse entire shell scripts, just individual commands.)

Either simply live with shell=True and pass in a string;

result = subprocess.run(
    'docker build -f /root/magneto/docker_env/Dockerfile.build -t magneto_build /root/magneto && docker run -ti --mount src=$(pwd),dst=/tmp/magneto/build,type=bind -e BUILD_NUMBER=${BUILD_NUMBER:-local} magneto_build'.
    shell=True, text=True, check=True, capture_output=True)
output = result.stdout
error = result.stderr

or painstakingly refactor this to avoid shell features (including, but not limited to, command substitution and variable interpolation).

first = subprocess.run(
    shlex.split('docker build -f /root/magneto/docker_env/Dockerfile.build -t magneto_build /root/magneto'),
    check=True, text=True, capture_output=True)
# Because of check=True, && is implicit -- there would be
# an exception if the first command failed
second = subprocess.run(
    ['docker', 'run', '-ti',
     '--mount', f'src={os.getcwd()},dst=/tmp/magneto/build,type=bind',
     '-e', f'BUILD_NUMBER={os.environ.get("BUILD_NUMBER", "local")}',
     'magneto_build'],
    check=True, text=True, capture_output=True)
output = first.stdout + second.stdout
error = first.stderr + second.stderr

Notice also how this avoids bare Popen; like its documentation clearly tells you, you should use subprocess.run() and friends when you can.

Perhaps see also Actual meaning of shell=True in subprocess and Running Bash commands in Python