Trying to run subprocess commands with carriage returns and newlinees

Question

I'm trying to perform some commands to remove useless output from a file, I've figured out the necessary commands to do this although when trying to automate it through python, find the command is cut off midway through because part of the text I am trying to remove, is a carriage return.

def do_stuff():
    count = 0
    while True:
            try:
                count += 1
                basic_filename = fn + str(count) + ".txt"
                temp_filename = "temp_"+fn + str(count) + ".txt"
                final_filename = "final_"+fn + str(count) + ".txt"
                finalfinal_filename = "finalfinal_"+fn + str(count) + ".txt"

                cmd1 = b"sed -e 's/\\x1b\[[0-9;]*[mK]//g' -e 's/\r//g' -e 's/\\n//g' " + str.encode(basic_filename) + b" > " + str.encode(temp_filename)
                cmd1 = base64.b64encode(cmd1)
                cmd1 = "echo '"+bytes.decode(cmd1)+"' | base64 -d | bash"
                print(cmd1)
            except:
                break

An example output includes the Base64

"c2VkIC1lICdzL1x4MWJcW1swLTk7XSpbbUtdLy9nJyAtZSAncy8NLy9nJyAtZSAncy9cbi8vZycgcmVzdWx0czEudHh0ID4gdGVtcF9yZXN1bHRzMS50eHQ=" which when decoded shows: 

"sed -e 's/\x1b\[[0-9;]*[mK]//g' -e 's/

//g' -e 's/\n//g' results1.txt > temp_results1.txt"

Answer 1

You can't execute OS commands that are bytes

Of course you can. Instead of writing a shell script, you can run your commands like this:

import subprocess

basic_filename = "example.txt"
with open(basic_filename, "wb") as fd:
    fd.write(b"there is no \x1b[12;m escape")

out = subprocess.check_output(
    [
        b"sed",
        b"-e",
        b"s/\x1b\\[[0-9;]*[mk]//g",
        basic_filename.encode(),
    ]
)

print(out)

That will print out:

b'there is no  escape'

...having successfully removed the escape sequence. Of course, as others have pointed out in the comments, there's no reason to be calling out to sed to perform these transformations; just do it in Python instead:

import re

re_esc = re.compile(b"\x1b\\[[0-9;]*[mk]")

input_data = b"there is no \x1b[12;m escape"
out = re_esc.sub(b"", input_data)
print(out)