pathlib replace() fails to move existing files

Question

I have a script that dumps output into a "RUNNING" directory tree. When the script completes, I want to move the output into a PASS or FAIL directory tree, depending on how things went.

I'm having a devil of a time getting pathlib replace() to perform the move. For example:

source_dir = pathlib.Path('/home/me/output/12345-12345/RUNNING/20231018T160743')
dest_dir   = pathlib.Path('/home/me/output/12345-12345/PASS/20231018T160743')

source_dir.replace(dest_dir)

FileNotFoundError: [Errno 2] No such file or directory: '/home/me/output/12345-12345/RUNNING/20231018T160743' -> '/home/me/output/12345-12345/PASS/20231018T160743'

This is nuts because I can verify the source dir is a directory and exists, and I can verify the destination dir does NOT exist before attempting the move:

source_dir = pathlib.Path('/home/me/output/12345-12345/RUNNING/20231018T160743')
dest_dir   = pathlib.Path('/home/me/output/12345-12345/PASS/20231018T160743')

print(f"Moving source dir: {type(source_dir)} to dest dir: '{dest_dir}'")   

print(f"source dir is dir: {source_dir.is_dir()}")
print(f"source dir exists: {source_dir.exists()}")
print(f"dest dir is dir: {dest_dir.is_dir()}")
print(f"dest dir exists: {dest_dir.exists()}")


source_dir.replace(dest_dir)

>>> source dir is dir: True
>>> source dir exists: True
>>> dest dir is dir: False
>>> dest dir exists: False

Per some other questions on Stack Overflow, I've even enlisted psutil to make sure I don't have some open files that are preventing the directory to be moved.

import psutil

proc = psutil.Process()
print(f"{proc.open_files()=}")

Turned out I had a few logging File Handlers still open. I've closed them and still get the same error.

Answer 1

Ok, turns out that pathlib replace() isn't as clever as I had hoped.

The problem turned out to be that replace() has trouble creating a new path if one of the path sub-dirs doesn't already exist.

To illustrate the problem, notice that the difference between source_dir and dest_dir is that I renamed one of the sub-dirs from "RUNNING" to "PASS".

source_dir = pathlib.Path('/home/me/output/12345-12345/RUNNING/20231018T160743')
dest_dir   = pathlib.Path('/home/me/output/12345-12345/PASS/20231018T160743')

'/home/me/output/12345-12345/RUNNING' not existing is what is giving replace() trouble.

If I create the dest_dir ahead of time (note the -p option to mkdir!):

# Create the new output directory ahead of time
cmd = f"mkdir -p {str(dest_dir)}"
cmd = shlex.split(cmd)
subprocess.run(cmd, check=True)

pathlib replace() is happy to perform the move.

This is confirmed in that subsequent calls to pathlib replace() succeed once the '/home/me/output/12345-12345/PASS' sub-path exists.

EDIT: Per @Adam Smith's suggestion, I'll use pathlib mkdir() instead of the subprocess calls I was using while debugging.

dest_dir.mkdir(parents=True, exist_ok=True)