transferring local files to Databricks DBFS system using CLI in python code not working

Question

Here is the issue: when I run the stand alone command to transfer file from on-prem location to DBFS, it works

databricks fs cp /mnt/farm/DNAlab/WGL/Samples/Data\ Analysis/Emedgene/HPO/2213305.HPO.txt dbfs:/FileStore/LiveDataUpload/WES_HPO_Dec16/

but when I run it inside my python code, it fails.

 my_cmd = "databricks fs cp "+row_file+" "+upload_dir
                run_args = {"shell":True, "check":True, "capture_output":True}
                subprocess.run(my_cmd, **run_args)

please help... why ?

Answer 1

If we avoid using shell=True we'll have an easier time working with filenames that contain spaces.

Something like this should work:

src = "/mnt/farm/DNAlab/WGL/Samples/Data Analysis/Emedgene/HPO/2213305.HPO.txt"
target = "dbfs:/FileStore/LiveDataUpload/WES_HPO_Dec16/"

subprocess.run(["databricks", "fs", "cp", src, target], capture_output=True, check=True)