Problems with deleting file from sftp folder using Python

Question

I am trying to move files into a backup folder which is on the same directory level as the folder where the files are stored. The downloading of the file into a local directory and uploading the file back to the backup folder works but the files can't be deleted from the existing remote folder. I wish to remove the files one by one after each file has been processed instead of storing the paths in a list and removing all at once. I have no idea what is wrong. Hope to seek advice.

Error reading file test_abs_2022135201552.pslog: Bad message

Below is a snippet of the code. I have tested that the 'put' and 'get' part of the code works. The line sftp.remove(file_path) is the one giving problems.

source_path = "/project/"
local_path = r'C:\Users\Desktop\project'

 df_staging = pd.DataFrame()
    
def main():


    transport = paramiko.Transport((sftp_host, sftp_port))
    transport.connect(username=sftp_username, password=sftp_password)
    sftp = paramiko.SFTPClient.from_transport(transport)

    global df_staging

    try:

        main_folders = main_folders = sftp.listdir(source_path) 

        for main_folder in main_folders:
            main_folder = os.path.join(source_path, main_folder) 
            station_folders = sftp.listdir(main_folder_path) # List the station folder names 

            # To create a backup folder if it does not exist in remote server
            if 'backup' not in sftp.listdir(main_folder_path):
                sftp.mkdir(os.path.join(main_folder_path, 'backup'))
                print("Backup folder created")
                
            # To create family folder if it does not exist in local dir
            if main_folder not in os.listdir(local_path):
                os.mkdir(os.path.join(local_path, main_folder))

            for station_folder in station_folders:
                if station_folder != 'backup':
                    station_folder_path = os.path.join(main_folder_path, station_folder) 
                    remote_files = sftp.listdir(station_folder_path) 


                   local_file_path = os.path.join(local_path, main_folder, file)
                   remote_backup_path = os.path.join(main_folder_path, 'backup', file)
                   # Transfer file between remote to local
                   sftp.get(file_path, local_file_path)
                   # Transfer file back from local to remote
                   sftp.put(local_file_path, remote_backup_path)
                   # Remove file from remote station folder (This line is giving me the errors!!!)
                   sftp.remove(file_path)
                                            
                   print(f"Done transfering: {file} ")
                   print("\n")
                                        
                   except Exception as e:
                      print(f"Error reading file {file}: {str(e)}")


                    lst_files_to_delete = [path for path in lst_all_files if path not in lst_files_to_transfer]
                    print("Files to delete:", lst_files_to_delete)

                    for delete_file_path in lst_files_to_delete:
                        print(f"Deleting file: {delete_file_path}")
                        sftp.remove(delete_file_path)
                        print(f"File deleted successfully")

Answer 1

As my file was still opened to be read in the main code, the program was unable to delete the file. The problem is resolved by moving the remove line out of a loop that is reading the file

Answer 2

Essentially, sftp.remove can only work when the file is closed. In the previous example where I encountered the problem, sftp.remove was placed inside with sftp.open. After moving it out, it works. Below is the portion of the main code that I was referring to:

with sftp.open(file_path, 'r') as remote_file:
    # Main Code


### After file is closed, perform file upload and removal only after uploaded to DB ###
local_file_path = os.path.join(local_path, main_folder, file)
remote_backup_path = os.path.join(main_folder_path, 'backup', file)
# Transfer file between remote to local
sftp.get(file_path, local_file_path)
# Transfer file back from local to remote
sftp.put(local_file_path, remote_backup_path)
# Remove file from remote station folder (***Only works when file is not open!)
sftp.remove(file_path)