Random errors causing Autosys Job Failure

Question

I have an autosys running daily that basically fetches data from ftp server, then processes it and saves it to a shared drive and a database. The process uses an unzip method

curl_download(url=ftp_url,destfile=zip_filename, mode="wb",handle=h)
file_list <- unzip(zip_filename,list=T)$Name
unzip(zip_filename,files=file_list,exdir=dir)
unlink(zip_filename)

Where ftp_url is an ftp site and zip_filename is a shared drive file.

I am receiving random errors in the process, most of the times at the unzip part, some of them being 'permission denied', 'file does not exist' (as the unzip happens successfully but file is not extracted at the location).

These errors come up randomly at any day, occur for a few days and then stop showing up.

Does anyone know the reasons behind these errors

I tried re running the code when it failed, but that didn't work

Answer 1

I don't know that this will work, but I think the "shared drive" provides the possibility for problem. (This can be a real network drive, or a virtual one like dropbox, onedrive, nextcloud, owncloud, etc.)

Try to use a completely-local path for both the download and unzipping portions, and then when done copy the resulting files to the shared drive. (I am assuming that tempdir() (under which tempfile-created files are placed) is on a local drive, not a shared drive.)

tf <- tempfile(fileext = ".zip") # used for the download
td <- tempfile()                 # used for extraction
dir.create(td)

curl_download(url = ftp_url, destfile = tf, mode = "wb", handle = h)
file_list <- unzip(tf, list = TRUE)$Name
unzip(tf, files = file_list, exdir = td)
# assuming `dir` is a target directory
file.copy(file.path(td, file_list), dir)

# cleanup
unlink(tf)
unlink(td, recursive = TRUE)

If your zip file contains subdirs, you might need to instead use

file.copy(list.files(td, full.names = TRUE), dir, recursive = TRUE)

where the new call to list.files only returns the files and directories on the top-level within td, and the recursive=TRUE ensures that files within subdirs are copied as well.