RCurl::getURL exceeds maximum number of clients

Question

I am trying to list files hosted on the FTP servers of the MODIS Global Evapotranspiration Project (MOD16) recurrently.

## required package
library(RCurl)

## ftp server
ch_ftp <- "ftp://ftp.ntsg.umt.edu/pub/MODIS/NTSG_Products/MOD16/MOD16A2.105_MERRAGMAO/"

## list and reformat available subfolders
ch_fls <- getURL(ch_ftp, verbose = TRUE, dirlistonly = TRUE)

ls_fls <- strsplit(ch_fls, "\n")
ch_fls <- unlist(ls_fls)

## list files in current folder
for (i in ch_fls) {

  ch_hdf <- paste0(ch_ftp, i)
  getURL(ch_hdf, verbose = TRUE, dirlistonly = TRUE)
}

After some iterations, RCurl::getURL throws the following error message.

< 530 Sorry, the maximum number of clients (5) from your host are already connected.
* Access denied: 530
* Closing connection 16
 Show Traceback

 Rerun with Debug
 Error in function (type, msg, asError = TRUE)  : Access denied: 530 

Obviously, RCurl::getURL opens connections to the FTP server during each iteration without closing them fast enough. A few minutes later, the server is accessible again, but the same error message will be thrown when re-initializing the script and waiting for the first few iterations. Is there a way to manually close the connections established by RCurl::getURL immediately after having retrieved a list of files?

Answer 1

I was dealing with the same issue.

Using Sys.sleep(2) fixed it for me.

## list files in current folder
for (i in ch_fls) {

  ch_hdf <- paste0(ch_ftp, i)
  getURL(ch_hdf, verbose = TRUE, dirlistonly = TRUE)
  Sys.sleep(2)
}