I want to generate multiple parallel r workers that do not need to wait for the other workers to finish their job before getting new jobs.
require(snow)
# generate sockets
cl <- snow::makeSOCKcluster(3, type = "SOCK", outfile="")
# here is the function that will be run in parallel
myfunc <- function(x=2)
{
print(x)
Sys.sleep(time = x)
output = sample(x = 1:150, size = 1)
return(output)
}
# here are the arguments that needs to run in parallel
myfunc_argument <- c(5, 50, 150)
snow::clusterApply(cl = cl, fun = myfunc, x = myfunc_argument)
stopCluster(cl)
In this example, the 1st worker needs to wait for the third worker to get a new job.
The next set of inputs will be the outputs (i.e. random numbers between 1 and 150 in this example) of the current run