R access to the parallel workers dynamically

104 views Asked by At

I want to generate multiple parallel r workers that do not need to wait for the other workers to finish their job before getting new jobs.

require(snow)

# generate sockets
cl <- snow::makeSOCKcluster(3, type = "SOCK", outfile="")


# here is the function that will be run in parallel
myfunc <- function(x=2)
{
  print(x)
  Sys.sleep(time = x)
  output = sample(x = 1:150, size = 1)
  return(output)
}

# here are the arguments that needs to run in parallel
myfunc_argument <- c(5, 50, 150)

snow::clusterApply(cl = cl, fun = myfunc, x = myfunc_argument)
stopCluster(cl)

In this example, the 1st worker needs to wait for the third worker to get a new job.

The next set of inputs will be the outputs (i.e. random numbers between 1 and 150 in this example) of the current run

0

There are 0 answers