I understand that the recommended workflow for distributed computations in targets is via the crew package (built on mirai). I have successfully used crew to distribute work across multiple local cores using the following settings in my _targets.R file:
tar_option_set(
controller = crew_controller_local(workers = 7)
)
In previous projects I have distributed work across mutliple machines in a local network PSOCK cluster. Essentially, in my _targets.R, I set up the connection to the PSOCK cluster, create a future-plan, and run the pipeline using tar_future_make().
library("parallel")
cl <- makePSOCKcluster(...)
future::plan(future::cluster, workers = cl)
targets::tar_make_future()
As of yet, I have been unable to set up a PSOCK cluster with crew to use it with targets. I have looked at the code in crew and crew.cluster, but I'm not making a lot of head way.
I understand that it's straight forward to create a PSOCK cluster with mirai, but I don't understand how to fit this with crew. Any pointers would be appreciated.
library("mirai")
cl <- make_cluster(
url = "tcp://local.host.ip.address"
, remote = ssh_config(
remotes = c("ssh://local.ip.address.1", "ssh://local.ip.address.2"),
tunnel = FALSE
)
)
parallel::parLapply(cl, iris, mean)
stop_cluster(cl)