Using a local PSOCK cluster with the new crew-workflow in targets

16 views Asked by At

I understand that the recommended workflow for distributed computations in targets is via the crew package (built on mirai). I have successfully used crew to distribute work across multiple local cores using the following settings in my _targets.R file:

tar_option_set(
  controller = crew_controller_local(workers = 7)
)

In previous projects I have distributed work across mutliple machines in a local network PSOCK cluster. Essentially, in my _targets.R, I set up the connection to the PSOCK cluster, create a future-plan, and run the pipeline using tar_future_make().

library("parallel")

cl <- makePSOCKcluster(...)

future::plan(future::cluster, workers = cl)
targets::tar_make_future()

As of yet, I have been unable to set up a PSOCK cluster with crew to use it with targets. I have looked at the code in crew and crew.cluster, but I'm not making a lot of head way.

I understand that it's straight forward to create a PSOCK cluster with mirai, but I don't understand how to fit this with crew. Any pointers would be appreciated.

library("mirai")

cl <- make_cluster(
  url = "tcp://local.host.ip.address"
  , remote = ssh_config(
    remotes = c("ssh://local.ip.address.1", "ssh://local.ip.address.2"),
    tunnel = FALSE
  )
)

parallel::parLapply(cl, iris, mean)

stop_cluster(cl)
0

There are 0 answers