I have a cluster running Ubuntu 22.04.04 LTS and R 4.3.3. I am trying to use clusterApply with a simple network of workstations. When I call parallel::makecluster with an array of serveripaddresses, R just waits unless the serveripaddreseses all refer to the localhost. I tried allowing all traffic in and out from ports 22 and 11000 to 11999, and still have the same issue.
# Define the cluster
cl <- parallel::makeCluster(c(server1,server2,server3))
# Distribute the computation across the cluster
results <- parallel::clusterApply(cl, c(1, 2, 3), function(x) {
x^2
})
# Close the cluster when done
parallel::stopCluster(cl)
# Results will contain the output of the computations performed on each server
print(results)
I also tried the following
workers <- c(rep(server1,2), rep(server,2))
cl <- parallel::makeCluster(workers, outfile = "")
I got back the following: starting worker pid=2517175 on machinename:11168 at 22:30:24.300 Error in socketConnection(master, port = port, blocking = TRUE, open = "a+b", : cannot open the connection Calls: -> workLoop -> makeSOCKmaster In addition: There were 17 warnings (use warnings() to see them) Execution halted
The following seems to work: workers <- c(server_1, server_2) cl <- future::makeClusterPSOCK(workers, revtunnel = TRUE, outfile = "", verbose = TRUE) results <- parallel::clusterApply(cl, c(1, 2, 3), function(x) { x^2 }) parallel::stopCluster(cl)
print(results)
Does that mean IT is blocking traffic? RStudio is asking me for a password for every worker on localhost if called by ipaddress, and some servers. Is there an easy way to provide it in the code and not be prompted repeatedly? I likely want to run multiple processes on multiple nodes.
I am actually getting worse performance with the cluster than with a single server because I see far fewer R processes running on localhost than I specified, and am able to use with mcmapply. Any ideas why? I am assuming this is atypical because it defeats the purpose of using a cluster.