I am doing a large-scale simulation study on Ubuntu. I am wondering if it makes sense to try calling clusterApply with 1 worker per server, and have the clusterApply function use the cores using mcmapply. Also is there a way to get the number of unused cores rather the number of cores on a server?
Do you have suggested for other approaches? I am trying to create a portable process that someone else could run without having to manually kickoff and/or aggregate experiments from different machines, and reduce the runtime for a large experiment taking on the order of a couple weeks to run for 100+ cores.
I tried using clusterApply and so far had much lower throughput using a cluster of 4 machines with 50-100 workers per server as opposed to a single machine using mcmapply with 100 workers, perhaps because of overhead to manage ssh or IT port restrictions.
I also am wondering if I may need to move to or leverage Apache Spark. I was trying to create re-distributable software that could be run from R.