Limit of workers in future::plan() function higher than available CPUs cores

Question

I have a CPU of 8 cores which I'm trying to use to do parallel future_map_chr().
My question is as follows: the number of cores in my computer is equal to 8, so how is it possible to parallelize function with plan set as :

plan(multisession, workers = 50,gc=T)

Did I misunderstood the difference between workers and cores ? Is it better than workers = detectCores() ?

Thanks a lot

Answer 1

Author of the Futureverse here:

Is it better than workers = detectCores()?

I argue that parallelly::availableCores() is better and safer than parallel::detectCores(), e.g. https://www.jottr.org/2022/12/05/avoid-detectcores/.

Using:

plan(multisession)

is the same as:

plan(multisession, workers = parallelly::availableCores())

However, there is nothing in the Futureverse and plan() that prevents you from going over the limit by specifying workers manually, e.g.

plan(multisession, workers = 50)

So, yes, this will spin up 50 parallel workers in the background, even if you only got 8 CPU cores on your machine.

I guess I could add some kind of protection when one tries to set up more than parallelly::availableCores() local workers. For instance, it could throw an informative error. For anyone who wishes to override that, I could add an argument force = TRUE. I've created https://github.com/HenrikBengtsson/parallelly/issues/107 to track this idea.