foreach %dopar% long vectors not supported yet: memory.c:3782

122 views Asked by At

I am trying to use snow to perform some multiprocessing using this code:

cl <- makeCluster(32)
registerDoSNOW(cl)

result <-
    foreach(
      i = 1:iterations,
    ) %dopar%
    {
      current_value <- pull(nearby_genes[i,1])
      
      # DO ANALYSIS FOR CURRENT VALUE

}

This results in the following error: mpi.send(x = serialize(obj, NULL), type = 4, dest = dest, tag = tag, : long vectors not supported yet: memory.c:3782. I did some research and found out that it has something to do with memory issues. One of the dataframes that I am using inside the foreach loop is >5million rows long and has 14 columns so I think it occupies more than 2GB of RAM which triggers the error.

Am I right in this assumption and does anyone have an idea how to circumvent this problem?

Any help is much appreciated!

0

There are 0 answers