how to run evolutionary algorithms using pyspark

Question

I want to run evolutionary algorithms like GA,PSO using pyspark on spark.How to do this using MLLib using Deap python library.Is there any other library available to perform same task.

Answer 1

Have a look at my answer on how to use DEAP with Spark and see if it works for you.

Here is an example of how to configure the DEAP toolbox to replace the map function by a custom one using Spark.

from pyspark import SparkContext

sc = SparkContext(appName="DEAP")

def sparkMap(algorithm, population):
    return sc.parallelize(population).map(algorithm)

toolbox.register("map", sparkMap)

Answer 2

In https://github.com/DEAP/deap/issues/268 they show how to do this in the DEAP package. However this is an issue. but they mention there is a pull request (https://github.com/DEAP/deap/pull/76), and is seems the fixed code/branch is from a forked repo.
It sounds like if you rebuild the package with that code it should resolve the issue.

Another resource I found, haven't tried it, is https://apacheignite.readme.io/docs/genetic-algorithms. Also came across this https://github.com/paduraru2009/genetic-algorithm-with-Spark-for-test-generation