ImportError: cannot import name 'Pregel' from 'graphframes.lib'

Question

I am using pyspark and graphframes from jupyter. I am able to successfully import pyspark and graphframes, but when I try:

from graphframes.lib import Pregel 

I get the following error:

ImportError: cannot import name 'Pregel' from 'graphframes.lib'

This post is how I was able to get graphframes to work, but without graphframes.lib:

https://github.com/graphframes/graphframes/issues/104

wget https://github.com/graphframes/graphframes/archive/release-0.2.0.zip
unzip release-0.2.0.zip
cd graphframes-release-0.2.0
build/sbt assembly
cd ..

# Copy necessary files to root level so we can start pyspark. 
cp graphframes-release-0.2.0/target/scala-2.11/graphframes-release-0-2-0-assembly-0.2.0-spark2.0.jar .
cp -r graphframes-release-0.2.0/python/graphframes .

# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook

# Launch the jupyter server.
pyspark --jars graphframes-release-0-2-0-assembly-0.2.0-spark2.0.jar

I tried to repeat the above commands, without the environment lines as pyspark works just fine for me in jupyter, using a different version and was able to get graphframes.lib, but no Pregel:

wget https://github.com/graphframes/graphframes/archive/release-0.6.0.zip
unzip release-0.6.0.zip
cd graphframes-release-0.6.0
build/sbt assembly
cd ..

# Copy necessary files to root level so we can start pyspark. 
cp graphframes-release-0.6.0/target/scala-2.11/graphframes-assembly-0.6.0-spark2.3.jar .
cp -r graphframes-release-0.6.0/python/graphframes .

# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook

# Launch the jupyter server.
pyspark --jars graphframes-assembly-0.6.0-spark2.3.jar

Now I can see the graphrames.lib directory but all that's in there is aggregate_messages.py.

Finally, I tried the following but get a 404 error:

wget https://github.com/graphframes/graphframes/archive/release-0.7.0.zip

I expected that, because I was able to import graphframes, that I'd be able to import Pregel from graphframes.lib. It would seem that in my version, 0.6.0 now, there is a graphrames.lib but no Pregel and that there is no 0.7.0 release yet for graphframes.

Answer 1

I was able to resolve this error using the following method:

wget https://github.com/graphframes/graphframes/archive/f9e13ab4ac1a7113f8439744a1ab45710eb50a72.zip
unzip graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72.zip
cd graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72
build/sbt assembly
cd ..

# Copy necessary files to root level so we can start pyspark. 
cp graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72/target/scala-2.11/graphframes-assembly-0.7.0-spark2.4.jar .
cp -r graphframes-f9e13ab4ac1a7113f8439744a1ab45710eb50a72/python/graphframes .

# Set environment to use Jupyter (if jupyter working with pyspark, skip)
# export PYSPARK_DRIVER_PYTHON=jupyter
# export PYSPARK_DRIVER_PYTHON_OPTS=notebook

# launch pyspark
pyspark --jars graphframes-assembly-0.7.0-spark2.4.jar