can't find module 'graphframes'

Question

When I input in command line :

pyspark --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11

it will work well. But when I want to use ipython to launch my pyspark and use graphframes package, it doesn't work.

When I input in command line :

PYSPARK_DRIVER_PYTHON=ipython pyspark --packages graphframes:graphframes:0.2.0-spark2.0-s_2.11

the error is that: can't find module 'graphframes'

Answer 1

Lack of Python 3 compatibility of the Spark packages is a known bug in graphframes 0.2.0 which has been resolved in 0.3.0.

--packages graphframes:graphframes:0.3.0-spark2.0-s_2.11