TypeError: 'Builder' object is not callable Spark structured streaming

11.3k views Asked by At

On running the example given in the programming guide[link] for python spark structured streaming
http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

I get below Error :
TypeError: 'Builder' object is not callable

from pyspark.sql import SparkSession
from pyspark.sql.functions import explode
from pyspark.sql.functions import split

spark = SparkSession.builder()\
    .appName("StructuredNetworkWordCount")\
    .getOrCreate()

# Create DataFrame representing the stream of input lines from connection to localhost:9999
lines = spark\
   .readStream\
   .format('socket')\
   .option('host', 'localhost')\
   .option('port', 9999)\
   .load()

# Split the lines into words
words = lines.select(
   explode(
       split(lines.value, ' ')
   ).alias('word')
)

# Generate running word count
wordCounts = words.groupBy('word').count()

# Start running the query that prints the running counts to the console
query = wordCounts\
    .writeStream\
    .outputMode('complete')\
    .format('console')\
    .start()

query.awaitTermination()

Error :

omkar@rudra:~/thesis/backUp$ spark-submit structured.py 
Traceback (most recent call last):
  File "/home/omkar/thesis/backUp/structured.py", line 8, in <module>
    spark = SparkSession.builder()\
TypeError: 'Builder' object is not callable
2

There are 2 answers

0
OSK On

For

spark = SparkSession.builder()\
    .appName("StructuredNetworkWordCount")\
    .getOrCreate()

modify .builder() to .builder as :

spark = SparkSession.builder\
    .appName("StructuredNetworkWordCount")\
    .getOrCreate()

Source : https://issues.apache.org/jira/browse/SPARK-18426

0
Amith Rangadhol On

When running python example in Structured Streaming Guide, get the error:

spark = SparkSession.builder().master("local[1]").appName("Example").getOrCreate()

TypeError: 'Builder' object is not callable

This is fixed by changing .builder() to .builder

spark = SparkSession.builder.master("local[1]").appName("Demo").getOrCreate()

After removing this-() in builder while creating sparksession,the code will run.