I am new to Spark and need some guidance on the below issue - Whenever I am trying to create a SparkSession object using a SparkConf object I am getting the below error -

AttributeError: 'SparkConf' object has no attribute '_get_object_id'

I am using Spark 2.3 with Python 3.7 in local mode .

sconf=SparkConf.setAppName("test")
ss=SparkSession.builder.config(conf=sconf).getOrCreate()

I have read some of the solutions available in internet but none of them has resolved my issue .

Even when I am trying to create the SparkSession object directly i.e. without explicit SparkConf object , then also I am getting the same error -

ss=SparkSession.builder.master("local").getOrCreate()

AttributeError: 'SparkConf' object has no attribute '_get_object_id'

2 Answers

0
Community On Best Solutions

You might be missing the bracket sconf=SparkConf().setAppName("test") If it doesnt work i tried the below in local which worked

from pyspark.sql import SparkSession
from pyspark import SparkConf
sconf=SparkConf().setAppName("test")
ss=SparkSession.builder.config(conf=sconf).getOrCreate()
0
XEngineer On

You can define configuration like this -

from pyspark.conf import SparkConf
from pyspark.sql import SparkSession

conf = spark.sparkContext._conf.setAll([('spark.app.name', 'Spark Test')])

spark = SparkSession.builder.config(conf=conf).getOrCreate()