PySpark toPandas() gives TypeError: Unexpected obj type: <class 'int'>

29 views Asked by At

I have a spark dataframe df, with 2 columns and the following datatype

root
 |-- session_id: string (nullable = true)
 |-- user_id: string (nullable = true)

then I want to convert it to Pandas dataframe using df.toPandas(). I received the following error TypeError: Unexpected obj type: <class 'int'>

I have no int type in my spark dataframe, and dropna already. A bit confused why this is happening, any suggestions would be appreciated!

Thank you.

0

There are 0 answers