I have a spark dataframe df, with 2 columns and the following datatype
root
|-- session_id: string (nullable = true)
|-- user_id: string (nullable = true)
then I want to convert it to Pandas dataframe using df.toPandas()
. I received the following error TypeError: Unexpected obj type: <class 'int'>
I have no int
type in my spark dataframe, and dropna already. A bit confused why this is happening, any suggestions would be appreciated!
Thank you.