Unable to filter parquet file using where clause.... error "unsafe symbol Unstable"

1.3k views Asked by At

I am unable to fiter give parquet file.

I have dataframe with "family_id" of String and "lastStagedTs" of Date type. i.e. 2018-11-30 in format.

I am trying to filter the parquet file like below i.e. select data which is greater than 2018-11-23

  val lastRun:String = "2018-11-23"  
  val parquetDf = sparkSession.read.format("parquet")
                                .load(parquetFile) 
                                .select("family_id", "lastStagedTs")
                                .where($"lastStagedTs" > lastRun)   

When I'm running above line of code I getting the below error:

unsafe symbol Unstable (child of <none>) in runtime reflection universe
    at scala.reflect.internal.Symbols$Symbol.<init>(Symbols.scala:205)
    at scala.reflect.internal.Symbols$TypeSymbol.<init>(Symbols.scala:3030)
0

There are 0 answers