I am unable to fiter give parquet file.
I have dataframe with "family_id" of String and "lastStagedTs" of Date type. i.e. 2018-11-30 in format.
I am trying to filter the parquet file like below i.e. select data which is greater than 2018-11-23
val lastRun:String = "2018-11-23"
val parquetDf = sparkSession.read.format("parquet")
.load(parquetFile)
.select("family_id", "lastStagedTs")
.where($"lastStagedTs" > lastRun)
When I'm running above line of code I getting the below error:
unsafe symbol Unstable (child of <none>) in runtime reflection universe
at scala.reflect.internal.Symbols$Symbol.<init>(Symbols.scala:205)
at scala.reflect.internal.Symbols$TypeSymbol.<init>(Symbols.scala:3030)